How-to use SADMIN alerting system


Requirements:

  • First you need a Slack workspace, unless you want to only use email as the alerting system.
  • If you don't have one, you should read our How-to create a Slack workspace page and create one.
  • You will also need at least one channel and one Slack App. in order to receive alert on your channel.
    If you don't, have a look at the page How-to create Slack channel and Slack App..
  • You'll need your favorite editor, to modify some configuration files and to create a test script.

Understanding SADMIN alerting facilities

This guide, we will demonstrate with an example, how you can use the SADMIN alerting facilities.
 

The default alert group

  • An alert can be issue by a script or from the SADMIN System Monitor and it's sent to an alert group.
  • By default, the alert group name that will be use is the one specified in the SADMIN configuration file.
  • Every system have his own SADMIN configuration file ($SADMIN/cfg/sadmin.cfg).
  • The default alert group after the installation is called (guess what) 'default'.



  • You can change the 'SADM_ALERT_GROUP' name with your favorite editor whenever you need to.
  • The group name you specify, MUST exist in the Alert Group file ($SADMIN/cfg/alert_group.cfg).
 

The default alert type

  • The alert type is use only at the end of the script, to decide if an alert/notification need to be send.
  • The default alert type is determine at installation time and is normally set to '1'.
  • You can view that actual value by looking in the SADMIN configuration file ($SADMIN/cfg/sadmin.cfg).
  • When you run a Python or a Shell script using the SADMIN Tools, it use this default value.



  • The Alert type have four possible values :
    • 0 Meaning that you don't want any alert to be send, either if the script finish with success or failure.
    • 1 You want to send an alert only if the script finish with an error (Exit code not equal to 0).
    • 2 An alert will be send only when the script finish with success (Exit code = 0).
    • 3 Always send an alert, whether the script terminate with failure or success.
 

Override the default alert group and type (in a script)

  • You can override the default alert group and type for a particular script by changing the lines below.
  • Uncomment the lines below to override the default values of these in the SADMIN configuration file.
  • To override SADM_ALERT_TYPE and/or SADM_ALERT_GROUP default value, remove the '#' and change it to the value of your choice.
  • Remember that the alert group name you assign to SADM_ALERT_GROUP variable, must exist in the alert group file.
  • We have an override example, later on this page.
  • These lines are located in the SADMIN section near the top of the script.

  • Shell Script : #export SADM_ALERT_TYPE=1 # 0=None 1=AlertOnErr 2=AlertOnOK 3=Allways #export SADM_ALERT_GROUP="default" # AlertGroup Use for Alert(alert_group.cfg)
    Python Script : #inst.cfg_alert_type = 1 # 0=None 1=AlertOnErr 2=AlertOnOK 3=Allways #inst.cfg_alert_group = "default" # AlertGroup use for Alert(alert_group.cfg)
 

Override the default alert group in SADMIN System Monitor

  • For the System Monitor, alert group are specify in column J for Warning and in column K for Error.
  • Example to check filesystem space usage in SysMon configuration file (hostname.smon).
  • #Column 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L FS/opt 69 >= 85 90 000 0000 0000 Y Y Y Y Y Y Y Y 00000000 0000 sdevops sprod -
  • In the example above ;
    • If the filesystem usage percentage become greater or equal to 85%, a warning alert will be send to 'sdevops' alert group.
    • If the filesystem usage percentage become greater or equal to 90%, an error alert will be send to 'sprod' alert group.
  • If the Alert group used in SysMon configuration don't exist in the alert group file, the group 'default' is use.
 

Overriding default alert group and type in a script

  • In this example we will show you how to override the default alert group and type in your script.

  • This is the alert group file ($SADMIN/cfg/alert_group.cfg) we used for the example below.

    Here is a portion of the Slack alert file ($SADMIN/cfg/alert_slack.cfg), we used for our example.

  • Let's make a copy of the shell template script to create our test script.
  • In our test script, we will change the alert group from the default to 'sprod' and the alert type from 1 to 3.
  • Let's begin by typing the following command to create our test script name 'sadm_test_alert.sh' ;
  • # cd $SADMIN/usr/bin # /sadmin/usr/bin # /sadmin/usr/bin cp $SADMIN/bin/sadm_template.sh sadm_test_alert.sh # /sadmin/usr/bin nano sadm_test_alert.sh
    Script before the change


    Script after the change


    Running our test script


    Alert that we received in Slack


    A look at the 'log' and 'rch' file generated


It is simple as that, so I hope you have a better understanding of how the alarm system work in SADMIN.
We could have done the same test using the Python template ($SADMIN/bin/sadm_template.py) script.

Copyright © 2015-2019 - www.sadmin.ca - Suggestions, Questions or Report a problem at support@sadmin.ca