How-to use SADMIN alerting system
- First you need a Slack workspace, unless you want to only use email as the alerting system.
- If you don't have one, you should read our How-to create a Slack workspace page and create one.
- You will also need at least one channel and one Slack App. in order to receive alert on your channel.
If you don't, have a look at the page How-to create Slack channel and Slack App..
- You'll need your favorite editor, to modify some configuration files and to create a test script.
Understanding SADMIN alerting facilities
This guide, we will demonstrate with an example, how you can use the SADMIN alerting facilities.
The default alert group
- An alert can be issue by a script or from the SADMIN System Monitor and it's sent to an alert group.
- By default, the alert group name that will be use is the one specified in the SADMIN configuration file.
- Every system have his own SADMIN configuration file ($SADMIN/cfg/sadmin.cfg).
- The default alert group after the installation is called (guess what) 'default'.
- You can change the 'SADM_ALERT_GROUP' name with your favorite editor whenever you need to.
- The group name you specify, MUST exist in the Alert Group file ($SADMIN/cfg/alert_group.cfg).
The default alert type
- The alert type is use only at the end of the script, to decide if an alert/notification need to be send.
- The default alert type is determine at installation time and is normally set to '1'.
- You can view that actual value by looking in the SADMIN configuration file ($SADMIN/cfg/sadmin.cfg).
- When you run a Python or a Shell script using the SADMIN Tools, it use this default value.
- The Alert type have four possible values :
- 0 Meaning that you don't want any alert to be send, either if the script finish with success or failure.
- 1 You want to send an alert only if the script finish with an error (Exit code not equal to 0).
- 2 An alert will be send only when the script finish with success (Exit code = 0).
- 3 Always send an alert, whether the script terminate with failure or success.
Override the default alert group and type (in a script)
- You can override the default alert group and type for a particular script by changing the lines below.
- Uncomment the lines below to override the default values of these in the SADMIN configuration file.
- To override SADM_ALERT_TYPE and/or SADM_ALERT_GROUP default value, remove the '#' and change it to the value of your choice.
- Remember that the alert group name you assign to SADM_ALERT_GROUP variable, must exist in the alert group file.
- We have an override example, later on this page.
- These lines are located in the SADMIN section near the top of the script.
Shell Script :
#export SADM_ALERT_TYPE=1 # 0=None 1=AlertOnErr 2=AlertOnOK 3=Allways
#export SADM_ALERT_GROUP="default" # AlertGroup Use for Alert(alert_group.cfg)
Python Script :
#inst.cfg_alert_type = 1 # 0=None 1=AlertOnErr 2=AlertOnOK 3=Allways
#inst.cfg_alert_group = "default" # AlertGroup use for Alert(alert_group.cfg)
Override the default alert group in SADMIN System Monitor
- For the System Monitor, alert group are specify in column J for Warning and in column K for Error.
- Example to check filesystem space usage in SysMon configuration file (hostname.smon).
#Column 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L
FS/opt 69 >= 85 90 000 0000 0000 Y Y Y Y Y Y Y Y 00000000 0000 sdevops sprod -
- In the example above ;
- If the filesystem usage percentage become greater or equal to 85%, a warning alert will be send to 'sdevops' alert group.
- If the filesystem usage percentage become greater or equal to 90%, an error alert will be send to 'sprod' alert group.
- If the Alert group used in SysMon configuration don't exist in the alert group file, the group 'default' is use.
Overriding default alert group and type in a script
- In this example we will show you how to override the default alert group and type in your script.
This is the alert group file ($SADMIN/cfg/alert_group.cfg) we used for the example below.
Here is a portion of the Slack alert file ($SADMIN/cfg/alert_slack.cfg), we used for our example.
- Let's make a copy of the shell template script to create our test script.
- In our test script, we will change the alert group from the default to 'sprod' and the alert type from 1 to 3.
- Let's begin by typing the following command to create our test script name 'sadm_test_alert.sh' ;
# cd $SADMIN/usr/bin
# /sadmin/usr/bin cp $SADMIN/bin/sadm_template.sh sadm_test_alert.sh
# /sadmin/usr/bin nano sadm_test_alert.sh
Script before the change
Script after the change
Running our test script
Alert that we received in Slack
A look at the 'log' and 'rch' file generated
It is simple as that, so I hope you have a better understanding of how the alarm system work in SADMIN.
We could have done the same test using the Python template ($SADMIN/bin/sadm_template.py) script.