Using SADMIN Shell script template

13 minute read

Posted 2021-03-16 - Updated
Supported on Linux, Aix, MacOS

What’s in the shell script template

The shell script template ($SADMIN/bin/sadm_template.sh) is ready to use. You can run it now, by issuing the following command; ‘$SADMIN/bin/sadm_template.sh’. We will demonstrate later on, what is included in the template and what are the benefits of using it. The best way to learn how to use the SADMIN tools, is to create a script from the template and this is what we will demonstrate later on on this page.

This template include two main functions, the ‘process_servers()’ and the ‘main_process()’.
sadm_template_sh_03.png

  • The ‘main_process()’ is normally use when you’re writing a script that don’t need any SADMIN database information. By default, the ‘main_process()’ is activated, but you decide which one you need for the script you’re starting.
    sadm_template_sh_main_process.png

  • In the Shell Script template, you will also find a function that demonstrate how to access the information in the server database. The ‘process_servers()’ can be use when you need to perform action based on the content of the server database. It actually build a list of all active systems (active field set to true) and test the ‘ssh’ to each of the servers. To use it, just remove the ‘#’ in front of the ‘process_server’ line. Running the script won’t harm anything, so can run it to see it in action.

The template script include three things

1- The SADMIN definition section, you will find near the beginning of the template shell script, a code section that is require to use the SADMIN tools, we will call it the ‘SADMIN Section’. This is where defined and set variables that will influence the behavior of the library. Of course you can use them within your script. This is where the shell library is loaded. for a list of all the library functions and variables that are available to you, please visit the Shell Library page.

# ------------------- S T A R T  O F   S A D M I N   C O D E    S E C T I O N  ---------------------
# v1.56 - Setup for Global Variables and load the SADMIN standard library.
#       - To use SADMIN tools, this section MUST be present near the top of your code.    

# Make Sure Environment Variable 'SADMIN' Is Defined.
if [ -z "$SADMIN" ] || [ ! -r "$SADMIN/lib/sadmlib_std.sh" ]            # SADMIN defined? Libr.exist
    then if [ -r /etc/environment ] ; then source /etc/environment ;fi  # LastChance defining SADMIN
         if [ -z "$SADMIN" ] || [ ! -r "$SADMIN/lib/sadmlib_std.sh" ]   # Still not define = Error
            then printf "\nPlease set 'SADMIN' environment variable to the install directory.\n"
                 exit 1                                                 # No SADMIN Env. Var. Exit
         fi
fi 

# YOU CAN USE THE VARIABLES BELOW, BUT DON'T CHANGE THEM (Used by SADMIN Standard Library).
export SADM_PN=${0##*/}                                    # Script name(with extension)
export SADM_INST=$(echo "$SADM_PN" |cut -d'.' -f1)         # Script name(without extension)
export SADM_TPID="$$"                                      # Script Process ID.
export SADM_HOSTNAME=$(hostname -s)                        # Host name without Domain Name
export SADM_OS_TYPE=$(uname -s |tr '[:lower:]' '[:upper:]') # Return LINUX,AIX,DARWIN,SUNOS 
export SADM_USERNAME=$(id -un)                             # Current user name.

# YOU CAB USE & CHANGE VARIABLES BELOW TO YOUR NEEDS (They influence execution of SADMIN Library).
export SADM_VER='4.3'                                      # Script version number
export SADM_PDESC="SADMIN template shell script"           # Script Optional Desc.(Not use if empty)
export SADM_ROOT_ONLY="N"                                  # Run only by root ? [Y] or [N]
export SADM_SERVER_ONLY="N"                                # Run only on SADMIN server? [Y] or [N]
export SADM_EXIT_CODE=0                                    # Script Default Exit Code
export SADM_LOG_TYPE="B"                                   # Log [S]creen [L]og [B]oth
export SADM_LOG_APPEND="N"                                 # Y=AppendLog, N=CreateNewLog
export SADM_LOG_HEADER="Y"                                 # Y=ProduceLogHeader N=NoHeader
export SADM_LOG_FOOTER="Y"                                 # Y=IncludeFooter N=NoFooter
export SADM_MULTIPLE_EXEC="N"                              # Run Simultaneous copy of script
export SADM_USE_RCH="Y"                                    # Update RCH History File (Y/N)
export SADM_DEBUG=0                                        # Debug Level(0-9) 0=NoDebug
export SADM_TMP_FILE1=$(mktemp "$SADMIN/tmp/${SADM_INST}1_XXX") 
export SADM_TMP_FILE2=$(mktemp "$SADMIN/tmp/${SADM_INST}2_XXX") 
export SADM_TMP_FILE3=$(mktemp "$SADMIN/tmp/${SADM_INST}3_XXX") 

# LOAD SADMIN SHELL LIBRARY AND SET SOME O/S VARIABLES.
. "${SADMIN}/lib/sadmlib_std.sh"                           # Load SADMIN Shell Library
export SADM_OS_NAME=$(sadm_get_osname)                     # O/S Name in Uppercase
export SADM_OS_VERSION=$(sadm_get_osversion)               # O/S Full Ver.No. (ex: 9.0.1)
export SADM_OS_MAJORVER=$(sadm_get_osmajorversion)         # O/S Major Ver. No. (ex: 9)
#export SADM_SSH_CMD="${SADM_SSH} -qnp ${SADM_SSH_PORT} "   # SSH CMD to Access Systems

# VALUES OF VARIABLES BELOW ARE LOADED FROM SADMIN CONFIG FILE ($SADMIN/cfg/sadmin.cfg)
# BUT THEY CAN BE OVERRIDDEN HERE, ON A PER SCRIPT BASIS (IF NEEDED).
#export SADM_ALERT_TYPE=1                                   # 0=No 1=OnError 2=OnOK 3=Always
#export SADM_ALERT_GROUP="default"                          # Alert Group to advise
#export SADM_MAIL_ADDR="your_email@domain.com"              # Email to send log
#export SADM_MAX_LOGLINE=500                                # Nb Lines to trim(0=NoTrim)
#export SADM_MAX_RCLINE=35                                  # Nb Lines to trim(0=NoTrim)
#export SADM_PID_TIMEOUT=7200                               # Sec. before PID Lock expire
#export SADM_LOCK_TIMEOUT=3600                              # Sec. before Del. System LockFile
# --------------- ---  E N D   O F   S A D M I N   C O D E    S E C T I O N  -----------------------

2- The call to the ‘sadm_start’ function to initialize SADMIN environment

# Call SADMIN Initialization Procedure
sadm_start                                              # Init Env Dir & RC/Log File
if [ $? -ne 0 ] ; then sadm_stop 1 ; exit 1 ;fi         # Exit if Problem 

3- The call to the ‘sadm_stop’ function at the end of your script (With one parameter, ‘0’ = Success or a non-zero value = Error).

# Your Main process procedure
main_process                                            # Main Process
SADM_EXIT_CODE=$?                                       # Save Process Return Code 

SADM_EXIT_CODE=$?                                       # Save Process Return Code 
sadm_stop $SADM_EXIT_CODE                               # Close/Trim Log & Del PID
exit $SADM_EXIT_CODE                                    # Exit With Global Err (0/1)

Example using the template

First thing to do is to make a copy of the template shell script, into the user (usr/bin) directory, so it won’t be touch by subsequent release update.

$ cp $SADMIN/bin/sadm_template.sh $SADMIN/usr/bin/helloWorld.sh`  

You should never modify the original template, always make a copy and use the copy. You will use it more than once and it could get overwritten when updating to a newer version of SADMIN. The user directory ($SADMIN/usr) is the right place to put any new script you create. This directory ($SADMIN/usr) will never be modified by newer version of SADMIN.

To create our first ‘Hello World’ shell script using SADMIN tools, with the editor of your choice just add the line below in the function called ‘main_process()’ of your new script “helloWorld.sh”.

sadm_write "Hello World\n"

One of the function included in the SADMIN Shell Library is “sadm_write()”. By default, it write the string it received to the screen and to the script log. But you can modify this behavior, by modifying the variable “SADM_LOG_TYPE” in SADMIN section.

export SADM_LOG_TYPE="B"             # Write goes to [S]creen [L]ogFile [B]oth`   

Let’s run our script now and see the output we get.

sadm_template_sh_06.png

The Script Header

The script header, is a block of information about the script you are running. The script header is generated automatically by the SADMIN tools shell library. Normally, it is shown of the screen and recorded with a time stamp in the script log.

sadm_template_sh_header.png

The script header is created by the ‘sadm_start()’ function that is part of the SADMIN Shell Library. By default, every script using SADMIN library will output (Log and Screen) a script header. Unless you set the variable ‘SADM_LOG_HEADER’ to ‘N’ in the SADMIN Section.

export SADM_LOG_HEADER="Y"      # Y=ProduceLogHeader N=NoHeader`  

The second line of the header show the date, the script name and version and the SADMIN library version. You set your script version number in the SADMIN section of the script.

export SADM_VER='1.0'           # Your Current Script Version`  

The Library version is available to you via a read only variable called “$SADM_LIB_VER”. The next line, show the system name (FQDN) and the system type (LINUX,AIX,DARWIN). The fourth line show the distribution name, it’s version number and version of the kernel

The script footer, is a block of information about the script that just ended. It is automatically generated by the SADMIN tools shell library when the ‘*sadm_stop()’ function is called. Normally, it is shown of the screen and recorded with a time stamp in the script log.

sadm_template_sh_footer.png

By default, the footer is generated, unless you set the variable ‘SADM_LOG_FOOTER’ to ‘N’ in the SADMIN section.

    export SADM_LOG_FOOTER="N"      # [Y]=Include Log Footer [N]=No log Footer

The first line show the exit code of the script (0=Success 1=Error) and the execution time of the script (HH:MM:SS). The second line show the name of the [R]eturn [C]ode [H]istory file and the maximum of lines, you decided to have in it. The next two lines inform you if an alert will be send or not and the name of the alert group that it to which it would send it. Just after come the log line, that give us the log file name and state that is was appended to previous content or a new log was created. The last line show the name of the script and the date and time it as terminated.

To suppress header/footer from log and screen, change “SADM_LOG_FOOTER” and “SADM_LOG_HEADER” variable from “Y” to “N”.
sadm_template_sh_20.png

As you can see now the header and footer doesn’t appear anymore on the screen (and in the log of the script).
sadm_template_sh_24.png


The [R]esult [C]ode [H]istory (.rch) file

The History (*.rch) file format

The Result Code History file, is a simple text file that is automatically created by the SADMIN Tools at the end of the execution of a script. It is use to record some information and statistics at the end of each script. Included in this file, is the date and time the script was started, when it ended and did it finish with success or failure. With this file we can also determine if a notification should be sent, who to send it and how to send it with the use of the alert group included on the line. Below is an example of what a RCH file look like.

$ cat $SADMIN/dat/rch/holmes_sadm_template.rch
holmes 2021.04.05 13:32:18 2021.04.05 13:32:22 00:00:04 sadm_template default 1 0
$ 
System Start Date Start Time End Date End Time Elapse Script Name Alert Group Alert Type Status
holmes 2021.04.05 13:32:18 2021.04.05 13:32:22 00:00:04 sadm_template default 1 0

First we have the host name where the script was executed, then the date and time it started and finished, followed by the elapse execution time of the script. The seventh field is the script name (without extension) and then the alert group that the notification (alert) would be sent.

The alert type is the ninth field, it dictate the action to do base on the success or failure of the script.
‘0’ mean never send an alert, whether the script was a success or not.
‘1’ mean if the script terminated with an error, send an alert.
‘2’ mean if the script terminated with success, send an alert.
‘3’ mean always send an alert whether the script was a success or not.

The last column indicate the script termination status, ‘0’ mean success and a ‘1’ a failure. It is important to know, that when a script fail it will always appear as failed in the SADMIN Web Monitor page as failed. It will stay on the monitor page until it run again and terminate with success or that you edit the ‘RCH’ file on the system that generated the error and remove the error line or change the last column from ‘1’ to ‘0’.

Do I want to use RCH file or not

By default, every time you execute a script that’s using the SADMIN Tools, there will be a line is added to the history file. If the file doesn’t exist it will be created. If the script failed, normally you get a notification (alert) by email, by SMS or by Slack. There are time when you don’t want to be alerted whatever the result of a script is. For example, if you created an interactive script you may not care about the exit code, you can then disable usage of the RCH file.
Within the shell script you can set variable ‘SADM_USE_RCH’ to ‘N’ in the SADMIN definition section of the script.

export SADM_USE_RCH="N"  # Generate or not Entry in Result Code History file`  

Within a Python script set attribute ‘st.use_rch’ to ‘False’ in the SADMIN definition section of the script.

st.use_rch = False             # Generate entry in Result Code History (.rch)

The History file name convention

The history file name (.rch) is prefix by the hostname, an underscore and followed by the script name and have an extension of ‘.rch’. Let’s assume we are on a host named ‘holmes’ and the script name is ‘helloWorld’, so the history file name will be ‘holmes_helloWorld.rch’. The RCH file are created in the SADMIN data directory ‘$SADMIN/dat/rch’.

RCH files automatically transferred to SADMIN Server

The SADMIN server collect every history files created or modified on every clients at regular interval (every 5min via a cron job in /etc/cron.d/sadm_server). It is one of the responsibility of script ‘sadm_fetch_clients.sh’ to transfer any history changed to the SADMIN server. Information included in this file will be visible from the Web interface and from the command line on the SADMIN server. It is also be used to trigger an alert, if you requested it.

Controlling the history file size

Depending how frequently you run a particular script you may want to control the size of the history file. If you run a script on a daily basis and you want to keep one month of history, then you may want to keep around 31 linus in the ‘RCH’ file. To same some disk space and to keep transfer time to a minimum, you can control the number of lines you want to keep in each history file. The default maximum number of lines to keep in the ‘.rch’ file is 35 lines and it’s taken from the SADMIN configuration file ($SADMIN/cfg/sadmin.cfg).

SADM_MAX_RCHLINE=35

You can override this default by setting the maximum number of line to trim directly in your script by changing the associated variable.

In a Shell script you change the value of the variable ‘SADM_MAX_RCLINE

`export SADM_MAX_RCLINE=31     # When Script End Trim rch file to 31 lines   

In a Python script you change the value of the ‘st.cfg_max_rchline’ attribute.

st.cfg_max_rchline  = 31       # When Script End Trim rch file to 31 Lines


Preventing the history from being trim in a Shell script, can be done by setting the value of the variable ‘SADM_MAX_RCLINE’ to ‘0’.

export SADM_MAX_RCLINE=0   # Don't trim the history file (.rch) 

Preventing the history from being trim in a Python script, can be done by changing the value of the ‘st.cfg_max_rchline’ attribute to ‘0’.

st.cfg_max_rchline  = 0    # Don't trim the history file (.rch) 

[R]esult [C]ode [H]istory File Summary

rch_file_format.png

The script log file

When a script using the SADMIN Tools is executed, there is a log file that is created or updated. All scripts logs are created in the “$SADMIN/log” directory. The log file name is prefix by the host name, an underscore and followed by the script name and have an extension of ‘.log’. For example, if the host name is ‘holmes’, the script name is ‘helloWorld’, then the log name will be ‘holmes_helloWorld.log’.

Log files automatically transferred to SADMIN Server

The SADMIN server collect every log files created or modified on every clients at regular interval (every 5min via a cron job in /etc/cron.d/sadm_server). It is one of the responsibility of script ‘sadm_fetch_clients.sh’ to transfer any ‘*.log’ changed to the SADMIN server. Information included in this file will be visible from the Web interface and from the command line on the SADMIN server.

Controlling the log file size

The log of a script can be cumulative or a new one can be created at each execution. The default in the template is set to created a new log file at each execution (SADM_LOG_APPEND=”N”). But if you want your script log to be cumulative, then set the SADM_LOG_APPEND=”Y” to “Y”.

    export SADM_LOG_APPEND="Y"     # [Y]=Append Existing Log [N]=Create New One

By default the log file is trim at the end of each execution. The log file is trim to the number of line specified in $SADMIN/cfg/sadmin.cfg by the variable ‘SADM_MAX_LOGLINE’ (default 500). You can override this default by changing ‘SADM_MAX_LOGLINE’ variable in the SADMIN Section of your script.

    export SADM_MAX_LOGLINE=500   # At end of script Trim log to 500 Lines  

Example of a log file

sadm_template_sh_30.png


Scripts view from the web interface

sadm_template_sh_34.png

Viewing the log from the web interface

sadm_template_sh_42.png

Viewing the History file (RCH) from the web interface

sadm_template_sh_38.png

Viewing all systems history (.rch) files from the command line

For more information on this command line script, see the man page of srch.

# srch  

sadm_template_sh_45.png