Supervisor - Monitor your Linux Servers and Processes

Some users need only to control process state and don’t want or need full-blown shell access to the machine on which the processes are running. They often don't need to know processes which listen on 「low」 TCP ports and often need to be started and restarted as the root user. There are processes which often need to be started and stopped in groups, sometimes regarding a priority order. A tool name supervisor helps users to manage your processes on your server and monitor each process or a group of processes.

What is supervisor?

Supervisor is a client/server system that allows its users to control a number of processes on Linux and UNIX-like operating systems. Processes can be grouped into 「process groups」 and a set of logically related processes can be stopped and started as a unit. It starts its subprocesses via fork/exec and subprocesses don’t daemonize. The operating system signals Supervisor immediately when a process terminates.

Supervisor is based on four components:

7 Best CMMS Software Solutions

To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video

7 Best CMMS Software Solutions

• supervisord: It is the server piece which is responsible for starting child programs at its own invocation, responding to commands from clients, restarting crashed or exited subprocesseses.
• supervisorctl: it is the command-line client piece which provides a shell-like interface to the features provided by supervisord. A user can connect to different supervisord processes, get status on the subprocesses controlled by, stop and start subprocesses of, and get lists of running processes of a supervisord
• a web server: it is a (sparse) web user interface with functionality comparable to supervisorctl may be accessed via a browser if you start supervisord against an internet socket
• an XML-RPC Interface: the same HTTP server which serves the web UI serves up an XML-RPC interface that can be used to interrogate and control supervisor and the programs it runs

The supervisor tool allows you to assign priorities to processes and allows user to emit commands via the supervisorctl client like 「start all」, and 「restart all」, which starts them in the preassigned priority order. It is written entirely in Python and the actual version used is 3.2.1. Supervisor does not work on Python 3

1) Install supervisor

Supervisor offers the possibility to be installed but I will show you the installation through distribution packages.

On centos 7

# yum install epel-release -y && yum update

# yum install supervisor
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.linode.com
 * epel: fedora-epel.mirrors.tds.net
 * extras: mirrors.linode.com
 * updates: mirrors.linode.com
Resolving Dependencies
--> Running transaction check
---> Package supervisor.noarch 0:3.1.4-1.el7 will be installed
--> Processing Dependency: python-meld3 >= 0.6.5 for package: supervisor-3.1.4-1.el7.noarch
--> Processing Dependency: python-setuptools for package: supervisor-3.1.4-1.el7.noarch

On Ubuntu 16.04

# apt update && apt upgrade

# apt install supervisor
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following additional packages will be installed:
 python-meld3 python-pkg-resources
Suggested packages:
 python-setuptools supervisor-doc
The following NEW packages will be installed:
 python-meld3 python-pkg-resources supervisor
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 392 kB of archives.

2) Start with the monitoring tool supervisor

The supervisor configuration file is /etc/supervisord.conf. It is used both for supervisord and supervisorctl commands.

a) Configure the supervisor web server

The supervisor configuration file contains two sections named [unix_http_server] and [inet_http_server] under which we have configuration parameters for an HTTP server that listens on a UNIX domain and TCP (internet) sockets should be inserted. Your sections should look like below

# vim /etc/supervisor.conf

[unix_http_server]

file=/var/run/supervisor/supervisor.sock ; (the path to the socket file) ;chmod=0700 ; sockef file mode (default 0700) ;chown=nobody:nogroup ; socket file uid:gid owner ;username=user ; (default is no username (open server)) ;password=123 ; (default is no password (open server)) [inet_http_server] ; inet (TCP) server disabled by default port=server-ip-address:9001 ; (ip_address:port specifier, *:port for all iface) username=user ; (default is no username (open server)) password=123 ; (default is no password (open server))

 b) Add a program to monitor

Before supervisord will do anything you’ll need to add at least one program section to its configuration so that you will be able to monitor the program itself. The program section will define a program that is run and managed when you invoke the supervisord command. So, to add a program we will edit the configuration file and add to the end, for example, the command below:

# vim /etc/supervisord.conf

[program:cat-command]

command=/bin/cat ; the program (relative uses PATH, can take args) process_name=%(program_name)s ; process_name expr (default %(program_name)s) numprocs=1 ; number of processes copies to start (def 1)

[program:apache2]

command=/usr/sbin/httpd -c "ErrorLog /dev/stdout" -DFOREGROUND redirect_stderr=true

c) Start supervisord

To start supervisord, we will run supervisord command. For the first time, You can use the -n option to launch it in the foreground which is useful to debug startup problems and the -c flag to indicate the full path of the configuration file. The resulting process will daemonize itself and detach from the terminal. It keeps operations log at /var/log/supervisor/supervisord.log by default. If you don't use the -c, supervisord will first search the configuration in the current folder

# supervisord -c /etc/supervisord.conf

Now we can check the supervisor log file

# tail -f /var/log/supervisor/supervisord.log 
2017-09-07 08:30:53,725 INFO daemonizing the supervisord process
2017-09-07 08:30:53,726 INFO supervisord started with pid 789
2017-09-07 08:30:54,729 INFO spawned: 'cat-command' with pid 790
2017-09-07 08:30:54,731 INFO spawned: 'apache2' with pid 791
2017-09-07 08:30:55,805 INFO success: cat-command entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-09-07 08:30:55,807 INFO spawned: 'apache2' with pid 794

We can see that it starts. If you use the -n option, the supervisor will be launch in foreground and you will need to send it in background to control your terminal

3) Monitor your processes with supervisor

We can use the web interface to manage the processes with supervisor or we can use thesupervisorctl command line. We have configured two processes but one process doesn't work well, so we will see how supervisor will check our processes

a) Monitor the process with supervisorctl

supervisorctl use some basics commands to manage your processes: stop, start, status, restart, reload, tail, etc.. To manage your process, use the supervisorctl command

# supervisorctl status
apache2 FATAL Exited too quickly (process log may have details)
cat-command RUNNING pid 790, uptime 0:02:52

You can see that the cat command works well but the apache command doesn't and we have a return.

You can restart the process with the command below

# supervisorctl restart all
cat-command: stopped
cat-command: started
apache2: ERROR (abnormal termination)

b) Monitor the processes via the web interface

The [inet_http_server] section is the one used to access the web interface. Remember that it contains should look like below but you can use another username and password

[inet_http_server] ; inet (TCP) server disabled by default
port=server-ip-address:9001 ; (ip_address:port specifier, *:port for all iface)
username=user ; (default is no username (open server))
password=123 ; (default is no password (open server))

Now you can open your web browser and enter your server ip address with the port as below

You can see that you can restart, stop or see the logs.

We have seen how we can monitor new process with supervisor which is can be useful for users who don't need to control the server itself but who only need to monitor if the process they need is launched or not so that, they will be able to restart it if need. When you add new processes, you need to use the command reread and update of supervisorctl command so that supervisor will add it in the current processes.