How to Setup Central Logging Server Using Rsyslog on Ubuntu 20.04

Rsyslog is an open-source utility for log processing. It permits the logging of data from different types of systems in a central repository. Rsyslog is a Syslog protocol with more extensions, features, and benefits.

What makes rsyslog so powerful? Rsyslog can deliver over one million messages per second to local destinations when limited processing is applied.

In this tutorial we learn how to install and configure rsyslog on Ubuntu 20.04.

Creating SSH Public and Private Key...

To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video

Creating SSH Public and Private Key in Linux / Ubuntu Rsyslog Features

Rsyslog came in 2004. and it extends syslog protocol with new features:

• RELP protocol support
• Buffered operation support
• Listening to TCP/UDP connections (with restrictions port, IP's)
• Support for loading a lot of modules ( for example module to support RELP protocol)
• Support for discard message that contains special configured rules

In Rsyslog, the config files remain the same as Syslog. That simply means you can copy a syslog.conf file directly into rsyslog.conf and it will work.

What are logs and syslog?

Before we go to the configuration process itself, let's see first what logs are used for? The log information can really help you to see what is exactly happened on your system, or what exactly is going on under the hood.

Let me give you an example: if suddenly your pc frizzed or maybe crashed or even downtime of your system and much more useful information that you cannot see in the usual way when error message display.

If you are known how to use logs properly, they can give you a lot of features and benefits for diagnostic information about the system itself.

By default, Linux uses the syslog daemon to record logs about how the system is running and then putting these logs into a central repository in: /var/log where we can examine and investigate them.
In simple terms of meaning everything from kernel events to user actions is logged by Linux, allowing you to see almost any action performed on your pc or servers.

In the Linux file system, there is one special directory for storing logs called /var/log. This directory contains all logs from the OS itself like: services, or various applications that running on the system.

Let's see how directories and structures of logs look like on fresh Linux ubuntu 20.04.

List /var/log

Directory where different sorts of logs stored are:

/var/log/syslog - Stores all startup messages, application startup messages etc. Practically stores all global system logs.
/var/log/cron - The Cron jobs are basically kind of scheduled and automated task created in the system, that runs periodically and repeatedly. You can see what this logs directory would store.
/var/log/kern.log - it stores kernel logs. No matter what logs they are. Event logs, errors, or warning logs.
/var/log/auth.log - Authentication logs
/var/log.boot.log - System boot logs
/var/log/mysql.d - Mysql logs
/var/log/httpd - Apache logs directory
/var/log/maillog - Mail server logs

Setup Rsyslog Centralized Log Server on Ubuntu 20.04

After a brief overview, what are logs, Syslog, and where rsyslog has a place, let us move on to the configuration process itself.

Prerequisites

• Two servers that running Ubuntu 20.04
• A static IP address: example 192.168.0.101 is needed to be configured on Rsyslog server machine and 192.168.0.102 is configured on Rsyslog client machine

Ubuntu 20.04 system has rsyslog installed by default which came with standard system packages.

You can check to see if it is running:

$ sudo systemctl status rsyslog
● rsyslog.service - System Logging Service
     Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-03-22 21:16:34 UTC; 12h ago
TriggeredBy: ● syslog.socket
       Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
   Main PID: 566 (rsyslogd)
      Tasks: 4 (limit: 2281)
     Memory: 2.7M
     CGroup: /system.slice/rsyslog.service
             └─566 /usr/sbin/rsyslogd -n -iNONE

Run the following command to check the Rsyslog version which is currently installed:

$ rsyslogd -v

Output:

rsyslogd  8.2001.0 (aka 2020.01) compiled with:
         PLATFORM:                               x86_64-pc-linux-gnu
         PLATFORM (lsb_release -d):
         FEATURE_REGEXP:                         Yes
         GSSAPI Kerberos 5 support:              Yes
         FEATURE_DEBUG (debug build, slow code): No
         32bit Atomic operations supported:      Yes
         64bit Atomic operations supported:      Yes
         memory allocator:                       system default
         Runtime Instrumentation (slow code):    No
         uuid support:                           Yes
         systemd support:                        Yes
         Config file:                            /etc/rsyslog.conf
         PID file:                               /run/rsyslogd.pid
         Number of Bits in RainerScript integers: 64

In case not installed or running, install rsyslog using the following commands:

$ sudo apt-get update
$ sudo apt-get install rsyslog

Now it's time to go to rsyslog.conf file, to uncomment and change some lines to run rsyslog service in server mode:

$ sudo nano /etc/rsyslog.conf

Uncomment these four lines that enable udp and tcp port binding:

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

The next step that we are gonna need to add is to create a new template. We need to create a new template for receiving remote messages. A template will give instructions to rsyslog server about how to store incoming syslog messages.

NOTE: Add template right before GLOBAL DIRECTIVES section:

$template remote-incoming-logs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" 
*.* ?remote-incoming-logs
& ~

The above line indicates that received logs will be processed and stored inside directory /var/log/. You can define in /etc/rsyslog.conf file where to rsyslog logs.

The file name contains %HOSTNAME% and %PROGRAMNAME% variables stand for client machine and client program name that produced the log message.

Configuration file for Rsyslog

Save this and than restart rsyslog service:

$ sudo systemctl restart rsyslog

Confirme that rsyslog service is listening on configured ports:

$ ss -tunelp | grep 514
udp    UNCONN  0       0                    0.0.0.0:514           0.0.0.0:*      ino:33591 sk:1 <->                                                             
udp    UNCONN  0       0                       [::]:514              [::]:*      ino:33592 sk:4 v6only:1 <->                                                    
tcp    LISTEN  0       25                   0.0.0.0:514           0.0.0.0:*      ino:33595 sk:7 <->                                                             
tcp    LISTEN  0       25                      [::]:514              [::]:*      ino:33596 sk:9 v6only:1 <->      

If you maybe have ufw firewall service, you need to allow rsyslog firewall port rules:

sudo ufw allow 514/tcp
sudo ufw allow 514/udp

To verify configuration, run the following command:

sudo rsyslogd -N1 -f /etc/rsyslog.conf

Configure Rsyslog on Client

Once this configuration of rsyslog server is done, the next step is to configure your rsyslog client machine to send logs to the remote rsyslog server.

As we go with rsyslog.conf file on a remote server, same will open this file on client-side with your favorite editor and edit some changes:

sudo nano /etc/rsyslog.conf

And allow preservation of FQDN: Alternatively, to have rsyslog send with the fully-qualified domain name (FQDN, such as system1.example.com) instead of simply the hostname (system1), use the directive:

$PreserveFQDN on

Add configured remote rsyslog server at the end.

 @192.168.0.101:514

Or you can enable to send logs over UDP. For TCP use @@ , instead of one

*.* @@192.168.0.101:514

For the end add these folowing variables in case when rsyslog server go down:

$ActionQueueFileName queue
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount -1

Then restart the rsyslog service:

sudo systemctl restart rsyslog

Verify the logs

After the configuration is completed on the client machine, we want to verify that everything went well.

Go to your Rsyslog server to verify the logs from your client machine:

$ ls /var/log/

You will find something like this:

List /var/log

In my case directory named obrad is the name of my client machine which I am currently using. We will enter this directory and see something like this:

List var/log/hostname

You should see all log files generated by Rsyslog.

You can check any of these logs with the following command: Let's for example inspect systemd.log.

$ tail -f /var/log/obrad/systemd.log

And you will see something like this:

2021-03-25T11:31:59+00:00 obrad systemd[1206]: Started VTE child process 42166 launched by gnome-terminal-server process 3186.
2021-03-25T11:32:28+00:00 obrad systemd[1206]: Created slice dbus\x2d:1.2\x2dorg.gnome.gedit.slice.
2021-03-25T11:32:28+00:00 obrad systemd[1206]: Started dbus-:[email protected].
2021-03-25T11:32:31+00:00 obrad systemd[1206]: dbus-:[email protected]: Succeeded.

As you can see I launched terminal and gedit text edit on my client machine and Rsyslog generated and display this in output.

Conclusion

In this tutorial, we learned how to configure the basic parameters for rsyslog using the client-server model on Ubuntu 20.04. The syslog-ng, FluentD, Logstash, GreyLog2, and Logagent, Filebeat are the other alternatives for Rsyslog.