All Linux distributions implement some kind of logging to the system. Inside logging files you will see short straightforward system messages which help administrators to see what is going on inside the system and what has been done, who did what. Also, you can see system service logs which explain to you how they behave, if there is any errors with services, etc.
Application can send its errors to:
- to its own log (like apache)
- rsyslog – which also can be used to centralize logging
- systemd-journald – which cannot provide centralized logging, but can send logs to rsyslog, which then can be used to centralize logs
The syslog protocol is a standard for logging system events and application events. The syslog protocol is separating messages to facility and severity codes. Both of these codes are explained in detail:
|0||kern||Messages from system kernel|
|1||user||Messages generated by user events|
|2||Messages from mail applications|
|3||daemon||Messages from system applications (services)|
|4||auth||Security and authentication messages|
|5||syslog||Messages from logging application itself|
|7||news||Messages from news application|
|8||uucp||Messages from Unix-to-Unix copy program|
|9||cron||Messages from cron job scheduler|
|10||authprv||Security and authentication messages|
Also, as we mentioned, each facility has a severity code with it. This puts each facility the severity or importance code that explains how serious the problem is, is it informational log message, is it critical, is it a warning, etc.
Here are seven severity codes syslog protocol is using:
|0||emerg||Event that causes system to be unstable|
|1||alert||Event that requires immediate attention|
|2||crit||Event that is critical but doesn’t require immediate attention|
|3||err||Error that allows system or application to continue|
|4||warning||Not-normal warning condition in the system or application|
|5||notice||Normal but significant message explaining bad condition|
|6||info||Informational message from the system|
|7||debug||Debugging message for developers|
So, if your syslog protocol generates facility mail and severity alert, then you know something horrific has happened with mail service. Another example, if you find that auth facility generated info severity code, then you know that something has happened with authentication on the system.
Viewing history of Linux logging
Over the years, there were multiple logging projects for Linux systems:
- sysklogd: Original syslog application which included two programs: syslogd to monitor for system and application events and klogd program to monitor for kernel events.
- syslogd-ng: Added advanced features to sysklogd
- rsyslog: Fast rsyslogd application that quickly became standard logging package for Linux distributions
- systemd-journald: Part of systemd application. Many Linux distributions are now using this application for logging. It does not use syslog protocol with facility and severity codes, but instead uses different way of logging (explained later)
We will discuss rsyslog and systemd–journald logging services
1. rsyslogd logging
rsyslog uses all of the features from the original syslog protocol. The main configuration file for rsyslogd is /etc/rsyslog.conf, and in some distributions all files within /etc/rsyslog.d/ directory. All files within that directory will define what events to listen for and how to handle them. The format for rsyslogd is same as syslog protocol. However, there is little tweak. If you define severity code with number 3, he will also log all severities that are higher than that (2,1, and 0). To log exact severity level, use =crit for example, and it will log only severity with crit code.
The action field is defining what rsyslogd should do with that syslog message. There are six options for that:
- Forward the message to regular file
- Pipe the message to an application
- Display message on terminal
- Send message to remote host
- Send message to a list of users
- Send message to all logged-in users
Sending log messages to a Log Server
Something we can do with all those log messages is to send them to remote central log server. This is called central logging. There is one or multiple servers that are accepting all log messages from hosts in the entire network. First, we need to configure logging client to send logs to the log server. To do this, we should edit /etc/rsyslogd.conf configuration file and go to the bottom, add line and save the file. Line will contain facility.severity part, and an action. Here, action is defined with special syntax:
- TCP|UDP: select either UDP or TCP protocols to transport your log messages to the central log server. Use @ sign to select UDP, and @@ to select TCP.
- [(z#)]: Square brackets indicate this is optional. z uses zlib compression to compress data and send over the network, # is compression level (1 to 9)
- HOST: FQDN of the central log server, for example logserver.company.com
- [PORT#]: Brackets indicate this is optional. This tells on which remote port to send these logs. On this port central server must listening for incoming connections.
On each client system, add something like this:
After you do this, reload the configuration file or restart rsyslogd service, and it should send all logs to that server.
Rotating log files
On busy servers log files are generated quickly, and they consume tons of storage space. To solve this problem, among pushing them to central logging server, we should install logrotate utility. The logrotate program will archive rsyslogd logs based on their time, it can compress files, delete, and mail log files.
The logrotate is usually run daily as cron job. The main configuration file for logrotate is /etc/logrotate.conf where you specify how long each log will be, whether you should log them weekly, how much logs to keep, etc.
When you open this file, you will notice two parts. First are global directives that affect logrotate as whole. The next part has settings for specific log files and they override global directives.
The include /etc/logrotate.d/ tells logrotate to use any configuration files within this directory for additional log rotation options.
|hourly||Log file is rotated hourly|
|daily||Log file is rotated daily|
|weekly X||Log file is rotated weekly on X day of the week|
|monthly||Log file is rotated first time logrotate is run|
|size X||Rotate logs based on their size, X triggers rotation|
|dateformat||Modify dateformat used when naming log files|
|missingok||If log is missing, do not send error message|
|notifempty||If log is empty, do not rotate this log file, continue on next log file|
Making log entries
We can also make our own log entries and it is done with logger command. Some popular options are -i to specify process ID of the program that created that log, -p to specify event priority, -t to specify tag to easily find message in the log file.
# logger This is test message # tail -5 /var/log/syslog Oct 15 20:20:11 arch themozak: This is test message
2. systemd-journald journaling
systemd-journald uses completely different method of storing event messages from syslog protocol. As we know, rsyslogd uses facility.severity options to filtrate how syslog stores log messages.
To see boot log entries from current and previous boots, starting at the oldest entry, type:
[aldin@arch ~]$ journalctl
To see boot log entry just for current boot, then add -b flag:
[aldin@arch ~]$ journalctl -b
To see only kernel messages (similar to dmesg messages), add -k flag, and -b for just this boot:
[aldin@arch ~]$ journalctl -k -b
To see all journal entries for given unit, then add -u flag with unit name (.service, .targer, .socket …)
[aldin@arch ~]$ journalctl -u haproxy.service
The systemd-journald service’s main configuration file is /etc/systemd/journald.conf configuration file. Inside this file, there are settings or directives that control how application works. Here are some commonly used directives for journald.conf:
|Storage=||Determines how systemd-journald stores event messages|
|Compress=||Set if journal files are compressed or not|
|ForwardToSyslog=||Messages are forwarded to other syslog program (rsyslogd for example)|
|ForwardToWall=||Messages are forwarded to all users|
|MaxFileSec=||Set time before journal file is rotated (archived)|
|RuntimeKeepFree=||Set amount of disk space systemd-journald must keep free for other disk usages when using volatile storage|
|RuntimeMaxFileSize=||Set amount of disk space systemd-journald journal files can consume when using volatile storage|
|RuntimeMaxUse=||Set amount of disk space systemd-journald can consume when using volatile storage|
|SystemKeepFree=||Set amount of disk space systemd-journald must keep free for other disk usages when using persistent storage|
|SystemMaxFileSize=||Set amount of disk space systemd-journald files can consume when using persistent storage|
|SystemMaxUse=||Set amount of disk space systemd-journald can consume when using persistent storage|
|SplitMode=||Divide journal file into multiple active files (one per user, and system journal file)|
Persistent, volatile and auto storage directive
As you noticed, Storage= directive inside /etc/systemd/journald.conf configuration file determines how systemd-journald stores event messages. There are four ways how it can be done:
- auto: Force systemd-journald to look at /var/log/journal/ directory and store event messages there. If directory does not exist, go to /run/log/journal/ directory and store them there (this directory is temporary and is deleted when system shuts down)
- persistent: Force systemd-journald to create /var/log/journal directory and store event messages there.
- volatile: Force systemd-journald to store event messages in temporary /run/log/journal/ directory.
- none: Event messages are not saved (discarded)
Looking at journal files
We can have multiple active journal files depending how systemd-journald is configured.
- Active system journal file is named system.journal.
- Active user journal files are named user-UID.journal.
- Archived journal files start with system or user-UID followed with @ and several letters and numbers.
Layering your logging
If we want, both systemd-journald and rsyslogd can work together to store log files. There are two ways to accomplish this:
- Journal client method: This allows syslog protocol to act as journal client, reading entries stored in journal files. For rsyslog, to configure this method go to /etc/rsyslog.conf main configuration file and uncomment imuxsock and imjournal modules under Modload directive.
- Forward to syslog method: This method uses /run/systemd/journal/syslog file. All event messages are forwarded to this file where syslog protocol can read them. To use this method, modify journal configuration file /etc/systemd/journald.conf and set ForwardToSyslog directive to yes.
Viewing journal message entries
systemd-journald does not store log files in clear text (ASCII) and cannot be read with cat or head/tail commands. These journal files are binary files that work similar to a database. However, there is journalctl utility that can read journal files.
# journalctl [OPTIONS] [MATCHES]
OPTIONS control how data is displayed, and there are plenty of them:
|-a||Display all data fields|
|-e||Go to the end of journal file and show entries|
|-k||Display only kernel entries|
|-n X||Show the most recent X number of journal entries|
|-r||Show last entries first|
|-S X||Show journal entries starting at date X|
|-U X||Show journal entries until date X is reached|
|-u unit||Show only journal entries for systemd unit or systemd units|
MATCHES filter what type of journal entries will be displayed. Here are the most used ones:
|OBJECT_PID=pid||Match only entries made by specified application process ID (PID)|
|PRIORITY=value||Match only entries with specified priority value |
emerg(0), alert(1), crit(2), err(3), warning(4), notice(5), info(6), debug(7)
|_HOSTNAME=host||Match only entries from this host|
|_SYSTEMD_UNIT=unit.type||Match only entries made by specified systemd unit (i.e network.socket)|
|_TRANSPORT=transport||Match only entries received by specified transport method|
|_UDEV_SYSNAME=dev||Match only entries received from specified device|
|_UID=userid||Match only entries made by specified user ID|
Maintain the Journal
Examine current disk usage by journal files with –disk-usage parameter:
# journalctl --disk-usage
Output will show how much disk space all journal files are using (both active and archived journal files). The journalctl can automatically clean up disk space with this:
# journalctl --vacuum-size=300MB # journalctl --vacuum-time
–vacuum-size removes journal files until the disk space consumed by journal files reaches designated size. In this case, it will remove all journal files until 300MB of files is on the drive.
Viewing different Journal files
If you have backed up journal files stored in other directory than the directories journalctl normally looks for journal files, those are /run/log/journal/ directory or /var/log/journal/ directory, then you can point him to that custom location in which you have journal files with -D /directory/path/ or –directory=/directory/path/. If the file you want to see has different name than system.journal or user-UID.journal, then tell systemctl with –file=filename, and if there is several files, use wildcards.
Also, you can merge two journal files with –m or —merge switches. This does not physically merge two files, but instead merges them logically inside journalctl so it can output results.
Making Journal entries
As with syslog protocol and logger command, we could generate our own event messages. With systemctl things are a bit different. To add journal entries to journal file, use systemd-cat tool.
# echo "This is test message" | systemd-cat # journalctl --no-pager | grep systemd-cat
If you set up imuxsock module, then logger can also make journal entries:
# logger "This is syslog test" # journalctl -r