Understanding Log and Journal files

All Linux distributions implement some kind of logging to the system. Inside logging files you will see short straightforward system messages which help administrators to see what is going on inside the system and what has been done, who did what. Also, you can see system service logs which explain to you how they behave, if there is any errors with services, etc.

Application can send its errors to:

  1. STDERR
  2. to its own log (like apache)
  3. rsyslog – which also can be used to centralize logging
  4. systemd-journald – which cannot provide centralized logging, but can send logs to rsyslog, which then can be used to centralize logs

syslog protocol

The syslog protocol is a standard for logging system events and application events. The syslog protocol is separating messages to facility and severity codes. Both of these codes are explained in detail:

0kernMessages from system kernel
1userMessages generated by user events
2mailMessages from mail applications
3daemonMessages from system applications (services)
4authSecurity and authentication messages
5syslogMessages from logging application itself
6lprPrinter messages
7newsMessages from news application
8uucpMessages from Unix-to-Unix copy program
9cronMessages from cron job scheduler
10authprvSecurity and authentication messages

Also, as we mentioned, each facility has a severity code with it. This puts each facility the severity or importance code that explains how serious the problem is, is it informational log message, is it critical, is it a warning, etc.

Here are seven severity codes syslog protocol is using:

0emergEvent that causes system to be unstable
1alertEvent that requires immediate attention
2critEvent that is critical but doesn’t require immediate attention
3errError that allows system or application to continue
4warningNot-normal warning condition in the system or application
5noticeNormal but significant message explaining bad condition
6infoInformational message from the system
7debugDebugging message for developers

So, if your syslog protocol generates facility mail and severity alert, then you know something horrific has happened with mail service. Another example, if you find that auth facility generated info severity code, then you know that something has happened with authentication on the system.

Viewing history of Linux logging

Over the years, there were multiple logging projects for Linux systems:

  1. sysklogd: Original syslog application which included two programs: syslogd to monitor for system and application events and klogd program to monitor for kernel events.
  2. syslogd-ng: Added advanced features to sysklogd
  3. rsyslog: Fast rsyslogd application that quickly became standard logging package for Linux distributions
  4. systemd-journald: Part of systemd application. Many Linux distributions are now using this application for logging. It does not use syslog protocol with facility and severity codes, but instead uses different way of logging (explained later)

We will discuss rsyslog and systemdjournald logging services

1. rsyslogd logging

rsyslog uses all of the features from the original syslog protocol. The main configuration file for rsyslogd is /etc/rsyslog.conf, and in some distributions all files within /etc/rsyslog.d/ directory. All files within that directory will define what events to listen for and how to handle them. The format for rsyslogd is same as syslog protocol. However, there is little tweak. If you define severity code with number 3, he will also log all severities that are higher than that (2,1, and 0). To log exact severity level, use =crit for example, and it will log only severity with crit code.

The action field is defining what rsyslogd should do with that syslog message. There are six options for that:

  1. Forward the message to regular file
  2. Pipe the message to an application
  3. Display message on terminal
  4. Send message to remote host
  5. Send message to a list of users
  6. Send message to all logged-in users

Sending log messages to a Log Server

Something we can do with all those log messages is to send them to remote central log server. This is called central logging. There is one or multiple servers that are accepting all log messages from hosts in the entire network. First, we need to configure logging client to send logs to the log server. To do this, we should edit /etc/rsyslogd.conf configuration file and go to the bottom, add line and save the file. Line will contain facility.severity part, and an action. Here, action is defined with special syntax:

TCP|UDP[(z#)]HOST:[PORT#]
  • TCP|UDP: select either UDP or TCP protocols to transport your log messages to the central log server. Use @ sign to select UDP, and @@ to select TCP.
  • [(z#)]: Square brackets indicate this is optional. z uses zlib compression to compress data and send over the network, # is compression level (1 to 9)
  • HOST: FQDN of the central log server, for example logserver.company.com
  • [PORT#]: Brackets indicate this is optional. This tells on which remote port to send these logs. On this port central server must listening for incoming connections.

On each client system, add something like this:

*.* @@(z6)centralserver.company.com:7750

After you do this, reload the configuration file or restart rsyslogd service, and it should send all logs to that server.

Rotating log files

On busy servers log files are generated quickly, and they consume tons of storage space. To solve this problem, among pushing them to central logging server, we should install logrotate utility. The logrotate program will archive rsyslogd logs based on their time, it can compress files, delete, and mail log files.

The logrotate is usually run daily as cron job. The main configuration file for logrotate is /etc/logrotate.conf where you specify how long each log will be, whether you should log them weekly, how much logs to keep, etc.

When you open this file, you will notice two parts. First are global directives that affect logrotate as whole. The next part has settings for specific log files and they override global directives.

The include /etc/logrotate.d/ tells logrotate to use any configuration files within this directory for additional log rotation options.

hourlyLog file is rotated hourly
dailyLog file is rotated daily
weekly XLog file is rotated weekly on X day of the week
monthlyLog file is rotated first time logrotate is run
size XRotate logs based on their size, X triggers rotation
dateformatModify dateformat used when naming log files
missingokIf log is missing, do not send error message
notifemptyIf log is empty, do not rotate this log file, continue on next log file

Making log entries

We can also make our own log entries and it is done with logger command. Some popular options are -i to specify process ID of the program that created that log, -p to specify event priority, -t to specify tag to easily find message in the log file.

# logger This is test message
# tail -5 /var/log/syslog
Oct 15 20:20:11 arch themozak: This is test message

2. systemd-journald journaling

systemd-journald uses completely different method of storing event messages from syslog protocol. As we know, rsyslogd uses facility.severity options to filtrate how syslog stores log messages.

To see boot log entries from current and previous boots, starting at the oldest entry, type:

[aldin@arch ~]$ journalctl

To see boot log entry just for current boot, then add -b flag:

[aldin@arch ~]$ journalctl -b

To see only kernel messages (similar to dmesg messages), add -k flag, and -b for just this boot:

[aldin@arch ~]$ journalctl -k -b

To see all journal entries for given unit, then add -u flag with unit name (.service, .targer, .socket …)

[aldin@arch ~]$ journalctl -u haproxy.service

Configuring systemd-journald

The systemd-journald service’s main configuration file is /etc/systemd/journald.conf configuration file. Inside this file, there are settings or directives that control how application works. Here are some commonly used directives for journald.conf:

Storage=Determines how systemd-journald stores event messages
Compress=Set if journal files are compressed or not
ForwardToSyslog=Messages are forwarded to other syslog program (rsyslogd for example)
ForwardToWall=Messages are forwarded to all users
MaxFileSec=Set time before journal file is rotated (archived)
RuntimeKeepFree=Set amount of disk space systemd-journald must keep free for other disk usages when using volatile storage
RuntimeMaxFileSize=Set amount of disk space systemd-journald journal files can consume when using volatile storage
RuntimeMaxUse=Set amount of disk space systemd-journald can consume when using volatile storage
SystemKeepFree=Set amount of disk space systemd-journald must keep free for other disk usages when using persistent storage
SystemMaxFileSize=Set amount of disk space systemd-journald files can consume when using persistent storage
SystemMaxUse=Set amount of disk space systemd-journald can consume when using persistent storage
SplitMode=Divide journal file into multiple active files (one per user, and system journal file)

Persistent, volatile and auto storage directive

As you noticed, Storage= directive inside /etc/systemd/journald.conf configuration file determines how systemd-journald stores event messages. There are four ways how it can be done:

  1. auto: Force systemd-journald to look at /var/log/journal/ directory and store event messages there. If directory does not exist, go to /run/log/journal/ directory and store them there (this directory is temporary and is deleted when system shuts down)
  2. persistent: Force systemd-journald to create /var/log/journal directory and store event messages there.
  3. volatile: Force systemd-journald to store event messages in temporary /run/log/journal/ directory.
  4. none: Event messages are not saved (discarded)

Looking at journal files

We can have multiple active journal files depending how systemd-journald is configured.

  • Active system journal file is named system.journal.
  • Active user journal files are named user-UID.journal.
  • Archived journal files start with system or user-UID followed with @ and several letters and numbers.

Layering your logging

If we want, both systemd-journald and rsyslogd can work together to store log files. There are two ways to accomplish this:

  1. Journal client method: This allows syslog protocol to act as journal client, reading entries stored in journal files. For rsyslog, to configure this method go to /etc/rsyslog.conf main configuration file and uncomment imuxsock and imjournal modules under Modload directive.
  2. Forward to syslog method: This method uses /run/systemd/journal/syslog file. All event messages are forwarded to this file where syslog protocol can read them. To use this method, modify journal configuration file /etc/systemd/journald.conf and set ForwardToSyslog directive to yes.

Viewing journal message entries

systemd-journald does not store log files in clear text (ASCII) and cannot be read with cat or head/tail commands. These journal files are binary files that work similar to a database. However, there is journalctl utility that can read journal files.

# journalctl [OPTIONS] [MATCHES]

OPTIONS control how data is displayed, and there are plenty of them:

-a Display all data fields
-eGo to the end of journal file and show entries
-kDisplay only kernel entries
-n XShow the most recent X number of journal entries
-rShow last entries first
-S XShow journal entries starting at date X
-U XShow journal entries until date X is reached
-u unitShow only journal entries for systemd unit or systemd units

MATCHES filter what type of journal entries will be displayed. Here are the most used ones:

OBJECT_PID=pidMatch only entries made by specified application process ID (PID)
PRIORITY=valueMatch only entries with specified priority value
emerg(0), alert(1), crit(2), err(3), warning(4), notice(5), info(6), debug(7)
_HOSTNAME=hostMatch only entries from this host
_SYSTEMD_UNIT=unit.typeMatch only entries made by specified systemd unit (i.e network.socket)
_TRANSPORT=transportMatch only entries received by specified transport method
_UDEV_SYSNAME=devMatch only entries received from specified device
_UID=useridMatch only entries made by specified user ID

Maintain the Journal

Examine current disk usage by journal files with –disk-usage parameter:

# journalctl --disk-usage

Output will show how much disk space all journal files are using (both active and archived journal files). The journalctl can automatically clean up disk space with this:

# journalctl --vacuum-size=300MB
# journalctl --vacuum-time

–vacuum-size removes journal files until the disk space consumed by journal files reaches designated size. In this case, it will remove all journal files until 300MB of files is on the drive.

Viewing different Journal files

If you have backed up journal files stored in other directory than the directories journalctl normally looks for journal files, those are /run/log/journal/ directory or /var/log/journal/ directory, then you can point him to that custom location in which you have journal files with -D /directory/path/ or –directory=/directory/path/. If the file you want to see has different name than system.journal or user-UID.journal, then tell systemctl with –file=filename, and if there is several files, use wildcards.

Also, you can merge two journal files with –m or —merge switches. This does not physically merge two files, but instead merges them logically inside journalctl so it can output results.

Making Journal entries

As with syslog protocol and logger command, we could generate our own event messages. With systemctl things are a bit different. To add journal entries to journal file, use systemd-cat tool.

# echo "This is test message" | systemd-cat
# journalctl --no-pager | grep systemd-cat

If you set up imuxsock module, then logger can also make journal entries:

# logger "This is syslog test" 
# journalctl -r

This Post Has One Comment

Leave a Reply