Chapter 7 – Monitoring and Logging

1. Reading local logs

  • Look at default place for logging
  • Log files are located inside /var/log directory
  • On CentOS main log file is the messages log.
  • Under Debian and Ubuntu main log file is syslog

Take a look at messages log

Take a look at secure log

  • Log for sshd, sudo, PAM and others

Take a look at cron logs

  • As all these log files are clear-text, we can manipulate it with basic tools
# grep “themozak” messages | cut -d “ “ -f 1,3,5

Logrotate

  • Logrotate is used to periodically run and change old files for new ones.
  • Saves space, removes too old log files.
# logrotate -f /etc/logrotate.conf

It will rename old logs (put the current date) and create new ones with the old names.

  • Daemon logging to text files rsyslogd or syslog-ng
  • It reads the messages from imuxsock and imjournal and creates these logs.
  • Configuration for rsyslogd is inside /etc/rsyslog.conf

Logger

The logger command is used to write to the log directly.

It can be handy for testing purposes and shell scripts

Log levels


2. Using journalctl on systemd systems

  • Modern Linux distros do not rely on syslog files
  • Since Debian, Ubuntu and CentOS have systemd as an init system, they are grouped with a service called journald (systemd-journald.service)
  • This services acts as a solution for journaling
  • It tracks binary logs instead of text-based logs. Meaning it cant be opened with text parsers and editors. Instead, we use journalctl to read logs
# journalctl

Follow log as it is written (live):

# journalctl -f

Show only kernel logs:

# journalctl -k --no-pager

Show logs by time:

# journalctl --since=18:00 --until=18:03
# journalctl --since=”2019-10-08 19:19:00”
# journalctl --since=yesterday --until=now

Show systemd unit logs:

# journalctl -u chronyd

Show logs from the chronyd binary only

# journalctl /usr/sbin/chronyd

Use -x for verbose output:

Add specific matches to journalctl statements (format FIELD=VALUE)

# journalctl --since=yesterday _SYSTEMD_UNIT=sshd.service _PID=853
  • –since=yesterday to show all messages from yesterday
  • _SYSTEMD_UNIT=sshd.service to show logs generated by the systemd ssh unit,
  • _PID=853 to show and sift only those logs from PID 853

More on matches at systemd.journal-fields man page

Prioritize search output by level messages:

# journalctl -p err
  • Journalctl is configured in /etc/systemd/journal.conf
  • Logs are not persisted through reboots. Only good to query system since boot.

The files are in binary format and journald pulls various sources to create his log journals:

  1. Kernel log messages from /dev/kmsg
  2. Log messages from syslog libc
  3. Log messages from the Journal API imported into rsyslog 
  4. Stdout and stderr of service unit files
  5. Audit records from kernel audit subsystem

Use logger to save the logs (journalctl does not save results after reboot)

# logger -p cron.err

Limitations and configuration options

  • You can set limitation to how much space journald will use, governed in his conf file.
  • The options SystemMaxUse and RuntimeMaxUse are used for this. Default is 10% the size of filesystem and max is 4GB
  • The SystemKeepFree and RuntimeKeepFree options govern how much disk space journald leaves free. Default is 15%.
  • On CentOS, the log file vanishes upon a reboot. While it is live it lives inside /run/log/journal/ directory

3. Centralizing logging

  • Good idea, to centralize logs on one system, not separately store them in each machine
  • In the following examples, TLS wont work and logs will be streamed in plain text. Use HTTPS or tunneling if doing this inside production.

Remote logging rsyslog – UDP example

  • To enable logging to a remote machine with rsyslog, enable streaming to a remote location on your client (centos1), and enable receiving on your server (centos2)

Centos1

# sed -i ‘s/#*.* @@remote-host:514/*.* @192.168.100.102/g’ /etc/rsyslog.conf
  • Inside rsyslog.conf, there is part of code that explains forwarding. To change that, we replaced #*.* @@remote-host:514 with *.* @192.168.100.102
  • 514 is port number

# systemctl restart rsyslog

Centos2:

# sed -i ‘s/#$ModLoad imudp/$ModLoad imudp/g’ /etc/rsyslog.conf
# sed -i ‘s/#$UDPServerRun 514/$UDPServerRun 514/g’
  • Literally removing comments

See it working

centos2 :

# tcpdump port 514 -i eth1

centos1:

# logger -p syslog.info “Random message for test”

centos2:

# tail -5 /var/log/messages
  • The Random message for test will show
  • We successfully migrated log files that are happening at centos1, to centos2

Remote logging with rsyslog – TCP Example

  • Previous example covered UDP, which means server won’t confirm that he got the message log from centos1 at all.
  • With TCP connection, the syslog servers communicate with each other to establish a connection first, and then send message logs.

centos1:

  • Replace @ in your destination address with @@
# sed -i ‘s/*.* @192.168.100.102/*.* @@192.168.100.102’ /etc/rsyslog.conf
# systemctl restart rsyslog
  • Our client is now set, but no logs can be sent until the connection is established

centos2:

# sed -i ‘s/#$ModLoad imtcp/$ModLoad imtcp/g’ /etc/rsyslog.conf
# sed -i ‘s/#$InputTCPServerRun 514/$InputTCPServerRun 514/g’ /etc/rsyslog.conf
# systemctl restart rsyslog
# tcpdump port 514 -i eth1

centos1:

# logger -p syslog.err “Random test message”

centos2:

# tail -10 /var/log/messages
  • The log will be shown from centos1

Remote logging with journald

  • The systemd-journal-remote command allows you to receive journal messages over a network
  • It is fairly new addition to systemd suite. It is not yet available on CentOS systems

debian1

# sed -i ‘s/# URL=/URL=http:\/\/192.168.100.102/g’ /etc/systemd/journal-upload.conf

debian2

# systemctl edit systemd-journal-remote

Add the following lines to an empty file:

[Service]
ExecStart=
ExecStart=/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote
  • Save and exit

We need remote folder location, make sure it has the permissions before restarting the service:

# mkdir -p /var/log/journal/remote
# chown systemd-journal-remote /var/log/journal/remote
# systemctl restart systemd-journal-remote

debian1

Start the service:

# systemctl restart systemd-journal-upload

debian2

# journalctl -D /var/log/journal/remote/ -f

debian1

# logger -p syslog.err “Debian1 logs, on debian2”

How it works

  • We opened a listener for logs in both the syslog and journald solutions.
  • A network port is opened on both boxes, 

4. Local resource measuring tools:

  • Handy to know what is system doing right now.
  • free, top, netdata, htop
# yum installl epel-release -y
# yum install htop -y

top

  • Installed to all unix systems, 
  • must open each time you want to debug something
# top

1st row – System info

  • system time, how long the box is up – 17:09 up 8 min
  • logged users – 1 user
  • the load average (1,5,15 minutes) 

2nd row – Task info

  • number of tasks (task is number of processes that is running now) (total, running, sleeping, stopped, zombie)

3rd row – CPU usage info

  • us, user = time running un-niced user processes
  • sy, system = time rnning kernel processes
  • ni, nice = time running niced user processes
  • id, idle = time spent in the kernel idle handler
  • wa, IO-wait = time waiting for IO completion
  • hi = time spent servicing hardware interrupts
  • si = time spent servicing software interrupts
  • st = time stolen from this vm by the hypervisor

4th row Volatile memory information (RAM)

  • KiB Mem: in digits, amount of RAM available to your system, broken down into the total, free, user, and buffer/cach memory

5th row SWAP informaton (disk memory)

  • Also in digits, this is the amount of Swap available, brokendown once more.
  • PID = PID of the task
  • USER = User running the process
  • PR (Priority of task) higher PR is prioritized)
  • NI – nice value of a task (minus values have higher priority)
  • VIRT – vitual memory used by the task (all memory, including the likes of shared libraries)
  • RES – non-swapped physical memory used by the task
  • SHR – too technical
  • S – state of the process (running, sleeping, zombie,..)
  • %CPU – how much cpu time a task has used since the last refresh
  • %MEM – how much available physical memory a task is using
  • TIME+ – total CPU time the task has used since starting
  • COMMAND – name of the task

free

  • Great tool to see how busy system is.
  • Quickest way of finding out how much memory is being used
  • It has fewer options than top
  • The flags (arguments) change the output of the command
  • -h for human-readable output, -t to add Total line (sums Mem and Swap values)
  • available – tells you how much physical memory is available before the system starts swapping.

htop

  • Same as top, but the output is eye-friendly
  • Colors, nice alignment, modern terminal look
  • F5 for tree view 
  • Change what is shown in columns by pressing F2 Setup menu. 
  • Changes are persisted to disk in the form of a configuration file (~/.config/htop/htoprc)

NetData

  • collector of all system information it can get.
  • It uses a centralized server for logging some data (hostnames). Check for internal security policy if you want to use this tool.

Installing Netdata

Enable backports repository with:

# echo “deb http://ftp.debian.org/debian stretch-backports main” | sudo tee -a /etc/apt/sources.list
# apt update
# apt install netdata -y
  • By default, installed to listen on localhost

How these tools work

top – queries the kernel to gather information about the system. This makes it fast and on point. Incredibly light. It has been around since 1980s.

free – takes a look at the values available in /proc/meminfo. You can look at info at that location, or just use free just for nice looks.

htop – queries the system in much the same way as top. Htop uses the ncurses library to display itself. It has been around for 14+ years.

NetData – uses various sources (and uses custom plugins) to collect data every second.


5. Local monitoring tools

  • Tools that look at the historic data on your system

atop, sar, vmstat

Install the tools

# yum install epel-release -y
# yum install atop sysstat -y

atop

  • atop (advanced system and process monitoring)
# atop
  • shows system and process activity since bot
  • results are late 10 seconds
  • used to store data not just of the current boot, but periodically

Enable atop service

# systemctl enable atop --now
  • Logs from before are stored to /var/log/atop in binary format
  • Those files can be opened and replayed at future data in case we want to check what happened to the system.
# ls /var/log/atop/
atop_2018512 daily.log

To read a file

# atop -r <logname>

atop is restarted at midnight by a cron job

# cat /etc/cron.d/atop

To configure atop

# cat /etc/sysconfig/atop
  • When atop is started (using systemd) the /usr/share/atop/atop.daily script is triggered, using options from sysconfig at /etc/sysconfig/

sar

  • sar is a way of reading system information, but it also allows you to read historic information
  • Enabled with a systemctl command (# systemctl enable –now sysstat)
  • By default, sar is executed every 10 minutes to grab sys info. Then, it has daily summary created at 23:53

Check that file with:

# cat /etc/cron.d/sysstat

Open sar file

# sar -f /var/log/sa/sa14

Filter the file and open it:

# sar -f /var/log/sa/sa14 -s 19:00:00 -e 20:00:00

To configure sar

# cat /etc/sysconfig/sysstat

vmstat

  • great way to report on memory statistics

Leave a Reply