1. Reading local logs
- Look at default place for logging
- Log files are located inside /var/log directory
- On CentOS main log file is the messages log.
- Under Debian and Ubuntu main log file is syslog
Take a look at messages log
Take a look at secure log
- Log for sshd, sudo, PAM and others
Take a look at cron logs
- As all these log files are clear-text, we can manipulate it with basic tools
# grep “themozak” messages | cut -d “ “ -f 1,3,5
- Logrotate is used to periodically run and change old files for new ones.
- Saves space, removes too old log files.
# logrotate -f /etc/logrotate.conf
It will rename old logs (put the current date) and create new ones with the old names.
- Daemon logging to text files rsyslogd or syslog-ng
- It reads the messages from imuxsock and imjournal and creates these logs.
- Configuration for rsyslogd is inside /etc/rsyslog.conf
The logger command is used to write to the log directly.
It can be handy for testing purposes and shell scripts
2. Using journalctl on systemd systems
- Modern Linux distros do not rely on syslog files
- Since Debian, Ubuntu and CentOS have systemd as an init system, they are grouped with a service called journald (systemd-journald.service)
- This services acts as a solution for journaling
- It tracks binary logs instead of text-based logs. Meaning it cant be opened with text parsers and editors. Instead, we use journalctl to read logs
Follow log as it is written (live):
# journalctl -f
Show only kernel logs:
# journalctl -k --no-pager
Show logs by time:
# journalctl --since=18:00 --until=18:03 # journalctl --since=”2019-10-08 19:19:00” # journalctl --since=yesterday --until=now
Show systemd unit logs:
# journalctl -u chronyd
Show logs from the chronyd binary only
# journalctl /usr/sbin/chronyd
Use -x for verbose output:
Add specific matches to journalctl statements (format FIELD=VALUE)
# journalctl --since=yesterday _SYSTEMD_UNIT=sshd.service _PID=853
- –since=yesterday to show all messages from yesterday
- _SYSTEMD_UNIT=sshd.service to show logs generated by the systemd ssh unit,
- _PID=853 to show and sift only those logs from PID 853
More on matches at systemd.journal-fields man page
Prioritize search output by level messages:
# journalctl -p err
- Journalctl is configured in /etc/systemd/journal.conf
- Logs are not persisted through reboots. Only good to query system since boot.
The files are in binary format and journald pulls various sources to create his log journals:
- Kernel log messages from /dev/kmsg
- Log messages from syslog libc
- Log messages from the Journal API imported into rsyslog
- Stdout and stderr of service unit files
- Audit records from kernel audit subsystem
Use logger to save the logs (journalctl does not save results after reboot)
# logger -p cron.err
Limitations and configuration options
- You can set limitation to how much space journald will use, governed in his conf file.
- The options SystemMaxUse and RuntimeMaxUse are used for this. Default is 10% the size of filesystem and max is 4GB
- The SystemKeepFree and RuntimeKeepFree options govern how much disk space journald leaves free. Default is 15%.
- On CentOS, the log file vanishes upon a reboot. While it is live it lives inside /run/log/journal/ directory
3. Centralizing logging
- Good idea, to centralize logs on one system, not separately store them in each machine
- In the following examples, TLS wont work and logs will be streamed in plain text. Use HTTPS or tunneling if doing this inside production.
Remote logging rsyslog – UDP example
- To enable logging to a remote machine with rsyslog, enable streaming to a remote location on your client (centos1), and enable receiving on your server (centos2)
# sed -i ‘s/#*.* @@remote-host:514/*.* @192.168.100.102/g’ /etc/rsyslog.conf
- Inside rsyslog.conf, there is part of code that explains forwarding. To change that, we replaced #*.* @@remote-host:514 with *.* @192.168.100.102
- 514 is port number
# systemctl restart rsyslog
# sed -i ‘s/#$ModLoad imudp/$ModLoad imudp/g’ /etc/rsyslog.conf
# sed -i ‘s/#$UDPServerRun 514/$UDPServerRun 514/g’
- Literally removing comments
See it working
# tcpdump port 514 -i eth1
# logger -p syslog.info “Random message for test”
# tail -5 /var/log/messages
- The Random message for test will show
- We successfully migrated log files that are happening at centos1, to centos2
Remote logging with rsyslog – TCP Example
- Previous example covered UDP, which means server won’t confirm that he got the message log from centos1 at all.
- With TCP connection, the syslog servers communicate with each other to establish a connection first, and then send message logs.
- Replace @ in your destination address with @@
# sed -i ‘s/*.* @192.168.100.102/*.* @@192.168.100.102’ /etc/rsyslog.conf
# systemctl restart rsyslog
- Our client is now set, but no logs can be sent until the connection is established
# sed -i ‘s/#$ModLoad imtcp/$ModLoad imtcp/g’ /etc/rsyslog.conf
# sed -i ‘s/#$InputTCPServerRun 514/$InputTCPServerRun 514/g’ /etc/rsyslog.conf
# systemctl restart rsyslog
# tcpdump port 514 -i eth1
# logger -p syslog.err “Random test message”
# tail -10 /var/log/messages
- The log will be shown from centos1
Remote logging with journald
- The systemd-journal-remote command allows you to receive journal messages over a network
- It is fairly new addition to systemd suite. It is not yet available on CentOS systems
# sed -i ‘s/# URL=/URL=http:\/\/192.168.100.102/g’ /etc/systemd/journal-upload.conf
# systemctl edit systemd-journal-remote
Add the following lines to an empty file:
[Service] ExecStart= ExecStart=/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote
- Save and exit
We need remote folder location, make sure it has the permissions before restarting the service:
# mkdir -p /var/log/journal/remote
# chown systemd-journal-remote /var/log/journal/remote
# systemctl restart systemd-journal-remote
Start the service:
# systemctl restart systemd-journal-upload
# journalctl -D /var/log/journal/remote/ -f
# logger -p syslog.err “Debian1 logs, on debian2”
How it works
- We opened a listener for logs in both the syslog and journald solutions.
- A network port is opened on both boxes,
4. Local resource measuring tools:
- Handy to know what is system doing right now.
- free, top, netdata, htop
# yum installl epel-release -y
# yum install htop -y
- Installed to all unix systems,
- must open each time you want to debug something
1st row – System info
- system time, how long the box is up – 17:09 up 8 min
- logged users – 1 user
- the load average (1,5,15 minutes)
2nd row – Task info
- number of tasks (task is number of processes that is running now) (total, running, sleeping, stopped, zombie)
3rd row – CPU usage info
- us, user = time running un-niced user processes
- sy, system = time rnning kernel processes
- ni, nice = time running niced user processes
- id, idle = time spent in the kernel idle handler
- wa, IO-wait = time waiting for IO completion
- hi = time spent servicing hardware interrupts
- si = time spent servicing software interrupts
- st = time stolen from this vm by the hypervisor
4th row Volatile memory information (RAM)
- KiB Mem: in digits, amount of RAM available to your system, broken down into the total, free, user, and buffer/cach memory
5th row SWAP informaton (disk memory)
- Also in digits, this is the amount of Swap available, brokendown once more.
- PID = PID of the task
- USER = User running the process
- PR (Priority of task) higher PR is prioritized)
- NI – nice value of a task (minus values have higher priority)
- VIRT – vitual memory used by the task (all memory, including the likes of shared libraries)
- RES – non-swapped physical memory used by the task
- SHR – too technical
- S – state of the process (running, sleeping, zombie,..)
- %CPU – how much cpu time a task has used since the last refresh
- %MEM – how much available physical memory a task is using
- TIME+ – total CPU time the task has used since starting
- COMMAND – name of the task
- Great tool to see how busy system is.
- Quickest way of finding out how much memory is being used
- It has fewer options than top
- The flags (arguments) change the output of the command
- -h for human-readable output, -t to add Total line (sums Mem and Swap values)
- available – tells you how much physical memory is available before the system starts swapping.
- Same as top, but the output is eye-friendly
- Colors, nice alignment, modern terminal look
- F5 for tree view
- Change what is shown in columns by pressing F2 Setup menu.
- Changes are persisted to disk in the form of a configuration file (~/.config/htop/htoprc)
- collector of all system information it can get.
- It uses a centralized server for logging some data (hostnames). Check for internal security policy if you want to use this tool.
Enable backports repository with:
# echo “deb http://ftp.debian.org/debian stretch-backports main” | sudo tee -a /etc/apt/sources.list
# apt update
# apt install netdata -y
- By default, installed to listen on localhost
How these tools work
top – queries the kernel to gather information about the system. This makes it fast and on point. Incredibly light. It has been around since 1980s.
free – takes a look at the values available in /proc/meminfo. You can look at info at that location, or just use free just for nice looks.
htop – queries the system in much the same way as top. Htop uses the ncurses library to display itself. It has been around for 14+ years.
NetData – uses various sources (and uses custom plugins) to collect data every second.
5. Local monitoring tools
- Tools that look at the historic data on your system
atop, sar, vmstat
Install the tools
# yum install epel-release -y
# yum install atop sysstat -y
- atop (advanced system and process monitoring)
- shows system and process activity since bot
- results are late 10 seconds
- used to store data not just of the current boot, but periodically
Enable atop service
# systemctl enable atop --now
- Logs from before are stored to /var/log/atop in binary format
- Those files can be opened and replayed at future data in case we want to check what happened to the system.
# ls /var/log/atop/ atop_2018512 daily.log
To read a file
# atop -r <logname>
atop is restarted at midnight by a cron job
# cat /etc/cron.d/atop
To configure atop
# cat /etc/sysconfig/atop
- When atop is started (using systemd) the /usr/share/atop/atop.daily script is triggered, using options from sysconfig at /etc/sysconfig/
- sar is a way of reading system information, but it also allows you to read historic information
- Enabled with a systemctl command (# systemctl enable –now sysstat)
- By default, sar is executed every 10 minutes to grab sys info. Then, it has daily summary created at 23:53
Check that file with:
# cat /etc/cron.d/sysstat
Open sar file
# sar -f /var/log/sa/sa14
Filter the file and open it:
# sar -f /var/log/sa/sa14 -s 19:00:00 -e 20:00:00
To configure sar
# cat /etc/sysconfig/sysstat
- great way to report on memory statistics