Chapter 4 – Services and Daemons – LinuxWheel

Introduction:

systemd manages services and daemons on your machine
Before that job was done by init, now all distros use systemd
Systemd is the first process, and it has process ID of 1
All processes have systemd as a parent

Centos adopted systemd with version 7
Ubuntu adopted systemd with version 15.04
Debian adopted systemd with Jessie 8

1. Determining running services:

Good idea to check what software is running on each machine
To find out, modern systems will use systemctl to do that.

systemctl stands for system control
It is a software that manages all other software
systemctl is the main control mechanism for any systemd system.
systemctl is used to query systemd

Show systemd units:

# systemctl

Shows every unit systemd is aware of
Very noisy, not clean

Show only services and their status:

# systemctl list-units --type service

Show only running services and their status:

# systemctl list-units --type service --state running

Show only running services and their status: (cleaner output)

# systemctl list-units --type service --state running --no-legend

Show only services that were loaded, done the job, and exited:

# systemctl list-units --type service --state exited

Show only services that have failed:

# systemctl list-units --type service --state failed

Option –no-pager shows stdout in clear ASCII. An option used for scripting

2. List installed services:

# systemctl list-unit-files --type service

Options are: static, enabled, disabled.
This is a list of all services systemd knows about on your machine!

Enabled services are running
Disabled services are not running
Static services exist, they are not enabled, and cannot be enabled because they do not have [Install] section of its unit file.

Service files, and all types of unit file, exist in /usr/lib/systemd/system

From this location, files are symlinked into runlevel directories at /etc/systemd/system/

**targets** are basically different runlevels of a system
The most used services are inside **multi-user.target.wants** directory
In there we have services required for a multi-user session

Each line represents a service, and their symlink to location on disk.

List static services:

# systemctl --no-pager list-unit-files --type service --state static

Look at service config files:

# systemctl cat sshd-keygen.service

Insert any service name
The following image shows output of cat command

Since sshd-keygen.service is PartOf=sshd.service sshd.socket, it does not need to be enabled, and therefore is STATIC service, waiting for parent service to run.

# systemctl cat sshd.service

Output is showing that sshd.service wants sshd-keygen.service mentioned above.
Therefore, sshd-keygen.service is static, it depends on sshd.service and does not need to be enabled on its own (it is static, cannot be enabled by itself)

3. Starting and stopping services:

Stop a service:

# systemctl stop servicename

Start a service:

# systemctl start servicename

Check service status:

# systemctl status servicename

What systemd does when start/stop/reload command is used:

When we instruct systemd to stop/start/reload a service, it actually runs the ExecStart, ExecReload or ExecStop portions of its unit file:

Check unit file for a service:

# systemctl cat httpd

See service status:

# systemctl status httpd

To run service at each boot:

# systemctl enable servicename

Disable a service:

# systemctl disable servicename

By enabling/disabling service, we create symlink from default location of service’s unit file to the multi-user.target.wants directory.
On boot, systemd will check multi-user.target.wants directory and start the services it finds there

Start and Enable service at the same time:

# systemctl enable --now servicename

Command equivalent to systemctl start servicename and systemctl enable servicename.

4. Common services you might see:

# less systemctl list-unit-files --type service

Search with “/” to search the list.

auditd.service

“The Linux Audit Daemon”
Used to write audit records of your system
Stores logs at: /var/log/audit/ directory

chronyd.service

Responsible for keeping your system clock correct

crond.service

Responsible for scheduled commands and their run-time

lvm2-*.service

Collection of lvm2 service files
Used to maintain and monitor Logical volume Manager (LVM) setups.
Most servers will use LVM for their filesystem and volume management needs.

NetworkManager.service

Service responsible for networking on your system.

nfs.service

Suite of services for management of Network File System (NFS) interfaces and devices
Well maintained by distribution producers

prostifx.service

Default service on Red Hat, CentOS and Fedora.
On others it might be Exim mail server

rsyslog.service

rsyslog or syslog-ng will be probably installed on your system.
These are system log daemons
Responsible for writing logs to text files (/var/log

sshd.service

Service for SSH

systemd-journald.service

Competitor to syslog
journald is the logging daemon managed by systemd

systemd-logind.service

logind manages user logins to the systems of which you are an administrator

These services and others you are running make up the OS

Default services are disabled as part of server hardening tasks.

See how long system boots:

# systemd-analyze

See which process takes most of the booot time:

# systemd-analyze blame

5. Understanding service unit files:

Throughout this example, we will use sshd service.

Several options are available and more are being added with every systemd release

Read systemd.service and systemd.unit man pages

To understand this file, break it into parts [Unit] [Service] and [Install]

[Unit]

Area where generic entries live
It is the Main section of the unit file.

Description=OpenSSH server daemon

Short brief or description of a service

Documentation=man:sshd man:sshd_config

Shows where to look for documentation man pages

After= and Before=network.target sshd-keygen.service

Shows which services need to start after or before this service.
In this example, network service and sshd-keygen needs to start before sshd.service

Wants=sshd-keygen.service

Services which would be nice to trigger, prior to this service
Wants= is less-mandated version of Requires=.
If Wants= service fails to start, the parent service will give it a try to start
If Requires= service fails to start, the parent service will not start either

[Service]

This block is specific to service options

Type=notify

Explains process start up type – here it says notify
Notify means systemd expects the daemon to send a notification message once its finished starting up. Only when this notification is received, systemd continues starting services that depend on this one.

EnvironmentFile=/etc/sysconfig/sshd

This setting is used to point the service to the file that contains its environment variables.
For sshd, this contains one option/argument shown in ExecStart setting.

ExecStart=/usr/bin/sshd -D $OPTIONS

It gives us the command thats run
$OPTION value read from the EnvironmentFile line!

ExecReload=/bin/kill -HUP $MAINPID

Tells systemd what should be run if the systemctl reload sshd command is run
-HUP (hangup) = we are sending HUP signal to the sshd PID

KillMode=process

Setting specifies how processes of this unit should be killed.
Here, process means only the main process itself is killed by systemd.

Restart=on-failure

How systemd will restart the process – on failure.
Here, sshd will be restarted on an unclean exit code, unclean signal, a timeout…

RestartSec=42s

Time to take before restarting the service following the Restart= criteria is met.

[Install]

WantedBy=multi-user.target

This service is wanted by multi-user.target directory.
In multi-user mode, sshd will be started.

6. Customizing systemd unit files:

There’s no guarantee that your changes will be persistent across updates of your system.
Package managers might take offence to your mangling of their service files, and simply replace them without warning.
Correct way to modify the unit file is to write a whole other file with your custom changes
systemd’s method for doing this is called a snippet

# systemctl edit sshd.service

Empty file will be created
/etc/systemd/system/sshd.service.d/override.conf
Inside empty file, add your entries.
Save and exit

# systemctl show sshd

Test if the config is saved and loaded correctly

# systemctl daemon-reload

# systemctl restart sshd

7. Testing running services:

Three ways to see if a service is running: systemctl, ps, telnet

8. Working with systemd timers (and cron)

Timers act as the instruction for when another unit is to trigger
In old distros, such events would be controlled with cron

Let’s take a look what this timer does:

Let’s take a look what this service does:

Type=oneshot

Means it is expected to exit after running

9. Other init systems:

We have two systems at play:

CentOS 6 uses Upstart init
Debian 7 uses SysV init

CentOS 6 and Upstart init:

systemctl wont work
Instead of systemctl, we use service –status-all

# service --status-all

List services and their status

# /etc/init.d/sshd status

Show service status

Start/stop service:

# service servicename stop

# service servicename start

Disable/enable service:

# chkconfig --list servicename

# chkconfig servicename off

# chkconfig servicename on

Debian 7 and SysV init:

# /etc/init.d/ssh status

# update-rc.d servicename disable

# update-rc.d servicename enable

# chkconfig ssh on

# chkconfig ssh off