systemd

Systemd logging

Systemd writes messages to a binary log stored in /var/log/journal this cannot be viewed directly it can be viewed using the journalctl command. Without any options the entire log is viewed:

journalctl

Messages since the last boot can be viewed using:

journalctl -b

Messages since a point in time can be viewd using the -S or –since options. For example message today since 10:00 as displayed using:

journalctl --since=10:00

Message since 11:00am on 10/04/2023 can be displayed using (quotes are required since there is a space between the date and the time):

journalctl --since="2023-04-10 11:00"

To list the amount of disk space used by the journal files:

journalctl --disk-usage

Remove old entries (e.g. greater than 10 days) from systemd journal:

sudo journalctl --vacuum-time=10d

Or to restrict the journal to a specified size use:

sudo journalctl --vacuum-size=100M

Journal Configuration

Warning

/etc/systemd/journald.conf is the master configuration for the systemd journal - changes to this may be overwritten by an upgrade so it’s better to add a file to the /etc/systemd/journald.conf.d directory.

The latest salt states add a configuration setting to journal.conf.d/ to restrict the time messages are retained to 10 days.

The salt states add a file which performs the vacuum command automatically:

/etc/systemd/journald.conf.d/journald.conf

The contents of that file are:

[Journal]
MaxRetentionSec=10d

Changes will take effect from next time the systemd-journald service is restarted, Either at system boot or using:

sudo systemctl restart systemd-journald

Investigating issues

We’ve noticed that a typical system with 10 days retention will have from 32M to 150Mb. If the log is larger than 150Mb, it may indicate an issue on the system that is causing messages to be logged.

Look for repeated error messages in the log. e.g. recently we found the following messages repeatedly in the log for a system:

Apr 09 00:00:06 SYSTEMNAME systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Apr 09 00:00:06 SYSTEMNAME systemd[1]: logrotate.service: Failed with result 'exit-code'.
Apr 09 00:00:06 SYSTEMNAME systemd[1]: Failed to start Rotate log files.

On a healthy system this would read:

Apr 09 00:00:02 SYSTEMNAME systemd[1]: logrotate.service: Succeeded.

Using this information, one can investigate the logrotate service and find the rule that is causing the failure.