108.2 System Logging

Weight: 4

Goal: Configure the system log daemon, and understand log rotation and journald.

1. Two Generations of Logging

A modern Linux system has two parallel logging systems, and the exam covers both:

System	Daemon	Storage	Format
Traditional syslog	`rsyslogd` (most common), `syslog-ng`, original `syslogd`	Plain text files in `/var/log/`	One line per message
systemd journal	`systemd-journald`	Binary database (`/var/log/journal/` or `/run/log/journal/`)	Indexed, structured records

Most distributions run both at the same time: journald captures everything early in boot, and forwards a copy to rsyslog, which writes the familiar text files.

2. Traditional Syslog Concepts

Syslog organizes log messages by two attributes:

Facility (where the message comes from)

Facility	Source
`auth`, `authpriv`	Authentication / authorization (login, sudo, ssh).
`cron`	The cron daemon.
`daemon`	Generic system daemons.
`kern`	The Linux kernel.
`lpr`	Printing.
`mail`	Mail subsystem.
`news`	Usenet news.
`syslog`	Syslog itself.
`user`	Generic user processes (the default).
`local0`–`local7`	Reserved for custom local use.

Priority / severity (how serious it is)

From least to most severe, with numeric codes:

#	Name	Meaning
7	`debug`	Debugging info.
6	`info`	Informational.
5	`notice`	Normal but significant.
4	`warning`	Warning.
3	`err`	Error.
2	`crit`	Critical.
1	`alert`	Action must be taken immediately.
0	`emerg`	System unusable.

A selector combines them as facility.priority, for example mail.info, cron.err, *.emerg.

The priority is a minimum — mail.info matches info and everything more serious. Use =info to mean “exactly info,” or * to mean “everything.”

3. `rsyslogd` Configuration

rsyslogd (rocket-fast syslog daemon) is the default syslog on most modern Linux systems and is backward-compatible with the original syslog format.

Files

Path	Purpose
`/etc/rsyslog.conf`	Main configuration file.
`/etc/rsyslog.d/*.conf`	Drop-in config files included from `rsyslog.conf`.
`/var/log/`	Where text logs are written.
`/etc/syslog.conf`	The older syslogd’s config file — same syntax, exam still references it.

Basic rule syntax

facility.priority    action

Examples from a typical /etc/rsyslog.conf:

# Everything from kernel goes to its own file
kern.*                          /var/log/kern.log

# Mail subsystem: keep info and above
mail.info                       /var/log/mail.info

# Authentication: priority "err" and above
auth,authpriv.err               /var/log/auth-errors.log

# Everything at "info" or higher, except mail and cron
*.info;mail.none;cron.none      /var/log/messages

# Emergency messages broadcast to ALL logged-in users
*.emerg                         :omusrmsg:*

# Send all errors to a remote loghost
*.err                           @loghost.example.com         # UDP
*.err                           @@loghost.example.com:514    # TCP

Notes:

Multiple facilities are comma-separated: auth,authpriv.err.
Multiple selectors are semicolon-separated: *.info;mail.none;cron.none.
none excludes that facility. mail.none means “no mail messages here.”
An action starting with - (e.g. -/var/log/messages) means “don’t sync after every write” — faster, but a crash may lose the last messages.
An action starting with @ sends to a remote host over UDP (port 514).
An action starting with @@ sends over TCP (port 514).
:omusrmsg:* writes the message to all logged-in users’ terminals (the modern form of the old * action).

Reloading rsyslog after a config change

systemctl restart rsyslog
# or
kill -HUP $(pidof rsyslogd)

4. Files You Will See in `/var/log/`

Exact filenames vary by distribution, but the exam expects you to recognize:

File	Contents
`/var/log/messages`	General system messages (Red Hat family especially).
`/var/log/syslog`	General system messages (Debian family).
`/var/log/auth.log` (Debian) / `/var/log/secure` (RHEL)	Authentication, sudo, SSH.
`/var/log/kern.log`	Kernel messages.
`/var/log/mail.log`	Mail subsystem.
`/var/log/cron.log`	cron activity.
`/var/log/boot.log`	Messages from the last boot.
`/var/log/dmesg`	Kernel ring buffer at boot.
`/var/log/wtmp`	Binary — login history (read with `last`).
`/var/log/btmp`	Binary — failed logins (read with `lastb`).
`/var/run/utmp`	Binary — current logged-in users (read with `who`, `w`).
`/var/log/lastlog`	Binary — last login time per user (read with `lastlog`).

Binary files must not be read with cat or less — they only make sense through their dedicated commands.

5. Sending Your Own Messages: `logger`

logger writes a message into syslog from the command line — useful in shell scripts.

logger "Backup script started"
logger -p user.warning "Disk space getting low"
logger -t backup-script "Backup finished"      # set the tag (program name)
logger -p local3.info -t myapp "user logged in"

Important flags:

-p facility.priority — choose facility and severity (default user.notice).
-t TAG — set the tag (the program name shown in the log line).
-i — include the PID.

6. Inspecting Logs in Real Time

tail -f /var/log/syslog               # follow new lines as they arrive
tail -n 50 /var/log/auth.log          # last 50 lines
less /var/log/messages
grep sshd /var/log/auth.log

tail -f is the standard way to watch a log as it changes.

7. Log Rotation: `logrotate`

Without rotation, log files would grow forever. logrotate periodically renames, compresses, and eventually deletes old logs.

How it runs

logrotate is not a daemon. It is a program run by cron — typically by /etc/cron.daily/logrotate. Each day it reads its config and decides what to do.

Configuration files

Path	Purpose
`/etc/logrotate.conf`	Main configuration file with global defaults.
`/etc/logrotate.d/`	One file per package (`rsyslog`, `apache2`, `nginx`, etc.).
`/var/lib/logrotate/status`	State file: when each log was last rotated.

A typical entry

/var/log/syslog {
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Key directives:

Directive	Meaning
`daily` / `weekly` / `monthly` / `yearly`	Rotation frequency.
`rotate N`	Keep N old versions before deleting.
`size N`	Rotate when the log reaches size N (e.g. `size 100M`).
`compress`	gzip rotated files.
`nocompress`	Don’t compress.
`delaycompress`	Delay compression by one rotation (so the most recent old log stays readable).
`missingok`	Don’t error if the log doesn’t exist.
`notifempty`	Don’t rotate an empty file.
`create MODE OWNER GROUP`	After rotation, create a new empty log with these permissions.
`copytruncate`	Copy the log, then truncate it (instead of renaming). Used when a process can’t be told to reopen its file.
`dateext`	Append a date to rotated filenames instead of `.1`, `.2`, etc.
`postrotate ... endscript`	Run these commands after rotation (typically to signal the daemon).

Running `logrotate` manually

logrotate -d /etc/logrotate.conf      # DRY RUN — show what would happen
logrotate -v /etc/logrotate.conf      # verbose, actually rotate
logrotate -f /etc/logrotate.conf      # FORCE rotation now, even if not due

-d (debug) is invaluable for testing a new config without touching real files.

8. The systemd Journal

systemd-journald is systemd’s logging service. It captures all kernel and service messages from the very start of boot — earlier than rsyslog can — and stores them in an indexed binary database.

Storage location

Path	Meaning
`/var/log/journal/`	If this directory exists, logs are persistent across reboots.
`/run/log/journal/`	If it doesn’t, logs are stored in RAM and lost at reboot (default on some distros).

Configuration

/etc/systemd/journald.conf controls journald. Key options:

Storage= — auto (default), persistent, volatile, or none.
SystemMaxUse= — total disk limit for journals (e.g. 500M).
MaxRetentionSec= — automatically delete entries older than this.
ForwardToSyslog=yes — also send messages to rsyslog (the usual setup).

To make the journal persistent on a system where it isn’t:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

9. `journalctl` — Reading the Journal

journalctl                          # all entries, oldest first
journalctl -r                       # newest first
journalctl -f                       # follow new entries (like tail -f)
journalctl -n 50                    # last 50 entries
journalctl -e                       # jump to the end

journalctl -b                       # entries from the current boot
journalctl -b -1                    # the previous boot
journalctl --list-boots             # list known boots

journalctl -u sshd.service          # only messages from this unit
journalctl -u nginx -f              # follow one unit
journalctl _PID=1234                # filter by PID
journalctl /usr/bin/sshd            # filter by executable

journalctl -p err                   # priority "err" or higher
journalctl -p warning..emerg        # priority range

journalctl --since "2026-05-10"
journalctl --since "1 hour ago"
journalctl --since today --until "1 hour ago"

journalctl -k                       # kernel messages only (like dmesg)

journalctl --disk-usage             # how much space the journal uses
journalctl --vacuum-time=7d         # delete entries older than 7 days
journalctl --vacuum-size=100M       # keep only the most recent 100MB

Key things to remember about journalctl:

It works on a binary index, so it’s much faster than grep-ing flat files.
Priorities use the same syslog levels (emerg, alert, crit, err, warning, notice, info, debug) or numbers 0–7.
-u UNIT is the most common filter you’ll use day to day.

10. Remote Logging (Awareness)

The objective expects you to know that syslog can send log messages to a central server. With rsyslog, in /etc/rsyslog.conf:

*.*    @loghost.example.com         # UDP, port 514
*.*    @@loghost.example.com:514    # TCP, port 514

On the receiving server, rsyslog needs the appropriate input module enabled (imudp or imtcp) and to listen on port 514.

11. Quick Reference for the Exam

Daemons and tools:

rsyslogd (and the older syslogd, syslog-ng)
systemd-journald
logger
logrotate
journalctl

Files:

/etc/rsyslog.conf, /etc/rsyslog.d/
/etc/syslog.conf (legacy name, same syntax)
/etc/logrotate.conf, /etc/logrotate.d/
/etc/systemd/journald.conf
/var/log/ (especially messages, syslog, auth.log/secure, kern.log, dmesg)
/var/log/journal/ (persistent journal)
/run/log/journal/ (volatile journal)

Concepts:

Facility and priority — be able to list them.
Selectors like mail.info, *.emerg, *.info;mail.none;cron.none.
@ for UDP, @@ for TCP in rsyslog actions.
Default syslog port: UDP/TCP 514.

12. Likely Exam Questions (Self-Check)

What is the main configuration file for the traditional syslog daemon? /etc/syslog.conf. For rsyslog (the modern replacement), /etc/rsyslog.conf plus /etc/rsyslog.d/.
What two attributes classify every syslog message? Facility (the source: auth, cron, kern, mail, …) and priority/severity (debug through emerg).
List the syslog priorities from least to most severe. debug, info, notice, warning, err, crit, alert, emerg.
What does the rule *.info;mail.none;cron.none /var/log/messages mean? Log all messages at priority info or higher, except those from the mail and cron facilities, into /var/log/messages.
What is the difference between @host and @@host in an rsyslog action? @host sends over UDP, @@host sends over TCP.
What port does syslog use by default? 514, both UDP and TCP.
What command writes a message to the log from a shell script? logger. Example: logger -p user.warning "Disk almost full".
What command tails the system log in real time? tail -f /var/log/syslog (or /var/log/messages), or journalctl -f.
How does logrotate actually run? Is it a daemon? No — it’s a regular program run by cron, usually from /etc/cron.daily/logrotate.
Where are package-specific logrotate rules stored? In /etc/logrotate.d/.
What is copytruncate and when is it used? logrotate copies the log file to a new name and then truncates the original to zero. Used when the program writing the log can’t be told to reopen its file.
What is the difference between /var/log/journal/ and /run/log/journal/? Logs in /var/log/journal/ are persistent across reboots. Logs in /run/log/journal/ are stored in RAM and lost on reboot.
How do you view all log messages from the current boot? journalctl -b.
How do you show only messages from the sshd service? journalctl -u sshd.service.
How do you display only kernel messages from the journal? journalctl -k (equivalent to dmesg).
What is the difference between last and who? who shows currently logged-in users (from /var/run/utmp). last shows historical login records (from /var/log/wtmp).