Logging and Syslog Analysis: rsyslog, journald & Key Log Files
Logs are usually the first place an investigation turns, and Linux hosts often run two logging systems side by side rather than one. This chapter covers where that evidence actually lives, how to read it, and what its absence tells you.
Logging Architecture: syslog vs journald
Most Linux distributions log through one or both of two systems. Knowing which is active on a host determines whether you go looking for flat files or reach for a query tool instead.
| System | Storage | Notes |
|---|---|---|
| rsyslog / syslog | Plain-text files under /var/log/ | The traditional approach, still widely used and readable with any text tool |
| systemd-journald | Structured binary journal | Not a flat text file, queried with a dedicated tool rather than grepped directly |
Many distributions run both side by side. journald often forwards entries to rsyslog, which is what keeps the traditional flat-log files under /var/log/ populated even on a system that logs natively into the journal first.
Key Log Files
Where a given category of event lands depends on which distribution family the host belongs to. Debian-family and RedHat-family systems use different filenames for the same purpose.
| File | Purpose |
|---|---|
/var/log/auth.log | Debian/Ubuntu-family authentication events (logins, sudo, SSH) |
/var/log/secure | The RedHat/CentOS-family equivalent, same purpose, different name |
/var/log/syslog | General system activity on Debian/Ubuntu-family hosts |
/var/log/messages | The RedHat/CentOS-family equivalent of general system activity |
/var/log/audit/audit.log | Present only when auditd is configured, detailed system call and file-access auditing |
/var/log/cron | Scheduled task activity, on some distributions mixed into the general system log instead |
Reading Authentication Events
Exact log line formatting varies by distribution and daemon version, so this is about what to look for conceptually rather than a specific string to grep for.
- Repeated failed SSH login attempts: a burst against one account, or many accounts from one source, points toward a brute-force attempt.
- A successful login right after a burst of failures: often the strongest single signal in an auth log, a credential-guessing attempt that landed.
- sudo invocations tied to a specific account and command: shows exactly who escalated privilege and what they ran with it.
- New user or group creation events: legitimate during provisioning, worth scrutiny anywhere else, especially outside a known change window.
Querying journald
journalctl is the standard tool for reading the structured journal. It can filter by systemd unit, by time range, and by priority level, which is often faster than grepping through flat files for the same information.
Log Tampering and Gaps
Clearing or truncating logs is a real, well-documented technique attackers use to cover their tracks. A gap or discontinuity in log timestamps and sequence numbers is itself a forensic tell, worth flagging even without knowing exactly what content was removed.
Being able to read logs accurately is the foundation the next three chapters build on:
- Persistence Mechanisms: many persistence techniques leave a log trace at the moment they are installed.
- Process and Network Internals: correlating live process and connection state against what the logs already showed.
- Shell Tradecraft and Detection: recognizing command-line activity that logging alone may only partially capture.
Key Takeaways
- Linux logs through rsyslog (plain-text files), systemd-journald (structured binary journal), or often both at once, with journald commonly forwarding to rsyslog.
- Key log locations split by distribution family: auth.log/syslog on Debian-family hosts, secure/messages on RedHat-family hosts.
- auditd, when configured, adds detailed system call and file-access auditing in
/var/log/audit/audit.log. - Auth log review centers on failed login bursts, a success following a burst, sudo activity, and new account creation.
journalctlfilters the structured journal by unit, time range, and priority, but journal persistence across reboots depends on configuration.- Cleared logs and gaps in timestamps or sequence numbers are themselves forensic evidence of tampering, detectable even without recovering what was removed.
Knowledge Check
Click an answer to reveal the explanation.
On a RedHat/CentOS-family host, which file serves the same purpose as /var/log/auth.log on Debian/Ubuntu?
What is journalctl primarily used for?
An investigator notices a discontinuity in log sequence numbers with no content explaining the missing entries. Why does this matter even without recovering what was deleted?