Linux Forensics: Triage and Artifact Collection

Disclaimer: This article is provided strictly for educational purposes and authorized security testing. Only run these techniques against systems you own or have explicit written permission to assess. Unauthorized access to computer systems is illegal in virtually every jurisdiction and can carry severe penalties.

Introduction

Linux forensics is a different discipline from Windows DFIR: there is no single unified event log, artifact locations vary by distribution and init system, and much of what an analyst needs — bash history, cron entries, package manager logs — was designed for administrators, not auditors. Linux hosts nonetheless dominate cloud, container, and server environments, and web shells, cryptominers, and SSH-based lateral movement leave a consistent, learnable set of traces across /var/log, the systemd journal, cron/timer configuration, and shell history files.

Effective Linux triage follows the classic order of volatility: capture what disappears first (running processes, network connections, in-memory state) before anything that requires a reboot or risks overwriting evidence, then move to disk acquisition and offline parsing. Because most production Linux incidents involve a live, running system that cannot simply be pulled from the network, live triage scripting is a core skill distinct from, but complementary to, full disk imaging.

Attack Prerequisites

This is a defensive analysis capability; using it requires:

• Root or sudo access to the live host, or an offline disk image/snapshot for dead-box analysis.
• A read-only or minimally invasive collection method — a dedicated triage collector (e.g. UAC, the Unix-like Artifacts Collector) or a documented manual command sequence, run from read-only media where possible to avoid contaminating the evidence.
• Awareness of log rotation (logrotate) and retention windows, since /var/log history is frequently compressed and rotated out within days.
• Offline parsing tooling — plaso/log2timeline for supertimelines, journalctl for the systemd journal, and standard find/grep/stat for targeted collection.
• A cloud/container awareness pass where relevant — ephemeral instances and containers may need their logs shipped centrally *before* termination, since local disk evidence disappears with the instance.

How It Works

Linux logging is fragmented across purpose-built files rather than a single structured store. /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS) records authentication events — SSH logons, sudo usage, PAM failures. /var/log/wtmp and /var/log/btmp are binary logon databases read via last/lastb rather than a text editor. On systemd-based distributions, the journal (journalctl) is a structured, binary-indexed log that unifies kernel, service, and application output and supports rich filtering by time, unit, and priority — often the fastest way to reconstruct service-level activity.

Execution and persistence artifacts are similarly scattered: shell history (~/.bash_history, ~/.zsh_history) records interactive commands but is trivially disabled (unset HISTFILE) or cleared by an attacker; cron (/etc/crontab, /etc/cron.d/, per-user crontabs under /var/spool/cron/) and systemd timers (.timer unit files paired with a .service unit) are the dominant persistence mechanisms; and auditd, when configured with rules, is the closest Linux equivalent to Windows process-creation auditing, logging execve calls, file access to watched paths, and more via the kernel audit subsystem.

The order of volatility for live triage runs, roughly: CPU/register state and running processes/memory, network connections and open sockets, temporary filesystem contents (/tmp, /dev/shm), disk contents, and finally archived logs and backups. A live-response script should capture process listings, network state, and loaded kernel modules *before* anything that requires writing to or reading extensively from disk, both to minimize evidence disturbance and because that state is gone on reboot.

Practical Example / Configuration

A representative live-triage command sequence, ordered by volatility, collecting output to a read-only or dedicated evidence location:

# --- volatile state first ---
ps auxww > proc_list.txt
ss -tulpn > sockets.txt              # or: netstat -tulpn
lsmod > kernel_modules.txt
who -a > logged_in_users.txt

# --- authentication and logon history ---
last -f /var/log/wtmp > last_logons.txt
lastb -f /var/log/btmp > failed_logons.txt
journalctl --since "2026-07-10" --until "2026-07-12" -u sshd > sshd_journal.txt

# --- persistence surfaces ---
for u in $(cut -f1 -d: /etc/passwd); do crontab -l -u "$u"; done 2>/dev/null > crontabs.txt
cat /etc/cron.d/* /etc/cron.daily/* /etc/cron.hourly/* 2>/dev/null > cron_files.txt
systemctl list-timers --all > systemd_timers.txt
ls -la /etc/systemd/system/ /lib/systemd/system/ > systemd_units.txt

# --- suspicious file discovery ---
find / -xdev -perm -4000 -type f 2>/dev/null > suid_files.txt      # SUID binaries
find / -xdev -mtime -1 -type f 2>/dev/null > modified_last_24h.txt # recent writes

# --- audit trail, if auditd is configured ---
ausearch -m USER_LOGIN -ts today > ausearch_logins.txt
ausearch -m EXECVE -ts today > ausearch_exec.txt

# --- volatile state first ---
ps auxww > proc_list.txt
ss -tulpn > sockets.txt              # or: netstat -tulpn
lsmod > kernel_modules.txt
who -a > logged_in_users.txt

# --- authentication and logon history ---
last -f /var/log/wtmp > last_logons.txt
lastb -f /var/log/btmp > failed_logons.txt
journalctl --since "2026-07-10" --until "2026-07-12" -u sshd > sshd_journal.txt

# --- persistence surfaces ---
for u in $(cut -f1 -d: /etc/passwd); do crontab -l -u "$u"; done 2>/dev/null > crontabs.txt
cat /etc/cron.d/* /etc/cron.daily/* /etc/cron.hourly/* 2>/dev/null > cron_files.txt
systemctl list-timers --all > systemd_timers.txt
ls -la /etc/systemd/system/ /lib/systemd/system/ > systemd_units.txt

# --- suspicious file discovery ---
find / -xdev -perm -4000 -type f 2>/dev/null > suid_files.txt      # SUID binaries
find / -xdev -mtime -1 -type f 2>/dev/null > modified_last_24h.txt # recent writes

# --- audit trail, if auditd is configured ---
ausearch -m USER_LOGIN -ts today > ausearch_logins.txt
ausearch -m EXECVE -ts today > ausearch_exec.txt

Walkthrough / Exploitation

For a suspected web-shell or cron-based compromise, the investigative flow layers live triage with targeted offline parsing:

# 1. Confirm persistence: does the timer/cron entry point at something odd?
cat /etc/cron.d/apache_cleanup   # attacker-planted cron entry disguised as maintenance
systemctl cat suspicious.timer   # inspect a systemd timer + its paired .service unit

# 2. Pull auth history around the suspected initial-access window
journalctl -u sshd --since "2026-07-09 00:00" --until "2026-07-10 00:00"

# 3. Full disk image for offline processing once the host is contained
dcfldd if=/dev/sda of=/evidence/host01.img hash=sha256 hashlog=/evidence/host01.sha256

# 4. Build a supertimeline from the acquired image
log2timeline.py plaso.dump /evidence/host01.img
psort.py -o l2tcsv -w timeline.csv plaso.dump

# 1. Confirm persistence: does the timer/cron entry point at something odd?
cat /etc/cron.d/apache_cleanup   # attacker-planted cron entry disguised as maintenance
systemctl cat suspicious.timer   # inspect a systemd timer + its paired .service unit

# 2. Pull auth history around the suspected initial-access window
journalctl -u sshd --since "2026-07-09 00:00" --until "2026-07-10 00:00"

# 3. Full disk image for offline processing once the host is contained
dcfldd if=/dev/sda of=/evidence/host01.img hash=sha256 hashlog=/evidence/host01.sha256

# 4. Build a supertimeline from the acquired image
log2timeline.py plaso.dump /evidence/host01.img
psort.py -o l2tcsv -w timeline.csv plaso.dump

Package manager logs (/var/log/dpkg.log on Debian/Ubuntu, /var/log/yum.log or the rpm database on RHEL) establish whether a suspicious binary arrived via a legitimate package install or was dropped independently — a quick way to separate expected software from an attacker-planted tool with the same name.

Note: /tmp and /dev/shm are common staging locations for downloaded payloads because they are world-writable by default and, in the case of /dev/shm, backed by RAM rather than disk — contents vanish on reboot, so they belong in the volatile-collection phase, not the disk-imaging phase.

Opsec / Analyst tip: An attacker clearing HISTFILE or running history -c does not erase commands already flushed to .bash_history, and does nothing to auditd EXECVE records or the systemd journal if either is configured — always check for these before assuming interactive activity is unrecoverable.

Detection and Defense

Linux hardening for forensic readiness is as important as the analysis technique itself, since most distributions log very little by default:

• Configure auditd rules for execve, and for writes to sensitive paths, e.g. -w /etc/passwd -p wa -k passwd_changes and -a always,exit -F arch=b64 -S execve -k exec.
• Forward logs off-host to a central syslog/SIEM collector so /var/log tampering or deletion on the source host does not destroy the only copy.
• Enable and centrally review shell history (HISTTIMEFORMAT, PROMPT_COMMAND logging, or session recording via auditd/tlog) since local history files are trivially disabled by an attacker with shell access.
• Baseline cron, systemd timers, and startup services across the fleet so new or modified entries are quickly visible.
• Increase journald retention (SystemMaxUse= in journald.conf) so the journal is not rotated out before an investigation begins.

Real-World Impact

Linux and cloud-hosted server compromises are extremely common in practice — web shells dropped via vulnerable applications, cron/systemd-timer persistence for cryptomining, and SSH key or credential abuse for lateral movement are recurring findings across cloud and hosting-provider incident response. Because many organizations run Linux fleets with minimal native auditing, the difference between a fast, conclusive investigation and an inconclusive one is frequently whether auditd and central log forwarding were configured before the incident, not after.

Conclusion

Linux triage rewards discipline more than tooling: capturing volatile state before disk state, knowing where the fragmented log and persistence artifacts actually live across /var/log, the journal, cron, and systemd timers, and corroborating shell history against auditd and package-manager logs turns a sparse-by-default logging story into a workable investigation.

You Might Also Like

If you found this useful, these related deep-dives cover adjacent techniques and their defenses:

• Windows Registry Forensics for DFIR
• NTFS Forensics: The Master File Table (MFT)
• Memory Forensics with Volatility 3
• Linux auditd: Configuration and Threat Hunting