Process and Network Internals: /proc, Process Trees & Socket Visibility
Once persistence has launched something, or an attacker is actively on a box, this is where you find it: live process state, network connections, and open files. Everything in this chapter reads the running system directly rather than a log or a saved artifact, which makes it the fastest path to ground truth during an active investigation.
The /proc Filesystem
/proc is a virtual filesystem. It is generated live by the kernel rather than stored on disk, and it exposes process and system state as if it were a normal directory tree you can cd into and read with everyday tools.
Every running process gets its own numbered directory under /proc, named after its PID. Inside that directory sit files and symlinks that describe exactly what the process is doing right now.
| Path | Reveals |
|---|---|
/proc/[pid]/cmdline | The exact command line a process was launched with, including arguments |
/proc/[pid]/exe | A symlink to the actual binary that's running, useful when a process name alone is misleading |
/proc/[pid]/fd/ | Every file descriptor the process currently has open |
/proc/net/ | Kernel-level network state, the socket tables netstat reads directly and that ss reports on via the faster netlink interface |
/proc/[pid]/exe still points at the real binary on disk. That mismatch between claimed identity and actual binary is a recurring way disguised processes get caught.Process Tree Analysis
Chapter 1 introduced PID and PPID as the identifiers that link a process to its parent. Applied practically, that ancestry chain is one of the most reliable things you can check on a live system, because for a given service it usually looks the same every single time.
A web server process is expected to have a predictable parent, and its children are expected to be predictable too, worker processes, not interactive shells. An unexpected parent, such as a web server process spawning a shell, is one of the strongest anomaly signals available, because it means something outside the service's normal behavior caused that execution. Legitimate application code does not usually need to hand control to a shell.
| Observation | What It Suggests |
|---|---|
| Service process with an unfamiliar parent | The service may have been started or re-launched outside its normal supervision path |
| Web server or database process spawning a shell | Likely command execution through an application vulnerability |
| Long-lived process with a very recent start time | Possible replacement, injection, or process masquerading as something older |
| Cron or init-owned process with an odd ancestry | Worth checking against the persistence mechanisms covered in chapter 4 |
Network Visibility: ss and netstat
Once a process looks suspicious, the next question is usually who it's talking to. Two tools answer that on Linux, and they overlap heavily in what they can show.
| Tool | Notes |
|---|---|
ss | The modern socket-statistics tool, faster and the current standard on most distributions |
netstat | The older tool, still present on many systems but considered legacy |
Both show listening ports and established connections, the two states that matter most during triage. With the right options, both can also show which process owns each connection, which is what turns "something is talking to this IP" into "this specific process is talking to this IP."
ss first on any modern distribution. Treat netstat as a fallback for older or minimal systems where ss isn't installed.Open File Visibility with lsof
lsof lists every file descriptor held open by every process. On Linux that scope is wider than it sounds, since sockets and pipes are file descriptors too, not just regular files on disk.
That breadth is why lsof is often used alongside ss and /proc: it can confirm which process owns a given socket, what regular files a suspicious process is reading or writing, and what a process is holding open that nothing else on the system references anymore.
Putting It Together
These tools are rarely used in isolation. A live investigation usually moves through them in the same order, narrowing from "something looks off" to a specific, evidenced conclusion.
That workflow covers what a process is doing right now, but a lot of live tradecraft happens inside a shell before a process ever looks this suspicious, which is exactly where chapter 6 picks up.
Key Takeaways
/procis a live, kernel-generated view of process and system state, not a stored filesystem, with per-PID directories exposing command lines, real binary paths, and open file descriptors.- Process ancestry (PID/PPID) is predictable for a given service, so an unexpected parent, like a web server spawning a shell, is a strong anomaly signal.
ssis the modern, faster socket-statistics tool and the current standard;netstatis older but still found on many systems.- Both
ssandnetstatcan tie a connection to the specific process that owns it, not just the remote IP involved. lsofreveals every file descriptor a process holds, including sockets and pipes, and can expose files that were deleted from disk but are still held open, a real hiding and recovery technique.- Live investigation follows a repeatable sequence: spot the process, verify it via /proc, check its network activity via ss, check its open files via lsof, then correlate with known persistence mechanisms.
Knowledge Check
Click an answer to reveal the explanation.
What does /proc/[pid]/exe show for a running process?
Why is ss generally preferred over netstat on modern Linux distributions?
During an lsof review, you find a process holding open a file that no longer appears anywhere on the filesystem. What does this indicate?