Advanced Evasion & Anti-Analysis

Why This Chapter Exists

Chapter 1 introduced static and dynamic analysis as complementary lenses on the same sample, and noted in passing that some malware detects sandbox and VM environments and suppresses its own malicious behavior in response. Chapter 3 picked the topic back up while covering dynamic analysis and sandboxing, deliberately declined to go deep on it, and pointed forward to this chapter instead. The reasoning in both places was the same: evasion and anti-analysis are big enough, and specialized enough, to deserve dedicated treatment rather than a paragraph bolted onto a chapter about something else.

That treatment matters because every technique taught in this module -- static triage, dynamic detonation, unpacking, injection and C2 analysis, YARA and Sigma authoring -- assumes the sample under study behaves the same way in the lab as it would on a real endpoint. Modern malware, particularly anything built by a competent crimeware developer or a nation-state toolkit team, does not make that assumption safe. It checks. If the check comes back "this looks like a lab," the sample can go dormant, exit cleanly, or feed the analyst a fake benign code path while the real payload stays locked behind a condition that never fires in the sandbox.

The structure of this chapter mirrors that reality. Every section below covers one category of evasion check, and each follows the same three-part shape:

1. What the malware is looking for. The specific artifact, API, or signal being queried.
2. Why that signal works as a proxy. What makes it a reasonable stand-in for "this is not a real victim machine" or "this process is being watched."
3. What the defender or analyst does about it. Present in every section, without exception.

The goal is not to hand over a checklist of tricks. It's to make sure that when one of these checks fires against your own tooling, you recognize it immediately instead of chasing a sample that has already gone quiet on you.

Anti-VM and Anti-Sandbox Detection

Anti-VM and anti-sandbox checks are the malware's first line of environment fingerprinting. The logic is almost always defensive from the malware's point of view: real infection targets vastly outnumber analysis environments, so a sample that only detonates on what looks like a genuine, unmonitored victim machine buys itself a longer operational life and denies analysts a clean detonation.

Hardware and Registry Artifacts

Virtualization platforms leave fingerprints that are trivial to query from user mode: VMware and VirtualBox register specific driver names, service names, and registry keys (HKLM\SYSTEM\CurrentControlSet\Services entries for vmci, VBoxGuest, and similar); MAC address OUI prefixes assigned to VMware and VirtualBox NICs are public and easy to check against; hypervisor-specific CPUID leaves and I/O port behavior (the classic VMware backdoor I/O port at 0x5658) reveal the platform without touching a single file. A sample checking any of these before deciding whether to unpack its real payload is checking "am I on real hardware," not doing anything a legitimate installer would ever need to do.

Resource Starvation as a Signal

Sandboxes and analysis VMs are commonly provisioned lean -- a single CPU core, 2-4 GB of RAM, a small disk -- because most sandbox farms run hundreds of instances concurrently and can't afford to over-provision each one. Malware that calls GetSystemInfo or checks total physical memory and bails out below some threshold (one core, under 2 GB RAM, a disk under a few dozen gigabytes) is betting, correctly in a lot of environments, that it's looking at automated analysis infrastructure rather than a real employee's laptop.

Analysis Tooling Fingerprints

A running-process enumeration that checks for process names like wireshark.exe, procmon.exe, x64dbg.exe, ida64.exe, or ollydbg.exe is looking for the analyst's own toolkit. Some samples extend this to checking loaded module names inside their own process for known monitoring hooks, or checking window class names and titles for analysis tool UIs sitting in the background.

Timing-Based Detection

API hooking, instrumentation, and heavyweight monitoring (Procmon's kernel filter driver, a debugger's single-stepping, an emulator's instruction-by-instruction interpretation) all add measurable latency to operations that are near-instant on bare metal. A sample that calls RDTSC (Read Time-Stamp Counter) before and after a benign API call, or times a small sleep against the wall clock, and finds the operation took orders of magnitude longer than it should, has detected that something is intercepting its execution -- without ever needing to know what that something is.

Human-Interaction Checks

An automated sandbox detonates a sample and lets it run for a fixed window, usually with no mouse movement, no keyboard input, and an uptime of seconds since boot. Malware that checks cursor position over a short interval and finds it never moves, or checks system uptime and finds it's implausibly low for a machine that's supposedly been in daily use, is inferring "no human is actually sitting at this keyboard" -- a reasonable proxy for automated analysis that a real victim's machine will never trip.

The Countermeasure Side

None of this is unbeatable, and none of it should be treated as a reason to give up on sandbox-based analysis. Three practical responses cover most of it:

1. Harden the sandbox VM itself. Rename or hide the telltale registry keys and services, randomize the MAC OUI, provision closer to real endpoint specs (multiple cores, realistic RAM), inject synthetic mouse movement and realistic uptime, and remove or rename analysis tooling process names before detonation. Commercial sandbox platforms and open-source projects both maintain active work on this front because it's a constant arms race.
2. Fall back to bare metal. For samples that specifically and successfully evade every hardened VM available, a dedicated physical machine that gets reimaged after each run removes the VM-detection surface entirely, at the cost of losing snapshot and rollback convenience.
3. Treat the evasion check itself as the signal. Often overlooked, and the cheapest of the three. A legitimate installer has no reason to query hypervisor I/O ports or enumerate process names looking for Wireshark. A sample that does this can be flagged as suspicious from the static analysis pass in Chapter 2, before a single dynamic detonation attempt, purely because that check exists in its code.

Anti-Debugging Techniques

Where anti-VM checks target the environment, anti-debugging checks target the specific tool an analyst is using to step through the sample's code live. These checks are narrower in scope than anti-VM detection but often more disruptive to an analyst mid-session, since a debugger check firing can crash the process, corrupt the analysis, or silently divert execution down a fake code path right as you're trying to reach the real payload.

The Four Check Categories

They escalate in technical sophistication, and in how hard they are to neutralize:

The Countermeasure Side

These checks are common enough across commodity malware and packers that the reverse-engineering community built general-purpose tooling rather than patching each one by hand:

• ScyllaHide covers the API and PEB checks. A plugin for x64dbg and OllyDbg, it hooks the relevant APIs (IsDebuggerPresent, CheckRemoteDebuggerPresent, NtQueryInformationProcess, and others) and the PEB fields they read, returning "no debugger here" regardless of the actual attached state. Off-the-shelf and actively maintained, not a one-off trick.
• Breakpoint placement strategy covers the hardware-register checks that API hooking can't reach. Software breakpoints in code the sample doesn't inspect, rather than hardware breakpoints it can detect by reading DR0-DR7.
• Patching the conditional jump covers the timing-based checks. Once the check is located in the disassembly, the branch it controls can be patched directly so the check's result no longer changes execution.

AMSI Evasion Concepts

What AMSI Does

AMSI, the Antimalware Scan Interface, is a real Windows component introduced to close a specific gap: script-based attacks running entirely in memory through PowerShell, JScript, VBScript, or .NET assemblies that never touch disk as a file for traditional antivirus to scan. Windows Defender and most third-party EDR products register as AMSI providers by default, which is why a lot of "in-memory only" PowerShell payloads still get caught without ever writing a file.

Where the Bypass Attacks It

The evasion category built against this is broadly known as an AMSI bypass, and it's public, widely documented technique territory rather than anything obscure -- Microsoft, EDR vendors, and the security research community have all written extensively about the technique class. Conceptually, the most common variant works by patching the in-memory scan function so that it always reports the content as clean, regardless of what's actually being submitted. Because AMSI's scanning logic lives as loaded code inside the calling process's own memory space once the script engine initializes, a script with sufficient privilege can locate that function in memory and overwrite a few bytes so the function returns "not detected" unconditionally, before the script goes on to run its actual payload unscanned. That is the level of detail this chapter goes to: what the technique accomplishes (defeating the scan by disabling the scanner rather than by evading detection logic) and why it works (the scan function is reachable, writable memory in the calling process). Working bypass code is deliberately not reproduced here -- this is a concept a defender needs to recognize, not a snippet to run.

The Registration-Layer Variant

A less common variant skips patching entirely and instead targets the registration layer: standing up a fake AMSI provider, or manipulating how the calling process discovers and loads its provider, so the "real" security product's provider never gets a chance to see the content at all. This is more involved to pull off than patching the scan function directly, and it leaves a different trace -- no in-memory patch signature, but anomalous provider registration or COM activity instead.

The Countermeasure Side

Trying to catch every bypass variant individually is a losing game, since the technique space keeps producing minor variations on the same idea. Three controls work at a different, more durable layer:

EDR Evasion Concepts

How User-Mode Hooking Works

EDR products build most of their visibility on user-mode API hooking. It's efficient and works without requiring a kernel driver on the monitored process's side, which is exactly why it's such a common architecture, and exactly why it has a well-known class of evasion built against it.

Unhooking

Because the EDR's hooks are just modified bytes at the start of each targeted function inside the monitored process's own memory, a sample with sufficient privilege can read a clean copy of the same system DLL (from disk, or by loading a fresh instance from a legitimate source) and overwrite the hooked function bytes with the original, unhooked bytes. Once that's done, calls to the "restored" function go straight to the real kernel transition with no EDR code in the path, and syscalls that used to be logged simply stop appearing in EDR telemetry.

Direct and Indirect Syscalls

A more advanced variant skips the hooked API wrapper entirely. Rather than calling CreateRemoteThread and passing through whatever hooks sit in its path, the code assembles the syscall number and arguments itself and issues the syscall instruction directly, dropping straight into the kernel without ever executing the user-mode function the EDR modified. Indirect syscalls take this a step further by jumping into a clean, unhooked copy of the syscall instruction sequence located elsewhere in memory, which also defeats simplistic call-stack analysis looking for syscalls issued from outside ntdll.dll. Both techniques remove EDR's ability to intercept the call via user-mode hooks entirely, forcing the EDR back onto kernel-level telemetry (ETW, kernel callbacks) as its only remaining visibility.

Why This Mostly Doesn't Matter for Commodity Threats

Unhooking and direct syscalls are real, documented, and effective against a purely hook-based EDR architecture. They are also more engineering effort than most attackers need to spend. The overwhelming majority of intrusions -- commodity ransomware, opportunistic loaders, most phishing-delivered payloads -- succeed against EDR tuning gaps, unmonitored data sources, and analyst alert fatigue long before they need to remove a hook. These techniques show up disproportionately in nation-state tooling and dedicated red-team frameworks built specifically to operate against mature, well-tuned EDR deployments in high-security environments, not in the malware most SOC analysts triage day to day.

The Countermeasure Side

Modern EDR products have already moved defenses onto ground that user-mode unhooking can't reach:

• Kernel-level telemetry. ETW providers (particularly the Threat Intelligence ETW provider, introduced for exactly this reason) and kernel callback registration (process, thread, and image-load notify routines) give the EDR a vantage point below user mode entirely, immune to any patching a user-mode process can do to its own memory.
• Detecting the unhooking itself. An unexpected memory write to a loaded system DLL's executable code section, inside a process with no legitimate reason to modify its own imports, is anomalous regardless of which specific function got restored.
• Watching for the precursor. Both unhooking and direct syscalls require reading and rewriting memory that a hooked API would normally have flagged, so EDR products increasingly surface the setup step: opening a handle to ntdll.dll on disk, or memory-mapping a fresh copy of it, before the unhooking completes.

Fileless Malware and Living-Off-the-Land

Every evasion category covered so far assumes there's a malicious binary somewhere that has to survive scrutiny once it lands. Fileless and living-off-the-land (LOTL) tradecraft sidesteps that assumption by minimizing or eliminating the malicious file altogether, which removes an entire category of detection -- file hashing, static signature matching, on-write AV scanning -- before it ever gets a chance to apply.

Fileless Execution

A payload that never exists as a standalone file on disk, executing instead entirely from memory (reflectively loaded, as covered in Chapter 5's process injection material) or from a script interpreter that pulls its actual logic from a registry value, a WMI class property, or a remote source at runtime, gives static file-based detection nothing to scan. The initial access vector -- a phishing document's macro, a scheduled task, a scripted download cradle -- may still touch disk briefly, but the payload itself lives only in process memory or in a non-file location a file-hash-based control was never designed to inspect.

Living-Off-the-Land Binaries

LOTL tradecraft, covered at length in this site's LOLBAS module, uses Microsoft-signed binaries already present on the system (PowerShell, mshta, regsvr32, rundll32, certutil, and dozens of others) to execute malicious logic under a trusted process's identity. Combined with fileless execution, a LOTL chain can run an entire intrusion -- initial execution, staging, C2 beaconing -- without a single unsigned, unrecognized binary ever appearing in a process listing.

Why the Combination Is Effective

Each technique removes one layer of traditional defense, and each layer removed makes the next one in the chain more effective. That compounding is why sophisticated intrusions stack these categories rather than relying on any single one:

The Countermeasure Side

Because file-based detection has nothing to work with here, the defensive answer shifts almost entirely to behavioral and telemetry-based detection:

Module Capstone: Putting It Together

This chapter closes a module that started with a simple framing: static and dynamic analysis are two lenses applied to the same sample, and most real investigations move back and forth between both. Everything covered since then -- PE structure and strings in Chapter 2, sandbox detonation in Chapter 3, unpacking and memory forensics in Chapter 4, persistence and injection and C2 in Chapter 5, YARA and Sigma rule authoring in Chapter 6, and documented threat actor campaigns in Chapter 7 -- assumed a sample that, however obfuscated, eventually reveals its behavior to a careful analyst. This chapter is the reminder that some samples actively resist that assumption, and a working method for recognizing when one has.

The Throughline: Evasion Checks Are Proxies

Not one technique in this chapter detects what it claims to detect. Each one detects a proxy for it, and every proxy has a corresponding tell:

That's the pattern to carry forward. When a technique claims to defeat a defense, ask what specific mechanism it's actually attacking, because that mechanism's absence or disruption is usually more detectable than the technique's designers intended.

Where to Go Next

• The LOLBAS module's Chapter 8 (Advanced Evasion and Defense) covers AMSI bypass, AppLocker/WDAC limitations, and EDR evasion from the LOLBin-specific angle, with detection queries and Atomic Red Team validation guidance that pairs directly with the concepts introduced here.
• The Threat Hunting module's hypothesis-driven methodology is the natural next step for turning "this sample checks for VM artifacts" into a formal hunt for that behavior across an entire fleet.
• Chapter 6 of this module, revisited with this chapter's evasion categories in mind. A YARA rule matching anti-VM strings or AMSI-bypass API call patterns catches the evasion logic itself as a detection surface, independent of whatever payload it's protecting.

Continuing Education Resources

• MITRE ATT&CK Defense Evasion (TA0005), particularly Virtualization/Sandbox Evasion (T1497) and Impair Defenses (T1562), catalogs most of what this chapter covered with much greater technique-level detail.
• The ScyllaHide project documentation, for the specific anti-debug bypasses it implements.
• Microsoft's own AMSI documentation, for the provider architecture this chapter's AMSI section builds on.