Unpacking & Deobfuscation

Why Detecting a Packer Isn't the Finish Line

What Packer Detection Doesn't Tell You

High entropy, a thin import table, and a suspicious section name are all signals that something is compressing or encrypting most of the file's real content. They don't tell you what that content does. A YARA rule that fires on a UPX signature confirms a packer is present; it says nothing about whether the payload underneath is a keylogger, a ransomware dropper, or a benign installer that happens to use the same packer for legitimate size reduction. Static analysis on a packed file mostly analyzes the packer, not the malware.

Why Unpacking Is Its Own Phase

The real executable code, the real strings, the real import table, all exist somewhere, they're just not sitting in the file the way a disassembler expects to find them. They get reconstructed at runtime, in memory, by a small stub of code that ships inside the packed file specifically to do that reconstruction and then hand control to the real program. If you can catch that handoff, or reproduce what the stub does, you get the same executable an analyst working an unpacked sample would have, and every static analysis technique from Chapter 2 becomes usable again.

What This Chapter Covers

• How packers and crypters build that hidden layer.
• The manual technique of watching a sample unpack itself under a debugger and dumping the result.
• Why fully automated tools can't be trusted to do this reliably against custom or actively maintained packers.
• How to pull plaintext configuration out of the simple obfuscation schemes malware authors actually use.
• The memory forensics fallback that works even when a sample resists every attempt to unpack it cleanly on disk.

Packer and Crypter Mechanics

How a Packer Works

A packer, at its core, does one thing: it compresses the original executable and wraps it in a small stub program. The stub is what actually sits at the file's entry point. When the file runs, the stub decompresses the original executable into memory, resolves whatever imports and relocations that original executable needs, and then transfers control to the original program's entry point, which by this point exists only in memory, not on disk. The original binary was never modified in the sense of having its logic changed. It was compressed, wrapped, and given a loader.

UPX: The Common Example

UPX (the Ultimate Packer for eXecutables) is the example every analyst runs into early, precisely because it's free, fast, and genuinely useful for legitimate size reduction, which means it shows up in both clean software and malware. A UPX-packed sample is straightforward to recognize (it leaves a recognizable section name and a well-documented stub structure) and, not coincidentally, straightforward to reverse: upx -d against a UPX-packed file that hasn't been deliberately corrupted to break the standard unpacker will often just decompress it back to the original. That ease of reversal is exactly why malware authors who do use UPX often modify the stub or corrupt header fields the standard tool depends on, trading some of UPX's convenience for resistance against the one-command fix.

Packer vs Crypter

A crypter is conceptually the same pattern with a different goal emphasized. Where a packer's primary motivation is often shrinking file size (a legitimate goal in its own right), a crypter's primary motivation is defeating static signature detection: antivirus engines and YARA rules that match against known byte patterns in the malware's code find nothing to match, because the code sitting in the file is encrypted and looks like undifferentiated random data until a stub decrypts it at runtime. Compression and encryption aren't mutually exclusive; plenty of packing schemes do both, compress to shrink the file and encrypt (or at least obscure) to defeat signatures, and the line between "packer" and "crypter" in casual usage is more about which effect the author cares about than a hard technical boundary.

Commercial vs Custom Packers

The distinction that matters more for your workflow is commercial or well-known packers versus custom, malware-family-specific ones. A well-known packer, UPX being the standard teaching example, has public documentation, known stub signatures, and existing tooling built specifically to reverse it. A custom packer built for one malware family, or actively maintained by a crimeware-as-a-service operator, has none of that. Nobody's published a one-command unpacker for it, its stub structure changes across builds specifically to defeat static signatures, and it may include anti-analysis checks (Chapter 3 covered sandbox detection; the same techniques show up in packer stubs) that behave differently the moment they detect a debugger attached. Recognizing which category you're dealing with early saves real time: reach for the known tool against a known packer, and expect to do manual work against anything custom.

Manual Unpacking and OEP Identification

The manual technique is the one that works regardless of whether the packer is well-known, custom, or something you've never seen before, and it's built around a single concept: the Original Entry Point, or OEP. The OEP is the address where the real, unpacked program actually begins execution, as opposed to the packer stub's entry point, which is what the file's PE header actually points to on disk.

Loading the Sample

The workflow starts by loading the sample into a debugger, x64dbg and OllyDbg being the two tools most commonly taught and used for this on Windows PE files (OllyDbg is 32-bit only and has seen far less recent maintenance; x64dbg is the more current choice for anything targeting 64-bit code). The sample breaks at its actual entry point, which is the packer stub, not the real program. From there, the analyst lets the stub run, either stepping through it instruction by instruction or setting breakpoints on memory writes and API calls the stub is likely to make (allocation functions, decompression-library calls, or simply watching a region of memory get written to repeatedly as the stub reconstructs the original code there).

Recognizing the OEP: Three Heuristics

Recognizing the OEP once you're near it comes down to a small set of heuristics that show up across most packer stubs, even custom ones, because the underlying job (decompress, then jump) doesn't have many structurally different ways to get done:

• A large, unconditional jump following what looks like a decompression or decryption loop. Packer stubs tend to end with exactly this pattern: a tight loop that's clearly been iterating over memory (rewriting bytes, XORing values, calling a decompression routine), followed by a single jump or call to an address that's nowhere near the stub's own code.
• A sudden change in the entropy or character of a memory region. Before the jump, a region of memory that used to look like meaningless bytes suddenly resolves into something that disassembles as coherent, structured code, real function prologues, real string references, a real import table taking shape. That transition is often visible directly in a debugger's memory or dump view.
• The "tail jump" itself. Many stubs finish with a single jmp to a far-away address rather than a call, precisely because the stub has no further work to do once it hands off; it's not expecting a return.

Here is what that final stretch commonly looks like in a debugger's disassembly pane, illustrating the pattern rather than any specific real sample:

00401A02  MOV   ECX, DWORD PTR [EBP-4]
00401A05  XOR   BYTE PTR [ESI+ECX], AL
00401A08  INC   ECX
00401A09  CMP   ECX, DWORD PTR [EBP-8]
00401A0C  JB    00401A02          ; decompression/decode loop
00401A0E  MOV   EAX, DWORD PTR [EBP-C]   ; EAX now holds the OEP address
00401A11  JMP   EAX               ; tail jump -> Original Entry Point
00401A13  ; --- execution below this line is the real, unpacked program ---

Dumping and Rebuilding the Import Table

Once the debugger has stepped past that jump, execution is sitting inside the real program for the first time, still resident only in memory. The next task is dumping that memory region to disk as a usable PE file, and this is harder than it sounds, because a straight memory dump won't have a correct import table; the packer stub typically resolved the imports itself at runtime rather than leaving the normal import table structure intact. Scylla is the tool most commonly used to handle this: it can locate the OEP, rebuild the import table by walking the resolved addresses in memory and matching them back to the DLLs and functions they belong to, and write out a dump that behaves like a normal PE file a disassembler can open cleanly. Manual PE reconstruction without a tool like Scylla is possible but tedious, since it means manually walking the import address table and rebuilding section headers by hand.

Automated and Semi-Automated Unpacking

How Automated Unpacking Works

Manual unpacking works but takes real time and real skill, which is why generic unpacking tools exist to handle the common case automatically. These tools generally work by recognizing known packer stub signatures (the same kind of signature-matching a static analysis tool uses to detect that a file is packed in the first place), then running the packer's own logic to completion in a controlled way and extracting the result, sometimes using OEP-detection heuristics similar to the ones described above but scripted rather than done by hand. Against a well-known packer used in its default, unmodified form, this can turn a manual process that takes an analyst twenty minutes into something that finishes in seconds.

Where It Breaks Down

The limits show up exactly where you'd expect: a signature-based generic unpacker only recognizes packers it has a signature for, and any packer, custom or well-known, can be modified specifically to break that recognition. A malware operator who values a packer's resistance to automated tooling has every incentive to alter the stub in ways that don't change its function but do change the bytes a signature would match against. Malware-family-specific packers are frequently built with exactly this adversarial relationship to automated unpacking tools in mind: they exist, in part, specifically because generic tools can't handle them, and a family that gets a reliable automated unpacker published against it tends to get its packer updated shortly after.

Manual Technique as Necessary Fallback

This is the practical reason manual unpacking technique remains a necessary skill rather than a legacy one. An automated tool is the right first attempt against any packed sample, since it costs almost nothing to try and succeeds often enough to be worth running by default. But treating a failed or incomplete automated unpack as the end of the analysis, rather than the point where manual technique takes over, means giving up on samples specifically because their authors invested effort in resisting the easy path, which correlates poorly with those samples being unimportant.

String and Configuration Deobfuscation

Why Configuration Data Gets Obfuscated

Once a sample is unpacked, or even before, a separate obfuscation problem often remains: the malware's own configuration data, C2 domains, campaign or affiliate identifiers, encryption keys used for its actual network traffic, is frequently stored in the binary using its own lightweight obfuscation scheme, independent of whatever packer wrapped the file. This is a deliberate and rational design choice on the malware author's part. Strong cryptography is unnecessary for this purpose; the goal is defeating a casual strings dump and generic automated scanners, not resisting a determined analyst with time to spend. A single-byte XOR key, a short repeating XOR key, or base64 encoding layered on top of an XOR pass are all common enough to expect by default.

Known-Plaintext Recovery

The analytic approach to recovering plaintext from these schemes leans on the same weakness that makes them cheap to implement: they're simple, which means they leave detectable structure behind. A single-byte XOR key applied across a long stretch of data produces repeating byte patterns wherever the underlying plaintext repeats, and known-plaintext guessing is often the fastest way in. If you strongly suspect a decoded string is a URL, because the encoded blob sits next to network-related code or gets passed into a socket or HTTP function, you can guess that the plaintext starts with http:// or https://, XOR the first few encoded bytes against that guessed plaintext, and see what key byte or key sequence falls out. If the same key then correctly decodes other strings in the same blob into readable text, you've recovered the scheme without needing to reverse the encoding routine's code at all.

A worked, illustrative example of that single-byte case:

Encoded bytes (hex):     1F 0B 0B 06 3A 3E 3E 66 62 78 79 6D 6C 6E 2E 78 79 7B

Guessed plaintext start: "http://"

Byte-by-byte XOR against guess:
  0x1F ^ 'h' (0x68) = 0x77
  0x0B ^ 't' (0x74) = 0x7F
  0x0B ^ 't' (0x74) = 0x7F
  0x06 ^ 'p' (0x70) = 0x76
  ...

Recurring candidate key byte: 0x77

Applying 0x77 across the full blob:
  1F^77=68 'h'   0B^77=7C ... (adjust guess, key candidate refined)
  -- once the correct single-byte key is confirmed, it decodes cleanly to:
  "http://cdn-update[.]net/gate.php"

The numbers above are constructed to illustrate the method, not transcribed from a real sample. The point is the workflow: guess plaintext, derive a candidate key from the difference, confirm the key against the rest of the blob, and treat any string it fails to decode cleanly as a sign the scheme uses a longer or multi-byte key rather than a single repeating byte.

CyberChef for Fast Iteration

CyberChef is the tool most analysts reach for to run this kind of experimentation quickly rather than writing a one-off script for every sample: its XOR, base64, and "From Hex" operations can be chained together in a browser-based pipeline, and because it updates output live as you adjust a key or reorder operations, testing several candidate keys or a base64-then-XOR versus XOR-then-base64 ordering takes seconds rather than a script rewrite each time. For a repeating multi-byte key rather than a single byte, the same known-plaintext logic still applies, you're just recovering more key bytes before the pattern confirms itself, and a long enough known-plaintext guess (a full expected URL rather than just a scheme prefix) makes that recovery faster.

Custom Substitution Schemes

Custom substitution schemes, where each byte or character maps to another through a fixed lookup table rather than a mathematical operation like XOR, are less common but do turn up, particularly in malware families that specifically want to avoid the "looks like XOR" pattern-recognition an analyst develops over time. These require locating and reversing the substitution routine itself in the disassembly rather than guessing your way in from known plaintext, since there's no simple byte-difference relationship to exploit. This is meaningfully more work, and it's usually only worth the investment once you've confirmed simpler schemes don't apply.

Where Memory Forensics Picks Up What Disk-Based Analysis Misses

When Disk-Based Unpacking Isn't Enough

Every technique so far in this chapter assumes you can get a stable, unpacked file to disk, or at least reconstruct one from a memory dump taken at the right moment. Some samples make that assumption expensive to satisfy: packers with aggressive anti-debug checks that detect and alter behavior the instant a debugger attaches, multi-stage loaders that never write a fully unpacked second stage to disk at all, or genuinely fileless techniques that keep the payload resident only in memory from the moment of infection onward, with nothing corresponding to "the packed file" ever sitting on the disk in a form worth unpacking.

Why It's Still Beatable

The property that makes all of these approaches eventually beatable is the same one that made this whole chapter possible: to actually run, code has to exist in a directly executable, unpacked form in memory at some point, no matter how well the file on disk resists analysis. A sample can stay encrypted on disk forever and it still has to decrypt itself into memory before the CPU can execute it. That's the opening memory forensics exploits, and it's a fundamentally different angle than anything covered so far: instead of catching the unpacking moment live in a debugger, you acquire the memory of a system where the sample has already run, and analyze it after the fact.

Volatility: Memory Analysis Framework

Volatility is the framework most widely used for this kind of analysis. Given a memory image, whether from a live acquisition tool or a virtual machine snapshot, Volatility can enumerate running and terminated processes, walk a process's memory regions looking for characteristics consistent with an unpacked, injected, or otherwise anomalous executable image (a region marked executable that doesn't correspond to any file mapped from disk, for instance), and extract that region as its own file for the same kind of static analysis Chapter 2 covered, strings, PE structure, hashing, now working against code that was never available in that form anywhere on disk. Plugins built specifically for this purpose can scan a memory image for process regions that look like a fully formed, relocatable PE image sitting somewhere other than where the legitimate module list says it should be, which is exactly the signature a lot of injected or in-memory-only payloads leave behind.

This matters most for exactly the cases where the debugger-based technique struggles: fileless malware that has no packed file to catch mid-unpack, and heavily packed samples with anti-analysis checks tuned specifically to detect the debugging environment the manual technique depends on. A memory acquisition can, in principle, be taken from a system running the malware under close-to-normal conditions, sidestepping some (not all) anti-debug logic that specifically watches for a debugger process attached to it, since the acquisition tool isn't attaching a debugger to the malicious process at all.

Memory forensics as a discipline goes considerably further than this one use case, covering process injection artifacts, credential material resident in memory, and full incident timeline reconstruction from a compromised host's RAM. H3AD-LEARN's Digital Forensics domain covers that broader memory forensics skill set in depth; the coverage here is scoped specifically to how it functions as an unpacking fallback when disk-based technique hits a wall.

Debugger-Based vs Memory-Based: When to Use Each

These are complementary tools for different situations, not competing techniques.

Bringing the Workflow Together

Laid out end to end, the path this chapter covers runs from packer detection through to recovered configuration data, with manual technique and memory forensics as the fallbacks each earlier step can hand off to.

Every technique in this chapter exists to answer one question: what is this sample actually made of, once you strip away the layer built specifically to keep you from seeing it. Chapter 5 assumes you can answer that question and moves the focus forward, from getting to a sample's real code to understanding what that code actually does once it's running: how it persists across a reboot, how it injects itself into other processes, and how it talks to its command-and-control infrastructure.