Unpacking Packed Malware: From UPX to Custom Packers

Malware & C2
2025.10.28
Time it takes to read this article 6 minutes.

Disclaimer. This article is for education and authorized testing only. Analyze malware exclusively inside an isolated lab you own or are contracted to assess. Detonating live samples on a corporate or personal network, or outside the scope of an engagement, may be illegal and dangerous. Always know your authorization boundary.

Introduction

Most commodity malware ships packed: the real payload is compressed or encrypted on disk and reconstructed in memory at runtime by a small stub. Packing serves two goals at once — it shrinks the binary and it defeats naive static analysis and signature matching. Before you can meaningfully reverse a sample in Ghidra or IDA, you usually have to get it into its unpacked, in-memory form and dump it back to disk.

This post walks through the full manual unpacking workflow: spotting a packer with entropy, finding the Original Entry Point (OEP), dumping the process, and rebuilding the Import Address Table (IAT) with Scylla in x64dbg. We start with UPX (the easy case) and then generalize to custom packers.

How packers work

A packer wraps the original program. On disk the PE looks normal-ish, but the entry point now points at the unpacking stub instead of the original code. At runtime the stub:

  1. Allocates or maps memory for the decompressed payload.
  2. Decompresses/decrypts the original sections into that memory.
  3. Rebuilds the import table (resolves API addresses) and applies relocations.
  4. Jumps to the OEP — the original program's true first instruction.

That final jump is the moment we care about. If we let the stub finish its work and break exactly at the OEP, memory contains a fully reconstructed image we can dump. This maps to MITRE ATT&CK T1027.002 (Software Packing).

Telltale signs of packing:

  • High entropy. Compressed/encrypted data approaches 8.0 bits/byte. Normal code sections sit around 5.5–6.5.
  • Few or weird imports. A packed binary often imports only LoadLibraryA, GetProcAddress, VirtualAlloc, and VirtualProtect.
  • Odd section names. UPX0, UPX1, .packed, or sections with VirtualSize far larger than RawSize.
  • A tiny .text and a huge writable+executable section.

Prerequisites / Lab setup

  • An isolated Windows VM (no shared folders, host-only or no network) with a snapshot you can revert.
  • x64dbg with the Scylla plugin (bundled by default).
  • Detect It Easy (DIE) for packer/entropy detection.
  • upx from the official UPX project for the demo sample.
  • PE-bear or CFF Explorer for inspecting headers.

Build a harmless packed test binary so you can practice without touching live malware:

# On the lab VM (or any box) — pack a benign EXE with UPX
upx --best --ultra-brute calc_copy.exe
upx -t calc_copy.exe   # verify it is a valid UPX file
Bash

Walkthrough: unpacking UPX

UPX is the friendliest case and a good warm-up.

1. Confirm the packer and measure entropy

# Detect It Easy CLI
diec.exe calc_copy.exe
# -> PE32, packer: UPX(4.x)[NRV,best], entropy: ~7.9
Bash

A section entropy near 8.0 plus UPX0/UPX1 section names is conclusive. You can also script entropy yourself:

import math, sys
data = open(sys.argv[1], "rb").read()
freq = [0] * 256
for b in data:
    freq[b] += 1
ent = -sum((c/len(data)) * math.log2(c/len(data)) for c in freq if c)
print(f"entropy = {ent:.3f} bits/byte")
Python

2. The cheap path

For genuine UPX, the easiest unpack is simply:

upx -d calc_copy.exe   # decompress in place
Bash

But malware authors routinely tamper with UPX headers (zeroing the l_info magic or corrupting block sizes) so upx -d fails with "NotPackedException" or "CantUnpackException". When that happens, you unpack manually — which is the skill that transfers to every other packer.

3. Manual unpacking in x64dbg

Open the packed binary in x64dbg. The debugger breaks at the system breakpoint (TLS/ntdll), then at the module entry point — the UPX stub.

UPX's stub classically starts with PUSHAD (save all registers) and ends, right before jumping to the OEP, with POPAD followed by a JMP to a far-off address. We exploit that with a hardware breakpoint on the stack, the ESP/Stack trick:

1. Step over (F8) the first PUSHAD instruction.
2. Note the value of RSP (it now points to the saved register block).
3. In the dump pane: "Follow in Dump" -> RSP.
4. Right-click the first 4/8 bytes -> Breakpoint -> Hardware -> Access -> Dword/Qword.
5. Run (F9). When POPAD reads that memory back, the HW breakpoint fires.
6. You land on (or one step from) the JMP to the OEP. Step into it.
Plaintext

You are now sitting at the OEP. In a real PE this typically looks like a compiler stub: a call __security_init_cookie or a jump into the CRT startup (__scrt_common_main_seh).

4. Dump and rebuild imports with Scylla

With execution paused at the OEP:

  1. Open Plugins -> Scylla (or press the Scylla toolbar icon).
  2. Click IAT Autosearch — Scylla scans for the import table the stub just rebuilt. Confirm the advanced search if prompted.
  3. Click Get Imports. Review the list; resolve or cut any entries flagged with ? (invalid pointers).
  4. Click Dump to write the in-memory image to disk (e.g. calc_copy_dump.exe).
  5. Click Fix Dump and select the file you just dumped. Scylla appends a new import section and rewrites the PE entry point to your OEP.

The result, calc_copy_dump_SCY.exe, is a standalone unpacked binary you can load into Ghidra or IDA.

Custom packers: same idea, more work

Custom/commercial packers (ASPack, themed crypters, .NET reactors) don't give you a clean PUSHAD/POPAD pair. The general algorithm still holds — let it unpack, then catch the OEP — but you need a reliable OEP-finding strategy:

  • Memory breakpoints on execute. After the stub allocates a new RWX region with VirtualAlloc, set a breakpoint on VirtualProtect (the stub usually flips the payload to executable just before jumping). Break there, then set a memory-execute breakpoint on the freshly protected page.
  • Section-hop detection. Tools like the OllyDumpEx / ScyllaHide ecosystem and x64dbg's bp on the destination section catch the transfer from the stub section into the payload section.
  • API call tracing. Set breakpoints on LoadLibraryA/GetProcAddress; the last burst of resolutions usually precedes the jump to OEP.
  • Anti-debug. Custom packers check PEB.BeingDebugged, NtGlobalFlag, timing (rdtsc), and NtQueryInformationProcess(ProcessDebugPort). Use ScyllaHide to mask the debugger before you start.
# x64dbg command bar — break right before the typical "make executable -> jump" pattern
bp VirtualProtect
bp VirtualAlloc
run
Plaintext

Once you reach the OEP, the dump + Scylla workflow is identical.

Unpacking workflow

Unpacking Packed Malware: From UPX to Custom Packers diagram 1

The diagram shows the decision path: detect packing, try the easy automated route, fall back to manual OEP hunting in x64dbg, then dump and rebuild imports before deep static analysis.

For follow-up static work, see Getting Started with Ghidra and Reversing C2 Beacons. For broader triage, see Malware Triage with YARA.

Detection & Defense (Blue Team)

Packing is both an offensive technique and a high-value detection signal. Treat the presence of a packer as suspicious by default.

Detection

  • Entropy at scale. Flag PE files whose section entropy exceeds ~7.2 and whose .text is unusually small. YARA can express this directly:

    import "math"
rule HighEntropy_Section {
  condition:
    for any s in pe.sections : (
      math.entropy(s.raw_data_offset, s.raw_data_size) >= 7.2
    )
}
    Plaintext

  • Section anomalies. Alert on UPX0/UPX1 names, on writable+executable sections, and on VirtualSize >> SizeOfRawData. Sysmon Event ID 7 (Image Loaded) plus image-load signing checks surface unsigned packed modules.

  • Runtime behavior. The VirtualAlloc -> write -> VirtualProtect(PAGE_EXECUTE_READWRITE) -> jump sequence is classic self-modifying behavior. EDR telemetry on RWX allocations and on execution from non-image-backed memory (T1055-style) catches unpacking even when the disk artifact is novel.

  • Sysmon Event ID 1 (Process Create) combined with Event ID 8 (CreateRemoteThread) and Event ID 10 (ProcessAccess) highlights unpacked payloads injecting into other processes.

Defense & mitigation

  • Application allowlisting (Windows Defender Application Control / AppLocker) blocks unsigned binaries from executing regardless of how they are packed.
  • Attack Surface Reduction rules and exploit protection (ASLR, DEP/NX, CFG) raise the cost of memory-executing stubs.
  • Memory scanning. Defender and modern EDRs scan process memory post-unpack, where signatures match the decompressed payload that disk-only AV missed.
  • Network egress controls. Even a perfectly unpacked beacon is useless to the attacker if C2 callbacks are blocked by egress filtering and DNS monitoring.

The core blue-team insight: you cannot reliably signature the packed disk artifact, so detect on entropy + structure statically and on unpacking behavior dynamically.

Conclusion

Manual unpacking is a small set of repeatable moves: confirm the packer with entropy and section inspection, let the stub run, catch the OEP (stack trick for UPX, VirtualProtect breakpoints for custom packers), dump with Scylla, and rebuild the IAT. Master it once on UPX and the same muscle memory carries over to nearly every crypter you'll meet in the field. Defenders, meanwhile, should lean into the fact that packing is loud — high entropy and RWX self-modification are gifts that static and behavioral detection can both exploit.

References

You Might Also Like

If you found this useful, these related deep-dives cover adjacent techniques and their defenses:

Comments

Copied title and URL