Disclaimer: This article is provided strictly for educational purposes and authorized security testing. Only run these techniques against systems you own or have explicit written permission to assess. Unauthorized access to computer systems is illegal in virtually every jurisdiction and can carry severe penalties.
Introduction
Most malware that reaches an analyst’s desk does not arrive as clean, readable code. It is packed, obfuscated, and salted with anti-analysis tricks whose only purpose is to slow down or mislead the person trying to understand it. Anti-debugging is the family of runtime checks a sample uses to notice it is being watched by a debugger and change its behaviour — typically by exiting, looping forever, or executing a benign decoy path. Packing is the complementary technique of compressing or encrypting the real payload and unpacking it into memory only at runtime, so that static analysis of the on-disk file reveals almost nothing.
For a defender or malware analyst these are obstacles to be understood and removed, not deployed. This article looks at how the common anti-debugging checks work, why they are shallow enough to defeat, and a practical workflow for unpacking a sample to recover its original entry point (OEP) and a clean, analyzable payload. The same knowledge lets detection engineers write better behavioural signatures, because the anti-analysis behaviour is itself a strong indicator of maliciousness.
Attack Prerequisites
To analyse a hostile binary safely you need a controlled environment and the right vantage point:
- An isolated analysis lab — a snapshot-capable VM with no production network access (host-only or a simulated internet such as INetSim/FakeNet), because unpacking runs the sample.
- A debugger and disassembler — x64dbg/x32dbg for dynamic work, Ghidra or IDA for static review, and a PE inspection tool (PE-bear, CFF Explorer).
- The ability to recognise the packer — entropy analysis, section names (
UPX0/UPX1,.aspack), and import-table sparseness tell you what you are dealing with before you start.
How It Works
Anti-debugging checks fall into a few well-worn categories. The simplest query a flag the operating system maintains for every process: on Windows the PEB (Process Environment Block) contains a BeingDebugged byte that the IsDebuggerPresent API simply reads. Slightly deeper checks read NtGlobalFlag in the PEB or call CheckRemoteDebuggerPresent / NtQueryInformationProcess with the ProcessDebugPort class, which the kernel populates when a debugger is attached. Timing checks measure how long a short code sequence takes using rdtsc or GetTickCount: single-stepping through it in a debugger inflates the elapsed time far beyond what native execution would produce. Finally, code-integrity checks scan the process’s own memory for the 0xCC (INT3) byte a debugger writes when it sets a software breakpoint.
The reason these checks are defeatable is that they are all just data or instructions the analyst also controls. BeingDebugged is a single byte the analyst can zero out; a timing check is a conditional jump the analyst can patch or step over; a breakpoint scan only matters if you use software breakpoints, so a hardware breakpoint sidesteps it entirely. Packing works on a different axis: the file on disk is a small *stub* plus an encrypted blob. When the process starts, the stub allocates memory, decrypts or decompresses the blob into it, fixes up imports, and then transfers control to the recovered code — the OEP. Catch the process at that transfer and you can dump the now-decrypted image.
Practical Example / Configuration
Recognising a packed sample usually starts with entropy and section layout. High entropy (close to 8.0 bits/byte) across a section is the signature of compression or encryption. A quick triage in Python using pefile:
import pefile, math
def entropy(data):
if not data: return 0
freq = [0]*256
for b in data: freq[b] += 1
return -sum((c/len(data)) * math.log2(c/len(data))
for c in freq if c)
pe = pefile.PE('sample.bin')
for s in pe.sections:
name = s.Name.rstrip(b'\x00').decode(errors='replace')
print(f'{name:8} entropy={entropy(s.get_data()):.2f} '
f'vsize={s.Misc_VirtualSize:#x} rawsize={s.SizeOfRawData:#x}')
# A section with entropy ~7.9 and vsize >> rawsize is almost certainly packed.
For a classic UPX-packed binary the cleanest route is the tool that packed it. UPX ships a symmetric unpacker:
# Identify then statically unpack UPX
upx -l sample.bin # lists the compressed/uncompressed sizes
upx -d -o unpacked.bin sample.bin
Custom or tampered packers (where the UPX magic has been stripped so upx -d refuses) require manual unpacking in a debugger. The general procedure is to run the stub until it hands control to the OEP, then dump:
1. Load in x64dbg; note the entry point sits in the packer stub.
2. Find the tail jump to the OEP. Common tactics:
- Set a hardware breakpoint (BP on access) on the section the stub
writes the decrypted code into, so a breakpoint-scan can't see it.
- Use the 'run to user code' / ScyllaHide 'trace until' features.
3. When execution lands in freshly written memory with a normal-looking
function prologue, you are at the OEP.
4. Dump the process with Scylla, then 'IAT Autosearch' + 'Get Imports'
to rebuild the import address table, and 'Fix Dump' to write a
runnable, analyzable PE.
Anti-debugging is best neutralised with a plugin rather than by hand. ScyllaHide (for x64dbg) hooks the common checks — IsDebuggerPresent, NtQueryInformationProcess, NtSetInformationThread(ThreadHideFromDebugger), rdtsc — and returns the answers a non-debugged process would, so most samples run straight through their guard clauses.
Note: Timing checks are the trickiest to beat manually because they are stateful: patching one
rdtscdelta is easy, but some samples take a baseline early and compare much later. When a sample keeps bailing out at the same address despite PEB patches, suspect a timing or aGetTickCount/QueryPerformanceCountercomparison and neutralise the conditional jump directly rather than chasing the API.
Opsec / Lab hygiene: Malware frequently checks for the analysis environment itself — VM artifacts (registry keys, MAC OUI ranges, driver names), known analysis tool process names, or a low core/RAM count. Harden the lab by editing obvious VM artifacts, giving the VM realistic specs, and always revert to a clean snapshot between samples so residual state never contaminates the next analysis.
Detection and Defense
From a detection-engineering standpoint, anti-analysis behaviour is a gift: legitimate software rarely checks whether it is being debugged or unpacks itself from a high-entropy blob. Defenders can turn these behaviours into signal:
- Behavioural / EDR — flag processes that allocate
RWXmemory, write executable code into it, and then execute from it (the runtime-unpacking pattern), especiallyVirtualAlloc(PAGE_EXECUTE_READWRITE)followed by execution in the same region. - Static / hunting — high section entropy, a tiny import table (packers resolve imports at runtime), abnormal section names, and a raw size far smaller than the virtual size are cheap YARA-able indicators.
- API telemetry — repeated calls to
IsDebuggerPresent,CheckRemoteDebuggerPresent,NtQueryInformationProcess(ProcessDebugPort), andNtSetInformationThread(ThreadHideFromDebugger)are anomalous in benign line-of-business apps. - Sandbox correlation — a sample that behaves differently under instrumentation than on bare metal is exhibiting evasion; multi-environment detonation exposes it.
Real-World Impact
Packing and anti-analysis are near-universal in commodity and targeted malware alike — from crypters sold on underground forums to the custom loaders fronting ransomware and banking trojans. The techniques catalogued here map directly to MITRE ATT&CK’s Defense Evasion tactic, including Obfuscated/Compressed Files (T1027), Software Packing (T1027.002), and Debugger Evasion (T1622). Because analysts must routinely defeat them to produce indicators of compromise, unpacking is a core skill in any malware reverse-engineering or DFIR role.
Conclusion
Anti-debugging and packing raise the cost of analysis, but they are speed bumps rather than walls: the checks are shallow, and the act of unpacking always exposes the payload in memory eventually. A disciplined workflow — recognise the packer by entropy and imports, neutralise anti-debug tricks with ScyllaHide, catch the OEP, dump and rebuild — recovers a clean sample, and the evasion behaviour itself becomes some of the most reliable material for detection.
You Might Also Like
If you found this useful, these related deep-dives cover adjacent techniques and their defenses:
Comments