User-mode code runs in ring 3, where the CPU forbids it from touching hardware, reading kernel data structures, or executing privileged instructions. Yet every program eventually needs the kernel to open a file, allocate memory, or create a thread. The bridge across that privilege boundary is the system call: a tightly controlled doorway where user mode hands a request to ring 0, the kernel validates it, does the work, and returns. The boundary exists precisely so that untrusted code cannot corrupt the system — it can only ask, through a narrow interface the kernel fully controls. This article follows a single call all the way from a friendly Win32 API down through ntdll, across the syscall instruction, into the kernel’s dispatcher, and back, and then looks at why this path has become a battleground for endpoint detection and evasion.
The Windows API Is Not the Syscall Layer
A common misconception is that calling a Windows API function is making a system call. It is not. The documented Win32 API — functions like ReadFile, CreateProcess, or VirtualAlloc exported from kernel32.dll and kernelbase.dll — is a stable, backward-compatible abstraction layer. Underneath, those functions eventually call thin stubs in ntdll.dll (the native API, prefixed Nt or Zw), and it is those stubs that actually perform the transition into the kernel.
So a call to ReadFile flows roughly like this: kernel32!ReadFile → ntdll!NtReadFile (the syscall stub) → the kernel’s nt!NtReadFile. The crucial distinction is stability. Microsoft guarantees the Win32 API across releases, but the syscall interface itself is undocumented and unstable: the numbers that identify each kernel service change freely between Windows builds. That is exactly why applications are told to link against kernel32, never to hardcode syscall numbers — and, as we will see, why security tools that rely on those numbers have to work to discover them.
Ntdll Stubs and the syscall Instruction
Each native function in ntdll is a remarkably small stub. Its job is only to identify which kernel service is wanted and to trigger the transition. On x64 the stub loads a System Service Number (SSN) into the EAX register and executes the syscall instruction. A typical stub disassembles to something like this:
; ntdll!NtCreateFile (x64) - the number here is build-specific
mov r10, rcx ; syscall clobbers rcx, so ntdll preserves arg0 in r10
mov eax, 55h ; System Service Number (SSN) for NtCreateFile
test byte ptr [SharedUserData+308h], 1 ; use the alt entry if required
jne short syscall_alt
syscall ; transition to ring 0 (kernel)
ret ; back in user mode, NTSTATUS is in eaxThe syscall instruction is a hardware feature. Rather than looking anything up at runtime, the CPU uses values the kernel programmed into Model-Specific Registers (MSRs) at boot: IA32_LSTAR holds the kernel entry point (nt!KiSystemCall64) that execution jumps to; IA32_STAR supplies the code/stack segment selectors for the privilege switch; and IA32_FMASK specifies which bits of RFLAGS to clear on entry (masking interrupts and the trap flag). In one instruction the processor raises the privilege level, loads the kernel entry point, and begins executing kernel code. On 32-bit x86, the same role was historically played by sysenter, and before that by the software interrupt int 2Eh.
KiSystemCall64 and the SSDT
Execution lands in nt!KiSystemCall64, the kernel’s system-call entry point. Its first responsibilities are housekeeping: switch from the thread’s user stack to its kernel stack, and build a trap frame capturing the caller’s register state so the thread can be resumed exactly where it left off. It then uses the SSN in EAX to look up the target routine.
That lookup goes through the System Service Descriptor Table (SSDT). The kernel keeps a table, nt!KiServiceTable, that maps each service number to the corresponding Nt* function. The SSN is really a composite value: the low 12 bits are the index into the table, and a bit in the number selects which descriptor table to use. There are two:
KiServiceTable— the core executive/kernel services implemented inntoskrnl.exe(file, process, memory, registry operations).W32pServiceTable— the graphics and window-management services implemented inwin32k.sys, reached by GUI syscalls (theNtUser*/NtGdi*functions).
One subtlety worth knowing: on modern x64 Windows, KiServiceTable does not store raw 64-bit function pointers. Each entry is a 32-bit encoded offset relative to the table base (the low bits also encode the argument-stack size). The dispatcher decodes the entry to compute the real routine address. This is both a space optimization and a mild hardening measure, and it is why naively dumping the table as pointers gives nonsense unless you decode it.
Argument Passing and Validation
Arguments follow the standard x64 calling convention: the first four in RCX, RDX, R8, and R9, and any beyond that on the stack. (Because syscall destroys RCX, the ntdll stub first copies the original first argument into R10, and the kernel reads it from there.) But arguments coming from user mode cannot be trusted — a malicious caller could pass a pointer into kernel space or an unmapped address. So before dereferencing any user-supplied pointer, the kernel validates it with ProbeForRead or ProbeForWrite, which confirm the buffer lies entirely within the user address range and is accessible, and then it captures the data into kernel memory so it cannot be changed by another thread mid-operation (avoiding time-of-check/time-of-use bugs).
Whether that validation happens is governed by the thread’s previous mode, a per-thread flag recording where the call originated. This is the essence of the Nt versus Zw distinction. Both names refer to the same service, but a Zw call (used when the kernel itself calls a system service) sets the previous mode to KernelMode, which tells the routine to skip the user-pointer probing because the caller is trusted. An Nt call arriving from user mode leaves the previous mode as UserMode, forcing full validation. Confusing the two inside a driver is a classic source of vulnerabilities.
Returning to User Mode
When the Nt* routine finishes, the kernel restores the caller’s context from the trap frame and executes sysret (sysretq on x64), the counterpart to syscall, which drops the CPU back to ring 3 and resumes the user-mode ret that follows the syscall. The function’s result — an NTSTATUS code such as STATUS_SUCCESS (0x00000000) or STATUS_ACCESS_DENIED (0xC0000022) — is delivered back in EAX/RAX. The higher Win32 layer typically translates that status into a Boolean plus a GetLastError value for the application.
Observing Syscalls
All of this is directly observable. A user-mode debugger can disassemble a stub to read its SSN, and a kernel debugger can walk the service table:
# User mode: disassemble the ntdll stub to see the SSN and the syscall
u ntdll!NtCreateFile L6
# Kernel mode: locate the service table and the dispatcher
x nt!KiServiceTable
u nt!KiSystemCall64 L20
# Dump raw table entries (remember: these are ENCODED offsets on x64,
# not pointers - they must be decoded to get the real Nt* addresses)
dps nt!KiServiceTable L10
# Historical: on very old systems the transition used int 2Eh
!idt 2eFor continuous visibility rather than a one-off look, syscall-adjacent activity is best captured through ETW (Event Tracing for Windows) providers and tools like Sysmon, which record process, file, and registry operations after they cross into the kernel — a more robust vantage point than watching individual user-mode instructions.
Security Relevance
The syscall path is central to the cat-and-mouse game between endpoint security and offensive tooling. Many EDR products historically gained visibility by hooking the ntdll stubs in user mode — overwriting the beginning of functions like NtAllocateVirtualMemory with a jump into the EDR’s inspection code. Because that hook lives in the process’s own ntdll, it can be bypassed by code that avoids the hooked stub entirely:
- Direct syscalls. The attacker assembles their own
mov eax, SSN; syscallsequence and executes it, never touching the hooked ntdll function. The kernel sees a perfectly ordinary system call. - Indirect syscalls. A stealthier variant that still executes the
syscallinstruction located inside ntdll (so the return address points into a legitimate module), while supplying the SSN itself — defeating hooks while looking more like normal control flow.
The catch for the attacker is exactly the instability noted earlier: the SSN differs between Windows builds, so it cannot be hardcoded. Techniques such as sorting ntdll’s exported Nt* functions by address to recover their indices (the idea popularized as “Hell’s Gate” and its successors) resolve the correct numbers dynamically at runtime. Defensively, the response has been to move detection where user-mode code cannot reach it: kernel-mode callbacks (process/thread/image notification routines), the kernel Threat-Intelligence ETW provider, and the fact that PatchGuard protects KiServiceTable from being hooked in the kernel. Understanding the transition in detail is what lets a defender reason about which of these vantage points an evasion technique can and cannot blind. (Everything here is for defensive understanding and authorized testing.)
Conclusion
A system call is the disciplined handoff that makes the user/kernel split workable: an ntdll stub loads a service number and executes syscall, the CPU switches to ring 0 via MSRs into KiSystemCall64, the dispatcher indexes the SSDT to find and call the right Nt* routine, user pointers are validated according to the thread’s previous mode, and sysret carries an NTSTATUS back to ring 3. This single pathway is where the operating system enforces its trust boundary — and, increasingly, where security products and adversaries contest visibility. With the transition mapped, the next article in the series turns to one of the most security-relevant subsystems reached through it: the Windows registry and its Configuration Manager.
You Might Also Like
This article is part of the Windows Internals series. These related deep-dives cover adjacent parts of the system:
Comments