Spend enough time staring at debuggers and disassemblers, and you stop seeing memory as a flat, abstract space. The stack is a manicured garden—predictable, orderly, and increasingly walled off by compiler-level defenses. The heap, though? That’s the wild jungle. It’s a chaotic, dynamic arena where chunks of memory are carved out, tossed back, and recycled in ways that create weird, emergent patterns. And for decades, exploit developers have been hacking trails through that undergrowth.
Heap exploitation isn’t just about smashing the stack with a different register. It’s a discipline that forces you to internalize the allocator’s logic, to see the invisible metadata stitching the system together, and to twist the very algorithms that manage memory. The reason this field never sits still is straightforward: every time a new defense drops, it reshapes the terrain, and attackers have to find fresh—often more elegant—paths to code execution.
The Allocator as a State Machine
To get why heap exploitation is a moving target, you have to stop thinking of the heap as a dumb pile of bytes. A modern allocator like glibc’s ptmalloc is a state machine. It juggles free lists (fastbins, tcache, smallbins, largebins, the unsorted bin), coalesces adjacent free chunks to fight fragmentation, and manages a whole arena system for multi-threaded performance. Every allocation and deallocation is a state transition, and every transition is an opportunity.
Early heap exploits were blunt instruments. Take the classic unlink attack from the early 2000s. You’d corrupt the forward (fd) and backward (bk) pointers of a free chunk. When the allocator later unlinked that chunk from its doubly-linked free list, your crafted corruption would trigger an arbitrary write. The old unlink macro did something like FD->bk = BK; BK->fd = FD;. If you controlled the chunk’s fd and bk, you could write a value you controlled to an address you controlled. It was a clean primitive, but also a noisy one. The fix was a sanity check: before unlinking, verify that P->fd->bk == P and P->bk->fd == P. That one check, rolled out in glibc 2.3.4, killed the classic unlink attack practically overnight.
But the game didn’t end. It just moved up a layer of abstraction.
Metadata Mayhem: From Unlink to Unsorted Bin
With the front door bolted shut, exploit writers started checking the windows. The unlink check only protected the integrity of the doubly-linked free lists. What about the rest of the metadata? The allocator’s logic for sorting chunks into bins, carving out new allocations from larger free chunks, and consolidating adjacent free chunks all leaned on metadata that was still, in many cases, writable.
This kicked off the era of the unsorted bin attack. The idea was devilishly simple: corrupt the bk pointer of a chunk sitting in the unsorted bin. When the allocator iterates through the unsorted bin to service a request, it writes a pointer to the main arena into the location pointed to by that corrupted bk. You get a powerful, if somewhat wild, write primitive—a large, known value (a libc address) written to an arbitrary spot. This was often used to overwrite _IO_list_all or corrupt global_max_fast, setting the stage for a fastbin attack.
The fastbin itself became a prime target. Fastbins are singly-linked lists of small, freed chunks, built for speed. They have minimal security checks. A fastbin corruption attack overwrites the fd pointer of a freed fastbin chunk to point to a fake chunk. A series of allocations then hands you that fake chunk, giving you control over an arbitrary memory region. This technique was the workhorse of heap exploitation for years, enabling everything from House of Force to House of Spirit.
The Counter-Revolution: Hardened Allocators
The defenders weren’t sleeping. The glibc maintainers and the wider security community started systematically hardening the heap. A wave of patches added integrity checks to the very metadata attackers had been corrupting. The era of modern heap hardening had arrived, and it forced a fundamental shift in exploitation strategy.
One of the biggest changes was the tcache (thread-local cache) in glibc 2.26. The tcache is a per-thread cache of freed chunks, designed for raw speed. It sits in front of the main fastbins and smallbins. For exploit developers, the tcache was a double-edged sword. On one hand, it was a much simpler structure with almost no security checks in its initial implementation—a single-linked list with no integrity checks on its fd pointer. This made tcache poisoning (overwriting the fd pointer to get an arbitrary allocation) trivially easy. On the other hand, the tcache’s presence meant many classic techniques targeting the main bins stopped working, because freed chunks would get swallowed by the tcache first.
Glibc 2.29 then introduced a key defense for the tcache: a simple pointer mangling check on the tcache fd pointer. The stored fd value is now XORed with a per-thread random key and the address of the chunk itself. This was a direct counter to trivial tcache poisoning. To bypass it, you now need an information leak to disclose the heap base address and the tcache key, or you need to find a way to corrupt the tcache entry without touching the mangled pointer. This single change pushed people toward more complex techniques, like House of Lore or corrupting the tcache count to return a chunk from the wrong bin.
Similarly, the unsorted bin attack was mitigated by adding a simple check: the corrupted bk pointer must now point to a valid, writable location whose own fd pointer is a valid unsorted bin chunk. This made the classic unsorted bin attack—used to write a large value anywhere—much harder to pull off. Attackers adapted by chaining it with other primitives or moving to smallbin and largebin corruption techniques, which had their own, more involved, set of checks.
The Rise of the House of Lore and File Stream Exploitation
As the low-hanging fruit of metadata corruption got picked clean, the focus shifted to more esoteric attack surfaces. The “House of Lore” is a perfect example. This technique targets the smallbin allocation path. By corrupting the bk pointer of a chunk in the smallbin, you can trick the allocator into returning an arbitrary memory region as a chunk. The checks here are more subtle: the corrupted bk pointer must point to a location where you can craft a fake chunk with a valid bk pointer that points back to the original smallbin chunk. It’s a delicate dance of pointer crafting, but when it works, it gives you a powerful arbitrary-write primitive.
Another frontier that’s seen intense research is the exploitation of _IO_FILE structures. The standard I/O library in glibc uses a complex web of structures, function pointers, and virtual tables (vtables). By corrupting a FILE structure in memory—often through a heap overflow or an arbitrary write—you can hijack control flow when the corrupted stream is flushed, closed, or even during a call to malloc or free if the stderr stream is used. The “House of Orange” technique famously combined a heap overflow with a corrupted _IO_list_all pointer to trigger a chain of fake FILE structures, ultimately calling system(). While glibc has since hardened the vtable checks, the attack surface of the FILE structure remains a rich area for research, with techniques like FSOP (File Stream Oriented Programming) continuing to evolve.
The Allocator Itself as a Target: House of Mind and Beyond
Some of the most creative techniques don’t just corrupt chunks; they corrupt the allocator’s internal state variables. The “House of Mind” attack is a classic example. It targets the arena structure in glibc, specifically the fastbin array pointer within the main arena. By crafting a fake arena and tricking the allocator into using it, you can redirect fastbin operations to a memory region you control. This requires a deep understanding of how glibc manages multiple arenas for multi-threaded programs, but it provides a very clean and powerful exploitation path.
More recently, researchers have been probing the boundaries of the allocator’s own internal logic. The “House of Einherjar” technique, for example, exploits the chunk consolidation process. By corrupting the prev_size field of a chunk and setting the PREV_INUSE flag to zero, you can force the allocator to consolidate a chunk with a fake previous chunk, leading to an overlapping chunk scenario. This technique is a direct assault on the allocator’s bookkeeping, and it requires a precise understanding of how the unlink_chunk macro validates the backward pointer during consolidation. The checks are strict, but with a controlled leak, they can be satisfied.
Modern Defenses and the Shifting Battlefield
The cat-and-mouse game continues. Modern glibc versions have introduced even more stringent checks, such as pointer mangling for the fd and bk pointers in the smallbin and largebin, and a more hardened safe unlinking mechanism. The tcache has seen multiple rounds of hardening, including a check that the chunk being freed isn’t already in the tcache (a double-free check) and a key-based mechanism to detect double frees.
These defenses have pushed exploit development into new territory. The focus is now on logic bugs in the allocator itself, race conditions in multi-threaded programs, and the exploitation of other heap allocators entirely. Many high-performance applications use custom or alternative allocators like jemalloc or tcmalloc, which have their own internal structures and quirks. The principles of heap exploitation—understanding the allocator’s state machine, corrupting metadata, and hijacking control flow—remain the same, but the specific techniques are in constant flux.
Another growing trend is the use of heap manipulation to achieve more subtle goals, like type confusion or out-of-bounds access, rather than direct control flow hijacking. In a world of ubiquitous CFI (Control Flow Integrity) and PAC (Pointer Authentication Codes), corrupting a function pointer is harder than ever. Instead, attackers might corrupt a length field or a vtable pointer to an object to gain a read/write primitive that can be used to bypass ASLR or leak sensitive data. The heap is no longer just a stepping stone to shellcode; it’s the primary battlefield for achieving code reuse and data-only attacks.
FAQ: Heap Exploitation in the Modern Era
Why is the heap a more attractive target than the stack for modern exploits?
The stack is heavily protected by defenses like stack canaries, ASLR, and NX, which are relatively straightforward to implement. The heap, with its complex management structures and dynamic nature, presents a much larger attack surface. The variety of allocator states and metadata provides multiple avenues for corruption, making it harder to defend comprehensively.
What is the most significant defense that has changed heap exploitation?
The introduction of the tcache in glibc 2.26 and its subsequent hardening (like the safe-linking pointer mangling in 2.32) have been game-changing. The tcache altered the performance characteristics of the allocator, making many classic fastbin and unsorted bin attacks obsolete, while also introducing new, initially unprotected, attack surfaces that were later fortified.
Are heap exploitation techniques only relevant to glibc’s ptmalloc?
No. While ptmalloc is the most widely studied due to its use in Linux, the principles apply to any heap allocator. Windows’ Low Fragmentation Heap (LFH), jemalloc (used in FreeBSD and Firefox), and tcmalloc (used in some Google projects) all have their own internal structures and have been the subject of exploitation research. The core concepts of corrupting metadata and manipulating free lists are universal.
How do modern mitigations like ASLR and stack canaries affect heap exploitation?
ASLR makes it harder to predict the location of heap chunks and libraries, so heap exploits almost always require an information leak as a first step. Stack canaries don’t directly protect the heap, but they make it harder to pivot a heap-based attack into a stack-based one. This has driven the development of “heap-only” exploitation techniques that achieve code execution without ever corrupting the stack, such as overwriting __malloc_hook, __free_hook, or the GOT entry for a function called on a corrupted FILE structure.
