The Cat-and-Mouse Game in Memory Corruption
Heap exploitation never sits still. A fresh allocator version drops, a new mitigation lands in the kernel or libc, and the old tricks shatter. The shellcode that danced through glibc 2.31 is dead on 2.35. That one-shot gadget you memorized for a CTF? Gone in the next Ubuntu release. This isn’t a flaw. It’s a decades-long arms race between attackers and defenders, fought inside the guts of dynamic memory management.
If you’re still clutching techniques from the early 2010s, you’re already eating dust. Modern heap exploitation demands you know not just the allocator’s data structures, but the whole hardening ecosystem wrapped around it. Here, we’ll walk through the technical forces driving that evolution—from ptmalloc’s internals to the rise of new allocator designs.
The Allocator as a Target: ptmalloc’s Internal Mechanics
To see why techniques shift, you need to see what they’re hitting. The GNU C Library’s ptmalloc, forked from Doug Lea’s dlmalloc, is the default on most Linux boxes. It sorts free chunks into bins—fastbins, tcache, smallbins, largebins, the unsorted bin—each with its own size ranges and management quirks. Early attacks went straight for the linked-list pointers inside those bins. The classic unlink attack, for instance, overwrote a free chunk’s forward and backward pointers to score an arbitrary write when the chunk got yanked from a doubly-linked list.
But ptmalloc’s internals are now guarded by a stack of integrity checks. The unlink macro verifies that the chunk’s size matches the next chunk’s prev_size, and that the forward and backward pointers line up. Those checks, rolled out in glibc 2.3.4, made the simple unlink technique a museum piece. Attackers pivoted to other mechanisms: overwriting the global_max_fast variable to shove large chunks into the fastbin path, or corrupting the tcache_perthread_struct that arrived in glibc 2.26. Every new allocator feature—thread-local caching, transparent hugepage support, per-thread arenas—opens a fresh attack surface, and defenders scramble to bolt on corresponding checks.
Mitigations That Shaped Modern Techniques
Heap exploitation didn’t evolve in a vacuum. It’s a direct answer to a pile of hardening measures that now ship by default. Understanding these mitigations is the key to understanding why certain techniques even exist.
Safe Unlinking and Double-Free Detection
The safe unlinking check, as mentioned, validates that a chunk being removed from a bin has consistent forward and backward pointers. That killed the classic unlink write-what-where primitive. In response, attackers cooked up the House of Lore, House of Mind, and other techniques that manipulate the allocator’s state without directly triggering the unlink macro. Double-free detection, which checks if the top of the fastbin already points to the chunk being freed, got bypassed by alternating frees between two chunks, or by using the tcache to stash a duplicate pointer before freeing into a regular bin.
Pointer Mangling and Safe-Linking
Glibc 2.32 introduced safe-linking for singly-linked lists (fastbins and tcache). The stored pointer gets mangled by XORing it with the address of the pointer itself shifted right by 12 bits. So an attacker who can leak a heap pointer and has an arbitrary write can still craft a valid mangled pointer, but a simple linear heap overflow that overwrites a pointer with a known address no longer works. The technique requires a heap leak to compute the mangled value, which raises the bar for attackers who previously only needed a relative overwrite.
Seccomp, ASLR, and PIE
Heap exploits rarely end with heap corruption. The goal is usually code execution, and that means bypassing ASLR and PIE. Modern techniques often chain a heap vulnerability with an information leak to defeat randomization. The leak itself might come from the heap—reading a libc pointer from an unsorted bin chunk that overlaps with a UAF object, for instance. Once you have a libc base, you can compute the address of system() or a one-shot gadget. But with seccomp filters restricting syscalls, even that isn’t enough. Exploit developers now frequently chain multiple stages: a heap exploit for an arbitrary write, a stack pivot, and a ROP chain that uses open/read/write to exfiltrate a flag rather than spawning a shell.
The Tcache Era: A Double-Edged Sword
Glibc 2.26’s per-thread cache (tcache) was a performance optimization that drastically simplified heap exploitation—at first. Each thread gets a singly-linked list of freed chunks for sizes up to 0x408 bytes, with no integrity checks on the stored pointers. A UAF or double-free in the tcache gives an immediate arbitrary allocation primitive. The tcache poisoning attack, where you overwrite a freed chunk’s next pointer to return an arbitrary address on the next malloc, became the go-to technique.
But the low-hanging fruit didn’t last. Glibc 2.29 added a simple double-free check to tcache: a key field in the chunk’s user data is set to the tcache_perthread_struct pointer, and freeing a chunk with that key already set triggers a check. Attackers adapted by clearing the key before the second free, or by using a UAF to modify the key. Glibc 2.32’s safe-linking then mangled the next pointer, requiring a heap leak. The tcache is now a microcosm of the broader arms race: a feature introduced for speed, immediately weaponized, and then progressively hardened.
House of XXX: A Taxonomy of Evolving Techniques
The “House of” naming convention, popularized by Phantasmal Phantasmagoria and the Malloc Maleficarum, provides a useful map of how techniques adapt. Each “House” targets a specific allocator state or code path, and each has been patched, revived, or replaced over time.
- House of Force: Overwrites the top chunk’s size to a large value, then requests a size that wraps the top chunk pointer to an arbitrary location. Mitigated by a check that the requested size is less than the system’s total memory.
- House of Spirit: Frees a fake chunk that the attacker has crafted on the stack or in a controlled memory region, then reallocates it to gain overlapping access. Still viable if you can control the fake chunk’s size and next pointer, but ASLR makes it harder to predict the fake chunk’s address.
- House of Einherjar: Exploits an off-by-one null byte overflow to consolidate a chunk with a previously freed chunk, creating overlapping chunks. Modern glibc checks the prev_size field for consistency, making this technique more difficult but not impossible with careful heap feng shui.
- House of Lore: Corrupts the smallbin linked list to return an arbitrary chunk on allocation. Still works in certain scenarios, but the smallbin unlinking checks require a valid next chunk’s bk pointer to point back to the corrupted chunk, demanding precise heap layout control.
Each of these techniques has spawned variants that work around new checks. The House of Botcake, for example, combines a double-free in tcache with an unsorted bin consolidation to bypass both tcache and fastbin protections. The pattern is always the same: a new check is added, and attackers find a way to satisfy the check while still achieving their goal.
Heap Feng Shui: The Art of Deterministic Layout
Modern heap exploitation is as much about controlling the allocator’s state as it is about corrupting metadata. Heap feng shui refers to the practice of carefully arranging chunks in memory to create a predictable layout that enables a specific attack. This involves understanding the allocator’s internal algorithms for servicing requests, splitting chunks, and coalescing free chunks.
For example, to exploit a use-after-free in a modern browser, you might need to place a victim object of a specific size next to a free chunk, trigger the UAF to reclaim that memory with a different type of object, and then use the type confusion to leak a vtable pointer or corrupt a length field. This requires precise control over the heap’s state, often achieved by spraying the heap with many objects of controlled size and content, then selectively freeing some to create the desired layout. The evolution of heap feng shui mirrors the evolution of allocator internals: as internal algorithms change, so do the sequences of allocations and frees needed to achieve a given layout.
Cross-Platform and Custom Allocators
The conversation so far has focused on Linux and glibc, but the same dynamics play out on other platforms. Windows 10’s Segment Heap, introduced alongside the NT Heap, added a new layer of complexity. The Low Fragmentation Heap (LFH) uses a different allocation strategy, and the backend allocator manages virtual memory directly. Exploitation techniques for the NT Heap—like the Lookaside List attack—don’t translate directly to the Segment Heap, forcing researchers to develop new primitives.
In embedded systems and IoT devices, custom allocators are common. Many are based on older versions of dlmalloc or FreeRTOS’s heap implementations, which lack modern mitigations. But they also have unique constraints: limited memory, no MMU, and real-time requirements. Exploiting these systems often means going back to first principles, understanding the specific allocator’s code, and crafting an attack that works within the device’s constraints. The techniques are simpler, but the reverse engineering effort is higher.
The Role of Static Analysis and Symbolic Execution
Defenders aren’t just adding checks; they’re using advanced analysis tools to find and fix vulnerabilities before they’re exploited. Static analyzers can detect double-frees, use-after-frees, and heap buffer overflows in source code. Symbolic execution engines can explore program paths to find inputs that trigger heap corruption. This proactive approach means that the vulnerabilities available to attackers are increasingly subtle—logic bugs, race conditions, and complex interactions between components rather than straightforward memory errors.
For exploit developers, this means that the bar for finding a useful bug is higher. The bugs that remain are often in code that’s hard to analyze automatically, like kernel drivers or JavaScript engines. Exploiting them requires a deep understanding of the allocator’s internals and the ability to chain multiple bugs into a working exploit. The days of a single buffer overflow leading directly to code execution are largely over on modern, hardened systems.
FAQ: Common Questions on Heap Exploitation Evolution
Why can’t we just use the same heap exploits from a few years ago?
Because the underlying allocator code and the hardening features around it have changed. Glibc, the Windows NT Heap, and other allocators have added integrity checks, pointer mangling, and randomization that break the assumptions those old exploits relied on. A technique that worked on glibc 2.23 will almost certainly fail on glibc 2.35 due to added checks in tcache, fastbins, and the unlink macro.
What’s the most significant recent change in heap exploitation?
The introduction of safe-linking in glibc 2.32 is a strong candidate. By mangling singly-linked list pointers with the address of the pointer itself, it forced attackers to obtain a heap address leak before they could corrupt tcache or fastbin pointers. This raised the minimum requirements for a successful exploit from a relative overwrite to an absolute overwrite plus an information leak, which is a much harder combination to achieve.
Are heap exploits still relevant with all these mitigations?
Absolutely. While the techniques have become more complex, heap vulnerabilities are still found regularly in major software. Browsers, kernels, and server applications continue to have use-after-free bugs, heap buffer overflows, and double-frees. The mitigations raise the cost of exploitation, but they don’t eliminate the bug classes. Skilled exploit developers can still chain a heap bug with an information leak and a seccomp bypass to achieve code execution.
How do I start learning modern heap exploitation?
Begin with a specific allocator version and a well-documented technique. The “how2heap” repository is an excellent resource that demonstrates techniques for different glibc versions. Start with glibc 2.23 to understand the classic attacks, then work your way up to 2.35 to see how each mitigation changes the approach. Practice in a controlled environment with ASLR disabled initially, then enable it and add the required information leak. CTF challenges are a great way to apply these skills to realistic scenarios.
