Why Most People Stare at Dumps Like They’re Hieroglyphics

Why Most People Stare at Dumps Like They’re Hieroglyphics

You’ve been there. The system crashed. The screen went blue, or the process just vanished from the task list like a witness in a mob trial. A memory dump file sits on your disk, taunting you with its opaque binary silence. Most engineers crack it open, see a wall of hex addresses, and immediately close the file—convinced the answer must be somewhere else. Anywhere else.

Here’s the thing: that dump file is a crime scene. And right now, you’re the detective who doesn’t know how to read blood spatter. Memory dumps aren’t just forensic artifacts for Microsoft support engineers or security researchers with three letters after their names. They’re the raw, unfiltered truth of what your system was doing the millisecond it all went sideways. Learning to read them means you stop guessing and start knowing.

What a Memory Dump Actually Is

A memory dump is a snapshot—either partial or complete—of the system’s RAM at the moment of a crash. When Windows hits a fatal error (bug check, STOP error, blue screen), the kernel captures what it can based on configuration and writes it to disk. Linux does something similar with kdump and kexec. The file extension varies: .dmp, .mdmp, vmcore. The principle doesn’t.

There are different flavors. A minidump is compact—it carries thread stacks, loaded module lists, and basic context. A kernel dump includes kernel memory. A full dump grabs everything: user space and all. Most production systems are configured for minidumps because nobody wants a 64GB file eating disk space after every crash. But if you’re hunting a bug that crosses the user-kernel boundary, you’ll want more than the minimum.

Configuration Matters

On Windows, check your Startup and Recovery settings. The CrashDumpEnabled registry value under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl determines what gets written. On Linux, your kdump configuration and crashkernel reservation size dictate whether you even get a dump at all. No reservation, no dump. Configure first, crash second.

Getting Your Hands Dirty: Generating a Useful Dump

Sometimes you don’t wait for a crash. You force one. Sysinternals procdump lets you capture a process dump on demand or based on conditions—CPU spikes, handle leaks, hung windows. On Linux, gcore does similar work. For kernel-level investigation, NotMyFault from Sysinternals intentionally crashes the system so you can test your dump pipeline. Sounds reckless. It’s responsible.

If you’re dealing with an intermittent issue that won’t reproduce on your machine, make sure crash dumps are enabled on the affected system. No dump means you’re flying blind, relying on logs that tell you what happened before the crash but not what caused it. There’s a reason flight recorders survive the plane.

The Anatomy of a Dump File

Before you can read a dump, you need to understand what’s inside it. The file starts with a header that identifies the dump type, the operating system version, and the bug check code. This header is what debugging engines parse first—it tells them how to interpret everything that follows.

After the header comes the actual memory content. In a minidump, this is a compressed and filtered subset. In a full dump, it’s essentially a linear image of physical memory. The debugging engine maps virtual addresses to this content using the page tables stored in the dump itself. If those page tables are corrupt or missing, addresses won’t resolve, and you’ll see a lot of “unable to read memory” errors. That’s not a tool problem. That’s your dump telling you something was already wrong before the crash.

Reading the Tea Leaves: Key Sections That Matter

The Bug Check Code

On Windows, this is your starting point. The STOP code—like 0x0000007E (SYSTEM_THREAD_EXCEPTION_NOT_HANDLED) or 0x00000050 (PAGE_FAULT_IN_NONPAGED_AREA)—tells you the category of failure. The parameters that follow narrow it down. Parameter 1 in a 0x50 dump is the referenced memory address. Parameter 2 tells you if it was a read or write operation. The documentation for each bug check is on Microsoft’s debugger documentation, and it’s one of the few references worth reading cover to cover.

The Stack Trace: Your Best Friend

The stack trace is where the story lives. When you run !analyze -v in WinDbg, it automatically walks the stack of the crashing thread. Each frame represents a function call. The top of the stack is where execution stopped. The frames below show the call chain that got you there.

Here’s what most people miss: the crashing frame isn’t always the guilty frame. A null pointer dereference in DriverX::HandleRequest might be caused by DriverX::Initialize failing to set up a structure three seconds earlier. The crash is the symptom. Walk the stack. See who called what. Check the parameters passed between frames. k gives you the stack. dps lets you dump stack memory with symbol resolution. Use both.

Disassembly: Where the Ghost Lives

Symbols get you function names. Disassembly gets you the actual instruction that failed. Run u on the instruction pointer address, and you’ll see the exact assembly opcode that triggered the exception. Was it a mov trying to read from a null pointer? A call through a vtable that got corrupted? The disassembly tells you what the CPU was actually attempting when everything fell apart.

If you don’t have symbols—and sometimes you won’t—the disassembly is all you have. Learning to read x86 or ARM assembly is non-optional if you want to do this work for real. You don’t need to write it. You need to read it. There’s a difference.

Common Patterns and What They Signal

After you’ve read enough dumps, patterns emerge. Here are a few that show up repeatedly:

• A single reference count going to zero too early: Look for ObDereferenceObject in the stack. Something freed an object that was still in use. The subsequent access violates because the memory has been reclaimed or repurposed.
• Stack overflow in a driver: The stack base address will be suspiciously close to the current stack pointer. Recursive calls or excessively large local buffers are the usual suspects.
• Memory corruption across pool boundaries: If you see a crash in a pool allocation routine like ExFreePoolWithTag and the caller swears they passed the right pointer, something stomped on the pool header earlier. Use !pool to inspect surrounding allocations. The culprit is often in a different driver entirely.
• IRQL_NOT_LESS_OR_EQUAL with a user-mode address: Something tried to access pageable memory at an elevated IRQL. This almost always means a driver is touching user buffers without proper handling at DISPATCH_LEVEL or above.

Tools of the Trade

WinDbg remains the standard for Windows kernel debugging. Get it from the Windows SDK. Learn the extensions: !process, !thread, !irql, !pool, !vm. These are not optional. They are the interface between you and the dump.

For Linux, crash is your primary tool. Paired with gdb for user-space dumps and makedumpfile for filtering vmcore content, it gives you comparable capability. The learning curve is steep. The documentation is scattered. Learn it anyway.

GDB itself is indispensable for user-space core dumps on any Unix system. Run gdb /path/to/binary /path/to/core, then bt full for a detailed backtrace, info registers for CPU state, and x/20x $rsp to inspect the stack. The GDB documentation is thorough if you take the time to read it.

Symbols: The Difference Between Guessing and Knowing

A dump without symbols is like a map without labels. You can see terrain, but you can’t name a single street. Microsoft makes public symbols available through their symbol server. Configure WinDbg with .sympath srv*C:\Symbols*https://msdl.microsoft.com/download/symbols and most Windows modules will resolve. Third-party drivers won’t—not unless the vendor provides them, which almost none do.

When symbols are missing, you’ll see module names followed by offsets like mydriver+0x1a3f. That offset is a specific instruction within the driver binary. If you have the binary (and you should—find it in the dump’s loaded module list), you can load it with symbols you’ve generated locally. If you don’t have the source or PDB for a third-party driver, you can still disassemble the offset and reason about what the code was doing.

The Hard Truth About Reading Dumps

This work is tedious. It requires patience, familiarity with operating system internals, and a willingness to accept that some crashes won’t yield clean answers. Memory is ephemeral. State is complex. The dump you have is a single moment frozen in time, and the root cause might have been set in motion seconds or minutes before the actual crash.

But when you’re staring at a production outage that affects thousands of users, and the logs show nothing useful, and the monitoring dashboards just confirm that something died—opening that dump file and tracing the fault to a specific driver, a specific function, a specific line of logic—that’s not just debugging. That’s forensic engineering. And it’s a skill that will never be obsolete.

FAQ

Can I analyze a memory dump without symbols?

Yes, but it’s significantly harder. Without symbols, function names won’t resolve, and you’ll see raw addresses or module+offset notation instead. You can still disassemble the code at the crash address, inspect registers, and examine memory. The bug check code and parameters remain readable. However, you’ll need to rely more heavily on assembly analysis and pattern recognition. Always attempt to obtain symbols—public symbol servers cover all Microsoft binaries, and you should keep PDBs for your own builds.

What’s the difference between a minidump and a full dump?

A minidump contains only essential data: the bug check code, processor context for the crashing thread, stack memory, and a list of loaded modules. A full dump contains the entire contents of physical memory at the time of the crash. Minidumps are small (typically under 1MB) and cover most common debugging scenarios. Full dumps can be tens of gigabytes on modern systems but are necessary when you need to inspect user-mode memory, examine processes other than the crashing one, or investigate memory corruption that spans large regions.

How do I know if a crash was caused by hardware or software?

Start with the bug check code. 0x00000124 (WHEA_UNCORRECTABLE_ERROR) strongly suggests hardware—specifically, a machine check exception reported by the CPU. 0x0000009C (MACHINE_CHECK_EXCEPTION) is similar. Consistent crashes at the same address or in the same driver point toward software. Random addresses, varying error codes, and crashes in different modules each time suggest memory corruption from a bad DIMM, overheating, or power delivery issues. Run !mca in WinDbg to inspect machine check architecture data. Run MemTest86 overnight. Don’t assume software until you’ve ruled out hardware.