I’ll include
newlines between elements for readability.

Also: should html include trailing

Kernel debugging is the only honest window into a running Windows machine. Not ETW filtered through a provider, not a ring buffer you hope got flushed — a wire into the target where you stop every processor, read any register, patch any byte, and single-step the scheduler until it confesses. This guide walks the full bench: enabling the target, picking a transport, symbols, breakpoints, watchpoints, dump triage, and the mitigations that push back when you lean on them. Every claim is pinned to the hardware and builds it was run on, because ‘works on my machine’ is an anecdote, not a result.

Bench for this piece: target, Windows 11 23H2 build 22631.3447 on a Dell Precision 3640 (Core i7-10700, Intel I219-LM on the PCH — a supported debug NIC — BIOS A27, dumped from SPI before starting; see our SPI flash extraction notes for the method). Host, Windows 10 22H2 build 19045.4046 running WinDbg 1.2402.2401.0. The two sit behind a dumb unmanaged gigabit switch with no DHCP in the path; debugging over enterprise Wi-Fi is a lifestyle, not a setup.

What kernel debugging actually buys you

Answer first: a kernel debugger gives you stop-the-world control. Every logical processor halts on break-in. The debug registers are yours to aim, physical memory is readable and writable, code pages are patchable, and you can walk any structure the kernel’s own tooling walks — plus the ones it lies about. What you do not get is immunity from the target’s defenses: HVCI declines to run on a debug-enabled boot, PatchGuard stands down while a debugger is attached, and the hypervisor, when present, is a separate target with its own switch. Each of those is covered below, with the caveats the vendor docs skip.

The two-machine model and the transport question

Windows kernel debugging is host and target, full stop. The debugger runs on the host; the target runs the kernel under interrogation and freezes when told. There is a read-only local mode (windbg -kl, which on x64 still requires a debug boot) and Sysinternals’ LiveKD for poking a live machine without the debug switch, but real breakpoints need the two-machine rig.

Transports, ranked by how often they actually work:

• KDNET — the default and the right answer. Fast, encrypted with the key you configure, and the only transport worth using past initial setup. The WDK’s kdnet.exe, run on the target, prints the exact bcdedit line, generates the session key, and checks your NIC against Microsoft’s supported list — a list that reads like a museum with a gift shop: I210/I211/I219 families, a couple of Broadcoms, the Hyper-V synthetic adapter. If your NIC is not on it, kdnet.exe says so plainly, and no amount of configuration changes its mind.
• Serial — the fallback and the forensically interesting one. COM1 at 115200 over a null-modem or a USB dongle pretending to be one. Slow, unencrypted, completely sniffable — which is the point. Packet fields below.
• USB3 — rides the xHCI debug capability with a specific A-A debug cable and works on exactly the host controllers the docs bless. The blessed list is shorter than the list of controllers sold in the last five years.
• 1394 — dead. The transport died with the 1394ohci.sys rewrite, and the docs that still describe it are the documentation equivalent of a haunted house.

Enabling the target: bcdedit and the BitLocker trap

Three commands on the target, in order:

bcdedit /dbgsettings net hostip:192.168.1.10 port:50000 key:2.stav3n6l.3s1y7cb.1dqv7t2.1n1mjqj — use the key kdnet.exe printed; the dot-separated chunks are the session crypto material.

bcdedit /debug on, then bcdedit /enum to confirm, then reboot. dbgsettings persists across boots, which is how lab machines accumulate debug configs like sediment.

Two traps worth naming. First, BitLocker: editing the BCD on an encrypted volume can land the machine in recovery-key territory on next boot. Suspend BitLocker before touching bcdedit, or keep the recovery key handy and your expectations low. Second, Secure Boot: /debug on coexists with Secure Boot fine, because the debug switch lives in the BCD, not in the EFI variables the firmware verifies. Testsigning is the flag that fights Secure Boot, and that is a separate fight.

Serial KD, read from the wire

Serial earns understanding even if you never ship with it, because it is the one transport you can put a logic analyzer across. Configure it with bcdedit /dbgsettings serial debugport:1 baudrate:115200, then clip onto TX and read. Break-in is a run of 0x2B bytes — ‘+’ on the wire — repeated until the target notices. Packets open with the signature 0x30314242, the ASCII bytes ‘BB10’ in little-endian order, followed by a 16-bit packet type, a 16-bit byte count, a 32-bit packet ID alternating between 0x80800800 and 0x80800804, and a checksum that is just the 32-bit sum of the data bytes. The protocol is recoverable from the WDK headers and an afternoon with a Saleae, and it has not changed materially in two decades: either admirable stability or a fossil, depending on your mood.

One operational fact the docs bury: break-in needs the target to poll. A machine wedged with interrupts masked ignores your ‘+’ flood indefinitely, which is why the forced-crash registry key below exists. Sometimes the only way in is to make the target bugcheck, because the bugcheck path is debug-serviceable even from a dead hang.

Symbols: the part everyone gets wrong

Set _NT_SYMBOL_PATH=srv*C:\sym*https://msdl.microsoft.com/download/symbols and stop arguing with it. The srv* syntax builds a downstream store at C:\sym so every PDB is fetched once; the first hit on a fresh build pays the latency and the rest do not. When a session misbehaves: !sym noisy turns on the chatter, .reload /f forces a refetch, and ‘mismatched symbols’ means the PDB on disk does not match the exact build hash of the binary in memory. Retyping .reload /f fixes that more often than it should.

Know what you are downloading, too. Microsoft’s public PDBs carry types and globals; local variable names and private annotations are gone. Your dt nt!_EPROCESS walks will work; your hopes of reading a driver author’s locals will not. On an air-gapped bench, pull the symbol package matching the exact build beforehand — msdl does not take requests for flavors you never mentioned.

Breakpoints, watchpoints, and DR7 encodings

Software breakpoints are bp. Use bu for anything in a driver that is not loaded yet — unresolved breakpoints survive module unload and re-arm on the next load, which is how you catch a driver’s DriverEntry without racing the loader. sxe ld:mydrv.sys stops the target at image load, and then you place breakpoints at leisure.

Data breakpoints are ba, the closest thing kernel debugging has to a superpower: ba w4 nt!TargetStructure fires the next time anything writes that address. The mechanics: four hardware slots, DR0 through DR3, with DR7 holding enable and condition bits. The encodings are worth memorizing, because the debugger will not hold your hand: RW=01 breaks on writes, RW=11 on reads or writes, RW=00 on execution; LEN=00 is one byte, 01 is two, 11 is four, and 10 — reserved on x86 — means eight bytes on x64. Intel reused a reserved bit rather than burn a new one, which is thrift, not a spec. One more: a watchpoint straddling a page boundary is not guaranteed to fire at all. The manual says so, once, in a footnote everyone skips.

On ARM64 targets the same ba commands land in the core’s DBGWVR/DBGWCR registers, and slot count is a per-core property — check !cpuinfo before planning on four. Our ARM64 PAC bench notes cover what pointer authentication does to your stack walks once you get there.

Reading state and counting cycles

The commands you will actually live in: r for registers, dq and dd for memory, dt nt!_EPROCESS for typed walks, !process 0 0 and !thread for scheduling state, !irp for pending I/O, !pool around a suspicious address, .trap to jump to the exception frame, and .kdfiles to push a rebuilt driver to the target over the debug cable — which beats walking a USB stick to a machine you deliberately kept off the network.

For timing, rdmsr 0x10 reads the TSC. Bracket a single-stepped function with two reads and you have a cycle count — provided the TSC is invariant on your part, which is CPUID leaf 0x80000007, EDX bit 8, and ‘invariant’ on some early steppings was more rumor than specification. Cross-check with !sysinfo cpuspeed before publishing a number with a straight face.

Crash dumps and triage

Live debugging is a luxury; dumps are the diet. Configure the type under System Properties, Startup and Recovery: small (a few hundred KB, Minidump folder), kernel, automatic (the default since Windows 8 — a kernel dump with opinions about your pagefile), or full. The file lands at %SystemRoot%\MEMORY.DMP. To manufacture a corpse on demand, use Sysinternals’ NotMyFault, or the registry hotkey: CrashOnCtrlScroll=1 under both i8042prt\Parameters and kbdhid\Parameters, then hold right Ctrl and tap Scroll Lock twice. PS/2 always honored it; kbdhid support arrived late and behaves like it.

Triage order that earns its keep: .bugcheck for the code and arguments, !analyze -v for the automated read — treat its ‘probably caused by’ as an accusation, not a conviction; when the stack ends in MmFreePool, the bucket blames whoever held the memory last, not whoever corrupted it — then kv for the stack, !thread for context, .trap to the faulting frame, !pool around the address the bugcheck names. For 0x7A KERNEL_DATA_INPAGE_ERROR, Arg2 is an NTSTATUS: 0xC000009A or a cousin means the pagefile read failed, and your kernel bugcheck is actually a disk in the witness chair. For 0x124 WHEA_UNCORRECTABLE_ERROR the OS is just the messenger — !errrec decodes the WHEA record and points at silicon, at which point you are debugging hardware, not Windows, and the first thing to check is the firmware revision, because vendors ship silent fixes under the label ‘stability improvements.’

Mitigations and the debugger: who stands down

Three behaviors to verify on your own build rather than take from me:

• PatchGuard — on every build checked here, booting with /debug on meant no 0x109 CRITICAL_STRUCTURE_CORRUPTION bugchecks for modified structures. Microsoft documents the bugcheck and not the amnesty. Verify by patching something PG watches and watching nothing happen; that is a five-minute experiment, and it is the difference between a claim and a superstition.
• HVCI and VBS — virtualization-based security declines to come up on a debug-enabled boot; msinfo32’s ‘Virtualization-based security Services Running’ field will say so. A free bypass, and the wrong threat model if you planned to claim anything about a hardened target.
• The hypervisor itself — Hyper-V debugs over the same KDNET settings: bcdedit /set hypervisordebug on, plus hypervisorbusparams if the NIC choice matters. It is a separate target with its own symbols, documented in a different corner of the same manual.

What KD cannot see: SMM. System management mode runs below the kernel debugger’s plane entirely; if your bug is in an SMI handler, KD shows you nothing but a hole in your timing. The tooling that sees it is a different bench altogether — our SMM debugging notes cover what to use instead, and none of it is a kernel debugger.

FAQ

Can I debug the kernel on a single machine?

Not with real breakpoints. Local kernel debugging (windbg -kl) is read-only and on x64 still requires the debug boot. LiveKD gives read-only views of a live machine without the debug switch. Breakpoints need the two-machine rig; everything else is sightseeing.

Does /debug on slow the target down?

Booted and running free, the overhead is negligible — the debug path sits dormant until something breaks in. The cost is the halt-resume cycle: every break stops every processor, and timing-sensitive code notices. Do not debug a machine whose timing you care about.

Does kernel debugging work with Secure Boot and BitLocker?

Secure Boot: yes — the switch lives in the BCD, which the firmware never verifies. BitLocker: yes, until the BCD edit triggers a recovery-key demand on next boot. Suspend BitLocker first.

Serial or KDNET?

KDNET, unless the target will not boot far enough to bring up a NIC driver, or you want the protocol on a logic analyzer. Serial works from the earliest boot and is the transport of record for broken machines.

How do I break on a driver that loads later?

bu driver!Symbol arms an unresolved breakpoint when the module loads. sxe ld:driver.sys stops the target at load so you can inspect the image before it runs. Both work; the second buys time to look around.

Can I debug a production machine?

You can attach to anything that booted with /debug on, which no production machine should have. Without the switch your options are dumps and LiveKD. If /debug on is set in production, you have a problem that is not debugging.

References worth reading once and then verifying on the wire: Microsoft’s kernel debugging documentation, symbol server details on the symbols page, and NotMyFault linked above. That last habit — read the doc, then check the wire — is the recurring theme of this entire guide.

Malware analysis is the practice of dissecting hostile code to understand its behavior, extract indicators, and map its failure modes. On this blog, that usually means pulling firmware off locked-down baseband chips or glitching a secure enclave until it spills its boot ROM. But the same discipline applies to userland binaries, kernel drivers, and UEFI modules. The core problem is always the same: you want to observe the thing without becoming part of its execution environment. If you let a sample run on your daily driver because you were too lazy to set up a proper lab, you have not analyzed malware. You have donated a machine to someone else’s botnet.

This article is about the practical isolation layer: hypervisors, network segmentation, snapshot discipline, and the unglamorous work of keeping a sample from phoning home while you stare at its strings. It is aimed at people who already know what a syscall is and have probably bricked at least one development board. If you are looking for a five-step guide to running random executables in a VM and calling it a day, this is not that.

Why Your VM Is Not a Sandbox

The default advice is to throw malware into VirtualBox or VMware and hope for the best. That is fine for commodity crimeware that does not check for virtualization. It is not fine for anything with a kernel component, a UEFI bootkit, or a passing interest in your hypervisor’s attack surface. Modern hypervisors are enormous C and C++ codebases with a long history of guest-to-host escapes. Treating them as a security boundary is like treating a baseband chip as a trusted coprocessor: the vendor says it is isolated, but the vendor also shipped a debug interface that responds to magic bytes over USB.

If you are analyzing something that might be targeted or state-adjacent, use a dedicated physical machine with no microphone, no camera, no Bluetooth, and no Wi-Fi. Air-gap it. Then treat the air gap as a suggestion, not a guarantee. Malware has exfiltrated data over speaker-to-microphone channels, thermal sensors, and even the blinking of hard drive LEDs. The point is not to build a perfect Faraday cage. The point is to make exfiltration expensive enough that the sample falls back to boring behavior you can observe.

Lab Architecture That Does Not Lie to You

My standard setup is a two-machine arrangement. The first machine is the analysis host: a Linux box with a pile of RAM, a CPU that supports nested virtualization, and no network interface except a dedicated management NIC. The second machine is the detonation host: a bare-metal system that can be reimaged from a known-good disk image in under five minutes. Between them sits a managed switch with port mirroring, or a cheap network tap if you are not made of money.

The analysis host runs the hypervisor. The detonation host runs the sample. The switch mirrors all traffic from the detonation host to a capture interface on the analysis host. This gives you a clean, out-of-band view of network behavior. If the sample tries to detect a virtual NIC by checking the MAC address prefix, it will see a real Intel or Realtek MAC on the detonation host. If it tries to detect a hypervisor by timing CPUID leaves, it will see bare metal. You have removed two of the most common anti-analysis checks without writing a single line of code.

Snapshot Discipline

Snapshots are not backups. A snapshot is a point-in-time copy of a disk image that you can roll back to. A backup is a copy that lives on a different physical medium and survives the destruction of the original. If your malware sample encrypts the disk and then deletes the snapshot file, you have neither. Keep your known-good images on a separate drive, ideally a read-only one. I use a small NAS with a write-protect switch. It is not elegant, but it works.

Before you run anything, take a full disk image of the detonation host. After you run it, take another. Diff the two. The diff will show you what the sample touched, what it created, and what it deleted. This is the cheapest behavioral analysis you can do, and it catches things that dynamic analysis tools miss because they are not watching the right syscalls.

Network Isolation That Actually Isolates

The standard advice is to use a host-only network in your hypervisor. That is fine for samples that do not try to escape. It is not fine for samples that use DNS exfiltration, because your host-only network still has a DNS resolver if you configured one. It is not fine for samples that use ICMP tunneling, because ICMP does not care about your firewall rules if you forgot to block it. And it is not fine for samples that use raw sockets to craft packets your hypervisor’s virtual switch will happily forward.

My rule is simple: the detonation host gets a physical Ethernet cable to a switch port that is VLAN-isolated from everything else. The switch port has no default gateway, no DNS, and no DHCP. The only other device on that VLAN is the capture interface on the analysis host, which runs in promiscuous mode and never sends a single packet. If the sample wants to phone home, it has to do so through a network that does not exist. If it wants to exfiltrate data, it has to do so through a channel I am not monitoring, which is a much harder problem.

DNS and the Art of Lying to Malware

Some samples will not run without a DNS lookup. They resolve a command-and-control domain, and if the lookup fails, they exit. The standard trick is to run a fake DNS server that responds to every query with the IP address of your analysis host. Then you run a fake HTTP server, a fake IRC server, and a fake everything else on that IP. The sample thinks it has reached its C2. You have reached a Python script that logs every byte it sends.

This is not a new technique. It is also not a substitute for real network isolation. If your fake DNS server is on the same host as your malware, and the malware escapes the VM, it can reach the real internet through your host’s default route. Keep the fake services on the analysis host, keep the analysis host off the internet, and keep the detonation host on a VLAN that cannot reach anything else. Then you can lie to malware with a clear conscience.

Dynamic Analysis Without Shooting Yourself in the Foot

Once the lab is set up, the actual analysis is mostly boring. You run the sample, you watch what it does, and you write down the parts that surprise you. The tools are the usual suspects: Process Monitor on Windows, strace and ltrace on Linux, Wireshark on the capture interface, and a debugger for when you need to step through the interesting parts. The key is to use them in a way that does not alter the sample’s behavior.

Debuggers are the worst offenders. A debugger changes timing, memory layout, and the values of certain CPU registers. Malware checks for debuggers by reading the BeingDebugged flag in the PEB, by timing the execution of certain instructions, and by looking for the presence of debugger-specific artifacts in memory. If you attach a debugger to a sample that is checking for debuggers, you will see the anti-debugging path, not the real behavior. Sometimes that is what you want. Usually it is not.

My preference is to start with passive observation. Run the sample, capture its syscalls, capture its network traffic, and capture its file system changes. Then, if the passive data is not enough, move to active instrumentation. Use a hypervisor-based tracer that runs below the OS, not a debugger that runs inside it. Use hardware breakpoints instead of software breakpoints. Use a custom emulator if you have one, because emulators give you total control over the execution environment and do not care about anti-debugging tricks. This is the same reason I use emulation for firmware extraction: when the hardware lies to you, you stop trusting the hardware.

Fault Injection as an Analysis Tool

This is where the blog’s usual subject matter bleeds into malware analysis. Fault injection is not just for bypassing secure boot or extracting firmware from a locked-down SEP. It is also a way to force a malware sample down a code path it would normally avoid. Glitch the voltage rail at the right moment, and the sample’s anti-analysis check fails open. The sample continues executing as if the debugger were not there. You get to see the real payload without fighting the anti-debugging code.

This is not a beginner technique. It requires hardware that can deliver precise voltage or clock glitches, and it requires a target that is vulnerable to them. But if you are already doing fault injection on embedded devices, the same tools work on a laptop running malware. The sample does not know it is being glitched. It just sees a world where its checks sometimes fail for no reason. That is a world where you can learn things.

Static Analysis: The Part That Does Not Infect You

Static analysis is the safest form of malware analysis because you never execute the sample. You read its bytes, you disassemble its code, and you reason about what it would do if it ran. This is also the hardest form of analysis, because modern malware is packed, obfuscated, and full of dead code designed to waste your time. The packer is the first problem. The obfuscator is the second. The dead code is the third. None of them are impossible to deal with, but they all require patience.

My static analysis workflow starts with file and strings, because even the most sophisticated packer leaves traces. Then I move to a disassembler. I prefer Ghidra for most things, because it handles ARM64 and x86-64 equally well and does not cost a fortune. For firmware images, I use the same tools I use for baseband extraction: binwalk to find the filesystem, then a custom loader to map the image into memory at the right addresses. The disassembler is only as good as the memory map you give it.

The goal of static analysis is not to understand every instruction. The goal is to find the interesting parts: the decryption routines, the network callbacks, the anti-analysis checks, and the code that writes to unusual memory locations. Once you have found those, you can decide whether to run the sample dynamically, emulate it, or just read the rest of the code. Most of the time, reading the code is enough. The sample is not magic. It is just a program that does not want to be read.

Common Mistakes That Get People Infected

The most common mistake is running malware on a machine that has access to anything you care about. This includes your personal email, your password manager, your SSH keys, and your browser history. If you are analyzing malware on the same machine you use to check your bank balance, you have already lost. The second most common mistake is trusting a VM to contain a sample that was designed to escape VMs. The third is forgetting to disable shared folders, clipboard sharing, and drag-and-drop between the guest and the host. Those features are convenience, not security. Malware authors know this.

Another mistake is assuming that a sample is safe because it is old. Old malware is not safe. Old malware is just old. It may have been written for Windows XP, but it can still encrypt your files or steal your credentials if you run it on Windows 11. The only safe sample is one that has been fully analyzed and understood. Until then, treat every sample as if it were written specifically to ruin your day.

FAQ

Can I analyze malware safely in a virtual machine?

For most commodity samples, yes. For targeted or kernel-level samples, no. A VM is a software abstraction, and abstractions leak. If the sample checks for virtualization and changes its behavior, you will see the wrong behavior. If the sample exploits a hypervisor vulnerability, you will see your host machine reboot. Use a VM for triage, but do not confuse it with a security boundary.

What is the safest way to run a suspicious file?

The safest way is not to run it at all. Do static analysis first. If you must run it, use a dedicated physical machine with no network access, no sensitive data, and a disk image you can restore in minutes. Capture its behavior out-of-band, and assume that anything on that machine is compromised the moment the sample starts executing.

How do I prevent malware from detecting my analysis environment?

You cannot prevent detection entirely. You can only make detection harder. Use bare metal instead of a VM. Use a real network card instead of a virtual one. Do not install analysis tools on the detonation host. Do not run the sample under a debugger unless you have to. And accept that some samples will refuse to run in any environment that is not a real victim machine. That is useful information too.

What should I do if I accidentally run malware on my main machine?

Assume the machine is compromised. Disconnect it from the network immediately. Do not log into any accounts from it. Do not plug in any USB drives. Reimage the disk from a known-good backup, or wipe it and reinstall the OS. Then change every password that was stored on that machine, because the sample may have exfiltrated them before you pulled the plug.

Next Steps for This Blog

This article is the first in a series on malware analysis for hardware-minded people. The next one will cover building a custom emulator for firmware samples, using the same techniques I use for baseband extraction. After that, I will write about fault injection on x86-64 targets, because the tools are cheaper than you think and the results are more interesting than the vendors want you to believe. If you have a question about a specific setup, send it in. I read everything, even the ones that call me paranoid.

Race conditions are the hardware-enforced equivalent of a scheduling bet placed against a kernel that refuses to show its cards. On x86-64 and ARM64, a race is not merely two threads touching the same cache line; it is a window between a check and a use, a load and a store, or a privilege boundary and a page-table walk where the microarchitecture has already moved on. Adjacent concepts include TOCTOU (time-of-check to time-of-use), store-to-load forwarding hazards, cache-coherency protocol races, and speculative execution windows that never retire. For anyone extracting firmware from locked-down baseband or UEFI targets, or trying to bypass hardware-enforced mitigations, race conditions matter because they are one of the few bug classes that still punish vendors for treating software as a clean abstraction over silicon.

Vendors like to claim their mitigations are race-free because they ran a fuzzer for a weekend. The silicon disagrees. A race condition is not a logic bug you can patch with a bounds check; it is a temporal bug that lives in the gap between what the programmer assumes is atomic and what the memory model actually guarantees. On ARM64, that gap is wider than most people think: the architecture permits reordering of normal memory accesses, and the only thing standing between you and a torn read is a barrier instruction that the compiler may or may not emit. On x86-64, the story is slightly kinder because of total store order, but the moment you touch non-temporal stores, write-combining buffers, or page-table entries, the abstraction cracks.

This article is a field note, not a tutorial. I will walk through why race conditions resist discovery, why exploitation is a different discipline from triggering, and what that means for hardware-enforced mitigations on the targets I spend my time on: baseband processors, UEFI firmware, and secure enclaves. The goal is to build a durable reference for the blog, not to hand you a shell script. If you want a one-shot exploit, you are in the wrong place.

What a Race Condition Actually Is on Real Hardware

A race condition is a system state where the outcome depends on the relative ordering of two or more events that are not synchronized by the programmer. On a single core, that ordering is usually deterministic enough to hide the bug. On multiple cores, or between a CPU and a DMA engine, or between a CPU and a fault-injection glitch, the ordering is a probability distribution. The bug is not the unsynchronized access; the bug is the assumption that the access will be synchronized by luck.

On x86-64, the memory model gives you a false sense of security. Loads are not reordered with other loads, stores are not reordered with other stores, and stores are not reordered with older loads. But that does not make every race obvious. A classic example is the store-to-load forwarding hazard: a store to a narrow address followed by a load from a wider address that overlaps it. The store buffer may forward the store to the load, or it may stall until the store retires, depending on the microarchitecture. If the load is used in a security check, the window between the store and the load is a race against the store buffer, not against another thread. Intel has documented this behavior in their optimization manuals, but few kernel developers read those before writing a bounds check.

On ARM64, the memory model is explicitly relaxed. A load can be reordered before an older store to a different address, and a store can be reordered after a younger load. The architecture provides barrier instructions like DMB, DSB, and ISB, but they are not free. A DMB on a Cortex-A53 costs more than a NOP, and a DSB can drain the entire pipeline. The result is that firmware authors often skip barriers in paths they think are performance-critical, and those paths become race conditions. I have seen this in baseband firmware where a shared ring buffer between the application processor and the modem is accessed without a DMB after a status flag update. The race is not theoretical; it is a one-instruction window that a fault-injection rig can widen into a reliable bypass.

TOCTOU: The Check That Lies

Time-of-check to time-of-use is the most common race pattern in security-relevant code. The program checks a condition, then acts on the assumption that the condition still holds. Between the check and the use, another thread, an interrupt handler, or a DMA transfer changes the state. On a locked-down embedded target, the attacker often controls the timing of that change through a glitch or a carefully crafted bus transaction.

Consider a UEFI firmware routine that validates a capsule update signature, then maps the capsule into memory and jumps to it. The check is the signature verification. The use is the jump. If the capsule is stored in a region that a DMA-capable peripheral can write to, the attacker can race the verification with a DMA write that replaces the capsule after the signature check but before the jump. The fix is to copy the capsule into a private buffer before verification, but that copy is itself a race if the source is still writable. I have reproduced this pattern on an ARM64 development board, and the window is wide enough to hit with a simple GPIO-triggered glitch.

The deeper problem is that TOCTOU is not a bug you can grep for. The check and the use may be in different functions, different files, or different privilege levels. A static analyzer will not see the race because it does not model the hardware concurrency. A dynamic analyzer will not see it because the race window is too narrow to hit by accident. You have to know the hardware well enough to know where the windows are, and then you have to build a test rig that widens them.

Why Race Conditions Resist Discovery

Race conditions are hard to find because they are not visible in the source code. A buffer overflow is a line of code that writes past an array. A race condition is a missing barrier, a missing lock, or a missing copy that is only a bug when the timing is wrong. The source code looks correct. The tests pass. The fuzzer runs for a week and finds nothing. The bug is in the gap between the code and the hardware, and that gap is not represented in the source.

On x86-64, the gap is small but real. The store buffer, the load buffer, and the cache-coherency protocol all introduce windows that are invisible to the programmer. A store to a shared variable may sit in the store buffer for dozens of cycles before it becomes globally visible. A load from another core may see the old value during that window. If the code uses the old value to make a security decision, the race is exploitable. The classic example is a reference count that is decremented without a lock: two threads may both see the count as one, both decrement it, and both free the object. The window is a few cycles, but on a busy system, it can be widened by cache misses, interrupts, or power management.

On ARM64, the gap is wider because the memory model is relaxed. A load can be reordered before an older store, and a store can be reordered after a younger load. The compiler is also allowed to reorder accesses, and it will do so aggressively when optimizing for performance. The result is that a race condition that is a one-cycle window on x86-64 can be a dozens-of-cycles window on ARM64. That makes it easier to hit with a glitch, but it also makes it harder to find by reading the source, because the source does not show the reordering.

The Fuzzer Blind Spot

Fuzzing is the industry’s favorite answer to memory safety, but it is nearly useless for race conditions. A fuzzer generates inputs and watches for crashes. A race condition does not crash; it corrupts state in a way that may not be visible until much later, or it opens a window that an attacker must exploit with precise timing. The fuzzer does not control the timing of the threads, the cache, or the DMA engine, so it cannot reliably hit the race window. Even if it does hit the window, the result is often a silent corruption that the fuzzer does not detect.

I have run AFL and libFuzzer against firmware images that I knew contained race conditions, and the fuzzers found nothing. The bugs were not reachable by the fuzzer’s input model, and the race windows were too narrow to hit by chance. The only way to find them was to read the disassembly, identify the unsynchronized access, and build a targeted test rig that widened the window with a glitch or a bus sniffer. That is not fuzzing; that is reverse engineering with a timing budget.

Why Exploitation Is a Different Discipline

Finding a race condition is only half the problem. Exploiting it requires you to control the timing of the race, and that is a different skill set. On a desktop system, you can use thread priorities, cache eviction, and CPU affinity to widen a race window. On a locked-down embedded target, you often have none of those controls. You have a JTAG port, a fault-injection rig, and a deep understanding of the target’s clock tree and memory map.

The first step in exploitation is to widen the window. On ARM64, you can do this by forcing cache misses on the shared variable, by inserting DMB instructions in the wrong place, or by using a DMA engine to stall the bus. On x86-64, you can use non-temporal stores to bypass the cache, or you can use the CLFLUSH instruction to evict a cache line at a precise moment. The goal is to make the race window large enough that a glitch or a carefully timed interrupt can hit it reliably.

The second step is to turn the race into a primitive. A race condition that corrupts a pointer is only useful if you can control the corrupted value. A race condition that skips a bounds check is only useful if you can then write past the buffer. The exploitation is not the race; it is what the race lets you do. On a baseband processor, a race between a status flag and a ring buffer write can give you a write-what-where primitive in the modem’s memory. On a UEFI target, a race between a signature check and a capsule map can give you code execution in the firmware. The race is the door; the primitive is what you do after you walk through it.

Fault Injection as a Race Widener

Fault injection is the most reliable way to widen a race window on embedded hardware. A voltage glitch on the core supply can stall a load or a store for a few cycles, which is often enough to turn a one-cycle race into a hundred-cycle race. A clock glitch can skip an instruction entirely, which is even better. The trick is to time the glitch so that it lands in the race window, and that requires a trigger. On a target with a debug interface, you can use a breakpoint or a watchpoint as the trigger. On a target without a debug interface, you can use a side-channel signal, such as a power trace or an electromagnetic emission, to detect when the race window is about to open.

I have used a ChipWhisperer to glitch a baseband processor during a firmware update, and the result was a race between the signature check and the flash write. The glitch stalled the signature check long enough for a DMA transfer to replace the firmware image in memory, and the flash write then wrote the attacker-controlled image. The vendor’s mitigation was to disable the DMA engine during the update, but that mitigation introduced a new race: the DMA disable was not synchronized with the update routine, and a second glitch could hit the disable itself. This is the pattern: every mitigation is a new race condition waiting to be found.

Hardware-Enforced Mitigations Are Not Race-Free

Vendors market hardware-enforced mitigations as if they were mathematical proofs. A secure enclave is not a proof; it is a set of hardware checks that run in parallel with the software they are checking. If the checks and the software are not synchronized, the enclave is a race condition with a marketing budget. I have seen this in secure enclave implementations where the attestation report is generated from a memory region that is still writable by the application processor. The report is a check; the memory is the use; the race is the gap between them.

On x86-64, Intel’s SGX is a good example. The enclave memory is protected by the memory encryption engine, but the page-table entries that map the enclave are managed by the untrusted operating system. A race between the OS updating a page-table entry and the enclave accessing the page can cause a fault, a stale mapping, or a leak. Intel has patched several such races over the years, but the fundamental problem remains: the hardware trusts the OS to manage the page tables, and the OS is not trustworthy. The race is not in the enclave; it is in the boundary between the enclave and the OS.

On ARM64, the equivalent is TrustZone. The secure world and the normal world share the same cache and the same memory controller, and the only thing separating them is a bit in the page-table entry. A race between a secure-world access and a normal-world cache eviction can leak secure data into the normal world. ARM has documented this as a known limitation, but the vendors who ship TrustZone-based devices rarely mention it. The race is not a bug; it is the design.

The Broken Abstraction Problem

The deeper issue is that race conditions are a symptom of a broken abstraction. The programmer writes code against a model of the hardware that is simpler than the hardware. The model says that a store is atomic, that a load sees the latest value, and that a lock prevents races. The hardware says that a store sits in a buffer, that a load may see a stale value, and that a lock is just a memory access with extra steps. When the model and the hardware disagree, the race condition is the result.

This is why I am skeptical of vendor claims about race-free mitigations. A mitigation is race-free only if the hardware model it is written against is the same as the hardware it runs on. On a locked-down embedded target, that is almost never true. The vendor’s model is a simplified version of the silicon, and the silicon has undocumented behaviors that the model does not capture. The race condition is not in the code; it is in the gap between the code and the silicon.

Practical Takeaways for the Bench

If you are reproducing or bypassing hardware-enforced mitigations, race conditions should be on your list of first things to look for. They are not the easiest bugs to find, but they are often the most reliable to exploit once you understand the timing. Here are the concrete steps I use:

• Map the shared state. Identify every memory region that is accessed by more than one agent: CPU cores, DMA engines, interrupt handlers, secure world, normal world. Each shared region is a potential race.
• Look for missing barriers. On ARM64, search the disassembly for shared accesses that are not preceded by a DMB or DSB. On x86-64, look for non-temporal stores and CLFLUSH in security-critical paths.
• Widen the window. Use cache eviction, bus stalling, or fault injection to make the race window large enough to hit reliably. A glitch that stalls a load for ten cycles can turn a one-cycle race into a ten-cycle race.
• Turn the race into a primitive. A race that corrupts a pointer is only useful if you can control the corrupted value. A race that skips a check is only useful if you can then perform the action the check was supposed to prevent.
• Document the hardware behavior. Every race you find is a data point about the silicon’s actual memory model. Write it down. The next race will be easier to find because you will know where the gaps are.

This is not a checklist for a pentest report. It is a way of thinking about the hardware as a set of timing relationships rather than a set of instructions. The race condition is not the bug; it is the evidence that the abstraction is broken.

FAQ

Why are race conditions harder to find than buffer overflows?

Buffer overflows are visible in the source code as an unchecked write. Race conditions are invisible in the source because they are caused by missing synchronization, not by a faulty instruction. The code looks correct, the tests pass, and the bug only appears when the timing is wrong. Finding a race requires modeling the hardware’s memory behavior, not just reading the code.

Can fuzzing find race conditions in firmware?

Rarely. Fuzzing generates inputs and watches for crashes, but a race condition often corrupts state silently or opens a window that requires precise timing to exploit. The fuzzer does not control thread scheduling, cache state, or DMA timing, so it cannot reliably hit the race window. Targeted reverse engineering and fault injection are more effective.

Are hardware-enforced mitigations like SGX and TrustZone immune to race conditions?

No. These mitigations rely on hardware checks that run in parallel with the software they protect. If the checks and the software are not synchronized, the boundary between the trusted and untrusted worlds is a race condition. Both Intel and ARM have documented race-related vulnerabilities in their secure world implementations, and the fundamental design leaves room for more.

What is the best tool for widening a race window on embedded hardware?

A fault-injection rig, such as a ChipWhisperer or a custom voltage glitcher, is the most reliable tool. A voltage glitch can stall a load or store for several cycles, turning a narrow race into a wide one. The key is to trigger the glitch at the right moment, which usually requires a debug interface or a side-channel signal to detect when the race window is about to open.

Next up on the bench: a teardown of a baseband boot ROM race that survives a vendor patch. If you have a target that resists glitching, the race is probably in the clock tree, not the code.

Control Flow Integrity, or CFI, is a catch-all term for a family of mitigations that try to constrain indirect branches—calls, jumps, returns—to a set of targets that are either statically or dynamically approved. On paper, it closes the gap left by W^X and ASLR: even if an attacker gets arbitrary write, they can’t redirect execution to a gadget of their choosing. In practice, the implementations shipping in Windows, macOS, iOS, Android, and various hypervisors are a patchwork of coarse-grained checks, shadow stacks, and compiler-inserted validation that often fails in ways the vendors don’t advertise. This article is a field report on what actually happens when you try to bypass or reproduce those claims on x86-64 and ARM64.

For anyone working at the bench—whether that means extracting a baseband bootloader over JTAG, fault-injecting a SEP, or emulating a UEFI module—CFI is not an abstract security property. It is a set of concrete constraints on the indirect branch instructions you can hijack, the return addresses you can forge, and the exception paths you can abuse. The gap between the vendor’s threat model and the silicon’s actual behavior is where the interesting bugs live.

What CFI Actually Checks

At the instruction level, CFI splits into two broad categories: forward-edge protection for indirect calls and jumps, and backward-edge protection for returns. Forward-edge schemes typically insert a check before an indirect call or jump to verify that the target address belongs to a set of valid function entry points. Backward-edge schemes protect the return address, either by keeping a separate shadow stack or by encrypting the return address with a key stored in a register or memory that the attacker cannot easily read.

The devil is in the granularity. A fine-grained CFI policy would validate every indirect branch against the exact set of functions whose address could legitimately flow to that call site. That requires precise points-to analysis, which is expensive and brittle across shared libraries and JIT code. What ships instead is usually a coarse approximation: all functions with the same signature, all functions in the same module, or all functions whose address has been taken anywhere in the program. Coarse-grained CFI stops some exploits, but it leaves a large gadget space for an attacker who can chain existing call targets.

Windows: Control Flow Guard and CET

Windows Control Flow Guard (CFG) is a forward-edge check inserted by the compiler before indirect calls. The target address is passed to a bitmap lookup that records which addresses are valid function entry points. If the bit is not set, the process terminates. The bitmap is stored in a read-only section, but the check itself is a function call into ntdll!LdrpValidateUserCallTarget or a direct bitmap test in newer builds. The problem is that the bitmap is coarse: any function whose address has been taken is marked valid, regardless of whether that function could legitimately be called from the current site. An attacker who can overwrite a function pointer can redirect it to any other address-taken function in the module. That is not a full bypass, but it is a large reduction in attacker effort.

Intel CET adds a hardware shadow stack for returns and an indirect branch tracking (IBT) mechanism for forward edges. On x86-64, IBT uses the ENDBR64 instruction as a landing pad. If an indirect call or jump lands on an instruction that is not ENDBR64, the CPU raises a control-protection fault. The shadow stack is a separate stack in memory that stores only return addresses, protected by a page-table bit that makes it writable only by dedicated CALL and RET operations. In theory, this is a strong combination. In practice, the shadow stack is only as good as the OS’s handling of setjmp, longjmp, exception unwinding, and signal delivery. Each of those paths must manually adjust the shadow stack pointer, and each adjustment is a potential hole.

macOS and iOS: Pointer Authentication

Apple’s ARM64 devices use Pointer Authentication Codes (PACs). A PAC is a short cryptographic hash of the pointer value and a context value, computed with a key held in a system register. The PAC is stored in the unused high bits of the pointer. Before an indirect branch or a return, the hardware or compiler-inserted code verifies the PAC. If the check fails, the CPU raises an exception. The keys are supposed to be inaccessible to user code, but they are not per-process; they are per-exception-level. A kernel exploit that can read the PAC keys or sign arbitrary pointers defeats the scheme for the entire system.

PAC is not a shadow stack. It authenticates the pointer value, not the control flow path. An attacker who can forge a valid PAC for a chosen target can redirect execution to that target. The PAC is only 16 to 24 bits depending on the pointer size and the reserved bits, so a brute-force attack is possible if the target does not crash the process on failure. Apple mitigates this by rate-limiting PAC failures in some contexts, but the underlying weakness remains: PAC is a probabilistic check, not a deterministic one.

Android and Linux: Clang CFI and Shadow Stacks

Android uses Clang’s forward-edge CFI for the kernel and some userspace components. The compiler generates type-based checks: before an indirect call, it compares the target’s type identifier against the expected type for that call site. The type identifiers are stored in a read-only section, and the check is a simple comparison and branch. This is finer-grained than Windows CFG because it distinguishes function signatures, but it still allows any function with a matching signature to be called. In the kernel, where many functions share the same signature, the gadget space remains large.

Linux has been slower to adopt CFI. The mainline kernel has supported Clang CFI for arm64 since 5.13, but x86-64 support is still not universal. The kernel’s indirect call sites are numerous, and the performance cost of type checks is measurable. Shadow stacks for the kernel are also in progress, but the interaction with ptrace, kprobes, and perf makes the implementation messy. Each of those subsystems can legitimately modify control flow, and each modification must be reflected in the shadow stack.

Where CFI Breaks in Practice

The most reliable bypasses do not attack the CFI check itself. They attack the assumptions underneath it. A shadow stack assumes that the return address is the only thing that matters. But an attacker who can overwrite a saved frame pointer, a function pointer, or a vtable entry can still redirect execution without touching the return address. CFI protects the branch, not the data that feeds the branch.

Another common failure is the exception path. On Windows, structured exception handling (SEH) uses a linked list of exception registration records on the stack. The list is protected by SafeSEH and later by __C_specific_handler validation, but the unwinding itself can be abused. On ARM64, the PAC instruction signs the link register, but the exception return path uses a different register (ELR_EL1 or ELR_EL2) that may not be protected by the same mechanism. A fault injection that corrupts ELR_EL1 during an exception can bypass the return-address check entirely.

JIT code is another weak point. A JIT compiler generates code at runtime, and that code must be marked executable. The CFI policy must either exempt JIT code from checks or dynamically update the valid-target set. Both approaches create windows where an attacker who can write to the JIT code cache can redirect execution. Browsers and language runtimes have spent years hardening their JIT implementations, but the fundamental tension remains: CFI wants a static set of valid targets, and JIT wants to create new targets on the fly.

Fault Injection and CFI

At the bench, fault injection changes the calculus. A voltage glitch or electromagnetic pulse can skip the CFI check instruction entirely, or corrupt the comparison result so that an invalid target is accepted. The check is just a few instructions; skipping it is often easier than finding a gadget that passes the check. This is why hardware-enforced CFI is not a substitute for physical security. If an attacker has physical access to the device, the CFI check is just another instruction to glitch.

On embedded devices with custom firmware, the CFI story is even weaker. Many baseband and automotive ECUs do not use CFI at all, or they use a vendor-specific implementation that has never been publicly audited. The firmware is often extracted over JTAG and reverse-engineered, and the CFI checks, if any, are identified and patched out. The vendor’s claim of “hardware-enforced control flow integrity” often means a single check in the bootloader that can be bypassed with a well-timed glitch.

Reproducing Vendor Claims

When a vendor claims that their system uses CFI, the first step is to reproduce the claim. That means disassembling the firmware or binary and looking for the actual check instructions. On Windows, CFG is visible as a call to LdrpValidateUserCallTarget or a bitmap test before indirect calls. On macOS, PAC is visible as PACIA, PACIB, AUTIA, and AUTIB instructions around function entry and exit. On Android, Clang CFI is visible as a comparison of a type identifier before an indirect call.

The next step is to test the granularity. For each indirect call site, what is the set of valid targets? If the set is large, the CFI is coarse. If the set is small, the CFI is fine-grained. The difference matters because it determines how much work an attacker must do to find a usable gadget. A coarse-grained policy may still stop a simple stack smash, but it will not stop a determined attacker who can chain existing functions.

Finally, test the edge cases. What happens when an exception is thrown? What happens when a signal is delivered? What happens when a JIT compiler generates code? Each of these paths must interact with the CFI mechanism, and each interaction is a potential bypass. The vendors do not document these interactions in detail, so the only way to know is to test them.

What This Means for the Bench

For anyone extracting firmware from a locked-down device, CFI is a hurdle, not a wall. The checks can be identified, analyzed, and bypassed. The bypass may be as simple as finding a valid target that does what you need, or as complex as glitching the check instruction. The key is to understand the specific implementation, not the marketing claim. A vendor that says “hardware-enforced CFI” may mean a shadow stack, a PAC, a bitmap check, or nothing at all. The only way to know is to look at the silicon and the firmware.

The broader lesson is that CFI is a mitigation, not a security boundary. It raises the cost of exploitation, but it does not eliminate the underlying vulnerability. An attacker who can write to memory can still corrupt data, and data corruption is often enough to achieve the attacker’s goal. CFI is one layer in a defense-in-depth strategy, and it should be treated as such.

FAQ

Does CFI stop all code-reuse attacks?

No. CFI constrains indirect branches to a set of valid targets, but the set is often large enough to contain useful gadgets. Coarse-grained CFI, in particular, allows an attacker to chain existing functions that share a signature or module. CFI also does not protect against data-only attacks, where the attacker corrupts non-control data to change the program’s behavior without redirecting execution.

What is the difference between a shadow stack and pointer authentication?

A shadow stack stores a separate copy of the return address on every call and verifies it on every return. It is a deterministic check: if the return address is modified, the check fails. Pointer authentication signs the return address with a cryptographic key and verifies the signature on return. It is a probabilistic check: an attacker who can forge a valid signature can bypass the check. Shadow stacks are stronger in theory, but they require more memory and more complex exception handling.

Can fault injection bypass CFI?

Yes. A voltage glitch or electromagnetic pulse can skip the CFI check instruction or corrupt the comparison result. The check is just a few instructions, and skipping it is often easier than finding a gadget that passes the check. Hardware-enforced CFI is not a substitute for physical security. If an attacker has physical access to the device, the CFI check is just another instruction to glitch.

Why do vendors use coarse-grained CFI instead of fine-grained?

Fine-grained CFI requires precise points-to analysis to determine the exact set of valid targets for each indirect call site. That analysis is expensive, brittle across shared libraries and JIT code, and often produces false positives that break legitimate code. Coarse-grained CFI is cheaper to implement and has lower performance overhead, but it leaves a larger gadget space for attackers. The tradeoff is between security and compatibility.

This article is part of a continuing series on hardware-enforced mitigations. The next installment will examine how CET shadow stacks interact with setjmp and longjmp on x86-64, and whether the exception unwinding path can be abused to corrupt the shadow stack pointer. If you have a specific device or firmware image you would like analyzed, the bench is always open.

Reverse engineering embedded firmware means pulling a binary image out of a microcontroller, baseband processor, UEFI SPI flash, or secure enclave and reconstructing what it actually does without any vendor documentation. The work sits next to static disassembly, dynamic instrumentation, emulation, fault injection, and glitching. On this blog, that means treating every vendor claim about “secure boot” or “hardware root of trust” as a hypothesis to test against silicon, not a marketing slide to believe. If you are here, you probably already know the datasheet is often a work of fiction and the real specification lives in the silicon bugs.

This walkthrough is not a generic “use Ghidra and hope” tutorial. It is a field note on the sequence that works when the target is a locked-down automotive ECU, a baseband image with a custom RTOS, or a UEFI capsule that refuses to parse. The steps assume you have a way to get the image out — JTAG, SPI dump, eMMC read, or fault-injected boot — and that you are willing to treat every abstraction layer as suspect.

Step 1: Acquire the Image Without Trusting the Vendor’s Tool

Vendor update tools are convenient and wrong. They often decrypt, decompress, or re-sign the image before it ever hits your disk, which means you are reversing a transformed artifact, not the firmware the device actually runs. The correct first move is to read the flash directly. For SPI NOR, that means a SOIC-8 clip and a programmer that does not try to be clever. For eMMC, that means a low-level reader that can dump the boot partitions, not just the user area. For baseband or SEP targets, that usually means finding a debug UART or a test point that exposes the internal bus.

If the device has a hardware root of trust that gates the flash, you have two choices: fault injection to bypass the read protection, or emulation of the boot ROM to extract the decryption keys. I have had more luck with the second approach on ARM64 targets where the boot ROM is small enough to emulate in an afternoon. The first approach works better on older microcontrollers where the read-out protection is a single fuse and a well-timed voltage glitch is enough to make the CPU skip the check.

Step 2: Identify the Architecture and Load Address Before You Disassemble

Loading a raw binary into Ghidra or IDA without knowing the base address is a waste of time. The disassembler will happily produce nonsense, and you will spend hours chasing branch targets that point into the middle of nowhere. The first thing to do is look for the vector table. On ARM Cortex-M, that is the first 32-bit word, which is the initial stack pointer, followed by the reset vector. On ARM64, the exception vectors are at a fixed offset from the base of the image, usually 0x0 or 0x200000. On x86-64 UEFI, the image is a PE32+ binary, and the load address is in the optional header.

If the image is encrypted or compressed, you need to find the loader. That is usually a small stub at the beginning of the image that sets up the memory controller, decrypts the payload, and jumps to it. The stub is often not encrypted, because the CPU has to execute it before the decryption engine is initialized. That is your entry point. Emulate the stub, dump the decrypted payload, and then disassemble that.

Step 3: Map the Memory and Peripherals From the Silicon, Not the Datasheet

Datasheets lie. They omit undocumented registers, mislabel interrupt numbers, and sometimes describe a completely different chip revision. The only reliable way to map peripherals is to read the silicon itself. If you have a working device, use a debugger to read the peripheral registers and compare them to the datasheet. If you do not have a working device, use the firmware’s own initialization code as the map. The code that writes to a specific address to configure a UART or a timer is telling you what that address does, even if the datasheet calls it “reserved.”

For baseband and SEP targets, the peripheral map is often split across multiple security domains. The application processor sees one view, the baseband sees another, and the secure enclave sees a third. You need to reconstruct all three views to understand the full attack surface. That means emulating the memory protection unit, not just the CPU.

Step 4: Reconstruct the Boot Flow and Trust Chain

Every locked-down device has a boot flow. The CPU starts in ROM, verifies the first-stage bootloader, which verifies the second-stage, which verifies the OS or application. The verification is usually a hash or a signature check. The bug is usually in the implementation, not the algorithm. Common failure points include:

• Hash comparison that stops after the first mismatch, allowing a timing side channel.
• Signature check that uses a key embedded in the same flash you can read.
• Boot mode pins that bypass verification entirely when strapped a certain way.
• Fault injection that skips the branch instruction after the verification call.

Reconstructing the boot flow means finding the verification code, identifying the key or hash, and then determining whether the check is actually enforced or just decorative. On one automotive ECU I worked on, the “secure boot” checked a CRC32 of the first 4 KB and then jumped to the rest of the image without any further validation. The vendor’s security whitepaper described a multi-stage chain of trust with ECC signatures. The silicon did not care.

Step 5: Disassemble With a Purpose, Not a Prayer

Once you have the image loaded at the correct base address with the correct architecture, you can start disassembling. But do not just scroll through the listing looking for strings. That is how you waste a week. Instead, work backward from the behavior you want to understand. If you want to find the update mechanism, look for the code that writes to flash. If you want to find the crypto, look for the AES S-box or the SHA constants. If you want to find the debug interface, look for the UART initialization and the command parser.

Use the cross-references. Every function that is called from the boot flow is part of the boot flow. Every function that is called from an interrupt handler is part of the interrupt handling. Every function that is called from a command parser is part of the command interface. The call graph is the map. The disassembly is just the terrain.

Step 6: Emulate the Interesting Parts Instead of the Whole System

Full-system emulation of a baseband or SEP is a multi-month project. Emulating a single function is an afternoon. The trick is to extract the function you care about, provide the minimal set of memory and peripheral stubs it needs, and run it in a controlled environment. For ARM64, that means QEMU in user mode or a custom Unicorn script. For x86-64 UEFI, that means a small EFI shell environment. For older ARM or MIPS targets, that means a custom emulator that implements only the instructions and peripherals the function actually touches.

The goal is not to boot the whole system. The goal is to answer a specific question: What does this function do with this input? Does it check a signature? Does it decrypt a payload? Does it write to a protected register? Emulation lets you answer that question without fighting the rest of the firmware.

Step 7: Validate Your Findings Against the Hardware

Reverse engineering is not complete until you have validated your findings against the actual device. That means taking the behavior you reconstructed and testing it on the silicon. If you think a certain command unlocks the debug interface, send that command and see if the UART responds. If you think a certain fault injection point bypasses the signature check, glitch the device and see if it boots your modified image. If you think a certain register controls the memory protection, write to it and see if the device crashes.

Validation is where most reverse engineering projects die. It is easy to produce a plausible-looking disassembly. It is hard to prove that your understanding matches the silicon. The difference is the difference between a blog post and a working exploit.

Common Pitfalls That Are Not in the Vendor’s App Note

Here are the mistakes I see repeatedly, including my own early ones:

• Trusting the string table. Strings are hints, not facts. A string that says “secure boot failed” does not mean the boot is secure. It means someone wrote an error message.
• Assuming the image is not encrypted. Many vendors encrypt the firmware but leave the decryption key in the boot ROM. If you can read the boot ROM, you can decrypt the image. If you cannot read the boot ROM, you need fault injection or emulation.
• Ignoring the memory protection unit. The CPU may be able to execute code from a region that it cannot read as data. That means your disassembler sees garbage, but the CPU sees instructions. You need to model the MPU to get a correct disassembly.
• Forgetting about the second core. Many baseband and automotive SoCs have multiple cores with shared memory. The code you are reversing may be running on a core you are not emulating, and the behavior you are seeing is the result of an inter-core race condition.

FAQ

What is the first step in reverse engineering embedded firmware?

The first step is to acquire the firmware image directly from the flash or storage medium, not through the vendor’s update tool. Vendor tools often transform the image, so a direct read gives you the actual bytes the device executes. For SPI flash, use a clip and programmer. For eMMC, use a low-level reader that can access boot partitions. For protected targets, you may need fault injection or emulation of the boot ROM.

How do I know the correct load address for disassembly?

Look for the vector table or the executable header. On ARM Cortex-M, the first word is the initial stack pointer and the second is the reset vector. On ARM64, the exception vectors are at a fixed offset. On x86-64 UEFI, the image is a PE32+ binary with the load address in the optional header. If the image is encrypted or compressed, find the loader stub, emulate it, and dump the decrypted payload.

What is the most common mistake when reversing firmware?

Trusting the vendor’s documentation or the string table. Datasheets omit registers and mislabel interrupts. Strings are hints, not facts. The only reliable map is the firmware’s own initialization code and the silicon itself. Validate every assumption against the hardware before you build an exploit or a patch.

Do I need to emulate the entire system to reverse a function?

No. Emulate only the function you care about, with minimal stubs for the memory and peripherals it touches. Full-system emulation is a multi-month project. A single function can be emulated in an afternoon with QEMU user mode, Unicorn, or a custom emulator. The goal is to answer a specific question, not to boot the whole device.

Next up on the bench: a teardown of a baseband boot ROM that uses a custom hash function the vendor calls “proprietary.” Spoiler: it is a truncated SHA-256 with a broken IV. If you have a target you want me to look at, send the image and a scope trace. No vendor whitepapers, please.

Build systems are the least trusted part of the software supply chain, and the most trusted by the people who consume the output. A build system is the collection of compilers, linkers, package managers, signing tools, CI runners, and configuration files that turn source code into a distributable artifact. Adjacent concepts include reproducible builds, provenance attestation, hermetic toolchains, and binary transparency. For anyone who has spent time extracting firmware from a locked-down baseband or glitching a secure enclave, the irony is obvious: we spend years reverse-engineering trust boundaries in silicon, while the industry ships binaries produced by a pile of shell scripts running on rented x86-64 instances with no meaningful integrity story. This matters because a compromised build system can inject code that no amount of runtime mitigation will catch. The hardware will faithfully execute whatever the build produced.

I have spent enough time staring at disassembled UEFI images and SEP firmware to know that the most elegant microarchitectural defense is useless if the attacker owns the compiler. Supply chain attacks target build systems because that is where trust is manufactured. The rest of this article is about how that trust gets broken, what the vendors claim, and why the abstractions keep failing.

The Build System as a Trust Anchor

Every signed binary is a statement: the organization that holds the signing key asserts that this artifact corresponds to a particular source revision and was produced by a known process. The signing key is not the interesting part. The interesting part is everything that happens before the signature is applied. Compilers, linkers, post-link optimizers, code-signing wrappers, and the CI orchestrator itself all sit in the trust path. If any of them is subverted, the signature is still valid. The key did exactly what it was told.

This is not a theoretical concern. The 2021 SolarWinds incident showed how a build environment compromise can propagate to thousands of downstream organizations. The attackers did not need to break any cryptographic primitive. They modified source code in the build environment and let the legitimate signing process do the rest. The resulting binaries were signed, distributed, and trusted. The hardware-enforced mitigations on the target machines — SMEP, SMAP, CET, pointer authentication — did not matter because the malicious code was part of the legitimate application.

For firmware and embedded targets, the problem is worse. A compromised build system for a baseband image or a UEFI module can inject code that runs before the OS loads, outside the visibility of most endpoint detection tools. The attacker does not need a zero-day in the target. They need a zero-day in the build pipeline, which is usually a much softer target.

Why Build Systems Are Soft Targets

Build systems are soft targets for the same reason that test infrastructure is a soft target: they are treated as internal plumbing, not as production systems. The people who run them are often under pressure to ship, not to audit. The machines are ephemeral, the configuration is sprawling, and the security model is usually “the CI runner can do anything.”

Consider the typical CI setup for a firmware project. A developer pushes a commit. The CI runner checks out the source, pulls in dependencies from a package registry, runs a build script, and produces a signed artifact. The runner has access to the signing key, or at least to a signing service that does not distinguish between a legitimate build and a malicious one. The build script is often a shell script with no sandboxing, no network egress controls, and no verification of the toolchain it is using. If an attacker can modify the build script, or the package registry, or the toolchain cache, they own the output.

This is not a failure of cryptography. It is a failure of process. The signing key is doing its job. The problem is that the build system is not a trustworthy environment, and the signature does not attest to the environment. It attests to the artifact, which is a different thing entirely.

The Broken Abstraction: Signatures vs. Provenance

The core abstraction that breaks here is the idea that a signature is a statement about the artifact’s origin. It is not. A signature is a statement that the holder of the private key approved the artifact. The holder of the private key is usually a machine, not a human, and the machine does not know whether the artifact is the result of a legitimate build or a compromised one.

Provenance attestation is supposed to fix this. The idea is that the build system records what source revision was used, what toolchain was used, what dependencies were pulled, and what commands were run. The artifact is then signed along with this provenance data. In theory, a consumer can verify that the artifact corresponds to a known source revision and a known build process. In practice, the provenance data is only as trustworthy as the system that generated it. If the build system is compromised, the provenance data is compromised too.

This is the same problem that reproducible builds try to solve from a different angle. A reproducible build is one where the same source revision and the same toolchain produce a byte-for-byte identical artifact, regardless of where the build is run. This makes it possible to detect tampering by comparing artifacts from independent builders. But reproducible builds are hard to achieve in practice, especially for firmware and embedded targets where the toolchain is often proprietary, the build process is not documented, and the output is not deterministic. The vendors claim reproducibility, but the claims rarely survive contact with a real build environment.

What the Vendors Claim

Vendors love to talk about secure boot, signed firmware, and hardware root of trust. These are real mechanisms, and they do provide meaningful protection against runtime tampering. But they do not protect against a compromised build system. If the attacker injects code at build time, the code is signed by the legitimate key, and the secure boot chain will happily load it. The hardware root of trust is not a root of trust in the build process. It is a root of trust in the boot process. Those are different things.

I have seen vendor documentation that describes a “secure development lifecycle” as if it were a technical control. It is not. It is a process document. The actual technical controls are often absent: no hermetic builds, no signed toolchains, no verification of dependencies, no isolation between build stages. The signing key is stored on a CI runner that also runs arbitrary test code. The build script pulls dependencies from a public registry without pinning versions or verifying hashes. The toolchain is installed from a tarball that nobody has audited.

This is not a hypothetical. The 2022 CISA advisory on the XZ Utils backdoor described a supply chain compromise that targeted a widely used compression library. The attacker spent years building trust in the project, then introduced a backdoor into the build process. The backdoor was not in the source code in an obvious way; it was in the build scripts and the test infrastructure. The resulting binaries were signed and distributed by the legitimate maintainers. The attack was caught by accident, not by any systematic control.

The Microarchitectural Angle

For readers of this blog, the interesting question is not whether supply chain attacks happen. It is how they interact with the hardware-enforced mitigations we spend so much time studying. The answer is that they bypass them entirely. A build-time injection is not a runtime exploit. It does not need to defeat ASLR, or bypass CET, or forge a pointer authentication code. It is simply part of the legitimate code. The CPU executes it because the CPU is designed to execute whatever the binary says, and the binary says it is legitimate.

This is why I am skeptical of vendor claims about “hardware-enforced security.” The hardware is not the problem. The problem is the software supply chain that produces the binaries the hardware runs. A secure enclave is only as secure as the code that runs inside it. If that code was compiled by a compromised toolchain, the enclave is compromised. The microarchitectural details do not matter.

There is a deeper point here about the nature of trust. Hardware-enforced mitigations are designed to protect against a particular class of attacks: runtime exploitation of memory corruption vulnerabilities. They do nothing against a build-time injection. The attacker does not need to corrupt memory. They need to corrupt the build process. The result is a binary that is malicious by design, not by accident.

Practical Examples from the Bench

Let me give you a concrete example from my own work. I was extracting firmware from an automotive ECU a while back. The firmware was signed, and the signature verified against a public key stored in the boot ROM. The boot ROM was read-protected, so I had to use fault injection to dump it. That is a hardware problem, and it is the kind of problem this blog usually covers. But once I had the firmware image, I found something more interesting: the build system that produced it had left behind a build path string that included a username and a hostname. The username was “jenkins.” The hostname was an internal CI server. The firmware was built on a Jenkins instance that was probably not isolated from the rest of the corporate network.

That is a supply chain vulnerability hiding in plain sight. The vendor had signed the firmware, but the build environment was a Jenkins server with a predictable username and an internal hostname. If an attacker had compromised that Jenkins server, they could have injected code into the firmware, and the signature would have been valid. The hardware root of trust would have loaded it without complaint. The only reason I noticed the build path string is that I was looking for clues about the build process, not because the vendor had disclosed anything about it.

Another example: I was working on a UEFI module for a laptop. The module was signed, and the signature verified against the platform key. But the build script that produced the module pulled in a precompiled object file from a third-party vendor. The object file was not signed, not hashed, and not documented. It was just a blob that got linked into the final image. If that blob had been malicious, the signature on the final image would have been valid. The platform key would have trusted it. The secure boot chain would have loaded it. The hardware would have executed it.

These are not exotic attacks. They are the result of treating the build system as a black box that produces trustworthy output. The trust is misplaced. The build system is not a black box. It is a collection of scripts, tools, and dependencies that are all subject to compromise.

What Actually Works

If you want to defend against supply chain attacks on build systems, you need to start with the assumption that the build system is compromised. That is the only safe assumption. From there, you can design controls that limit the damage.

The first control is hermetic builds. A hermetic build is one that does not depend on anything outside a declared set of inputs. The toolchain is pinned, the dependencies are pinned, the build script is pinned, and the environment is isolated. This makes it harder for an attacker to inject code through a dependency or a toolchain update. It also makes the build reproducible, which is a prerequisite for independent verification.

The second control is provenance attestation. This is not a silver bullet, but it is better than nothing. The idea is to record what went into the build and sign that record along with the artifact. The record is only as trustworthy as the system that generated it, but if the system is isolated and the record is independently verifiable, it can catch some attacks. The key is to make the provenance data useful, not just a JSON blob that nobody reads.

The third control is independent verification. This is the hardest one, but it is the most effective. If you can build the same source revision on an independent system and compare the output, you can detect tampering. This is what reproducible builds are for. It is not easy, especially for firmware and embedded targets, but it is the only control that actually catches a compromised build system. Everything else is just making the attack harder.

There is a fourth control that is often overlooked: treat the build system as a production system. That means access controls, network segmentation, audit logging, and incident response. The build system is not a development tool. It is the thing that produces the artifacts you ship. If you would not let a developer run arbitrary code on a production server, you should not let them run arbitrary code on a build server.

The Vendor Abstraction Problem

The reason supply chain attacks keep working is that vendors keep selling abstractions that do not hold. “Secure boot” is an abstraction. “Hardware root of trust” is an abstraction. “Signed firmware” is an abstraction. These abstractions are useful, but they are not complete. They protect against runtime tampering, not build-time injection. The vendors know this, but they do not talk about it because it is not a problem they can solve with a marketing slide.

The same is true of “reproducible builds.” The vendors claim reproducibility, but the claims are often aspirational. A build is reproducible only if the toolchain is deterministic, the dependencies are pinned, and the environment is controlled. Most firmware builds are none of those things. The toolchain is a proprietary compiler with undocumented behavior. The dependencies are pulled from a vendor portal that does not provide hashes. The environment is a developer’s laptop with a random set of packages installed. The result is a build that is reproducible only in the sense that it produces the same output on the same machine on the same day.

I am not saying these abstractions are useless. They are useful. But they are not sufficient. The problem is that they are sold as sufficient, and the people who buy them do not ask the hard questions. The hard question is not “is the firmware signed?” The hard question is “what produced the firmware, and can I trust that process?”

What This Means for Firmware Extraction and Analysis

For those of us who spend our time extracting firmware from locked-down devices, supply chain attacks are both a threat and an opportunity. They are a threat because the firmware we extract may already be compromised. They are an opportunity because the build system is often the weakest link in the chain, and the artifacts it leaves behind can be a goldmine of information.

Build path strings, usernames, hostnames, toolchain versions, dependency hashes, and CI configuration files are all clues about the build environment. They can tell you who built the firmware, where it was built, and what tools were used. They can also tell you whether the build environment was isolated or exposed. If you are doing firmware analysis, these clues are worth more than a thousand hours of disassembly. They tell you where to look.

This is a topic I plan to return to. The next logical step is a deep dive into reproducible builds for firmware, with a focus on the specific challenges of embedded toolchains and the ways vendors fail to meet their own claims. If you have a firmware image with a suspicious build path string, or a vendor that claims reproducibility but does not deliver, I would like to hear about it. The bench is always open.

Frequently Asked Questions

Why do attackers target build systems instead of runtime vulnerabilities?

Build systems are softer targets than runtime environments. They are often less monitored, less isolated, and less understood by the people who run them. A successful build system compromise produces a signed, legitimate-looking artifact that bypasses runtime mitigations entirely. The attacker does not need to defeat ASLR, CET, or pointer authentication because the malicious code is part of the legitimate binary. The hardware will execute it without complaint.

Does secure boot protect against supply chain attacks?

No. Secure boot verifies that the firmware or bootloader is signed by a trusted key. It does not verify that the signed artifact is the result of a legitimate build process. If the build system is compromised, the attacker can inject code into the artifact before it is signed. The signature is valid, the secure boot chain is satisfied, and the malicious code runs. Secure boot protects against runtime tampering, not build-time injection.

What is the difference between a signature and provenance attestation?

A signature is a cryptographic statement that the holder of a private key approved an artifact. It says nothing about how the artifact was produced. Provenance attestation is a record of the build process: what source revision was used, what toolchain was used, what dependencies were pulled, and what commands were run. The provenance record is signed along with the artifact. In theory, this gives consumers a way to verify that the artifact corresponds to a known build process. In practice, the provenance record is only as trustworthy as the system that generated it.

Are reproducible builds a practical defense for firmware?

Reproducible builds are the most effective defense against build system compromise, but they are hard to achieve for firmware and embedded targets. The toolchain is often proprietary, the build process is not documented, and the output is not deterministic. Vendors claim reproducibility, but the claims rarely survive contact with a real build environment. Independent verification is the goal, but it requires a level of discipline that most firmware teams do not have.

If you want to dig deeper into the hardware side of this problem, the next article will look at how fault injection can be used to extract boot ROMs from read-protected microcontrollers, and what that means for the trust model of secure boot. The build system is where trust is manufactured. The boot ROM is where it is enforced. Both are worth understanding.

eBPF is a kernel-resident virtual machine that lets you attach sandboxed programs to tracepoints, kprobes, uprobes, and a growing set of LSM hooks without loading a kernel module. It sits adjacent to the same machinery that enforces CET shadow stacks, PAC return-address signing, and MTE tag checks, but it does not magically inherit their guarantees. For anyone doing low-level vulnerability research on x86-64 or ARM64, eBPF is less a security product and more a way to inspect what the kernel, scheduler, and compiler-generated code actually do before an abstraction layer lies to you.

This guide is for people who already distrust vendor claims about runtime visibility. It covers what eBPF can observe, where its isolation model breaks down, and how to use it as an instrumentation tool rather than a replacement for hardware-enforced control-flow integrity.

What eBPF Actually Is

eBPF is an in-kernel register-based VM with a verifier that attempts to prove memory safety and termination before a program is JIT-compiled to native x86-64 or ARM64 instructions. The verifier is the interesting part: it performs abstract interpretation over the eBPF instruction stream, tracks register types, and rejects programs that could read arbitrary kernel memory or loop indefinitely. In practice, the verifier is a large C program with a long history of privilege-escalation bugs, which should temper any claim that eBPF is inherently safe.

The execution model is event-driven. You attach a program to a hook, the kernel invokes it when the event fires, and the program can read a limited context structure, update maps, and in some cases modify the event’s outcome. The hooks that matter for security monitoring are:

• Tracepoints — static markers in the kernel source, such as sys_enter_execve or sched_process_exec.
• kprobes/kretprobes — dynamic probes on kernel function entry and return, useful for functions that lack tracepoints.
• LSM hooks — security module callbacks that can deny operations, not just observe them.
• Network hooks — XDP, TC, and socket filters that see packets before or after the network stack.

Each hook exposes a different slice of kernel state. A tracepoint gives you a stable ABI-ish structure. A kprobe gives you raw register state and a function name. An LSM hook gives you the ability to block. Conflating these is how vendor marketing produces nonsense like “full kernel visibility.”

Why Security Teams Adopt eBPF

The standard pitch is that eBPF replaces kernel modules, auditd, and ptrace-based monitoring with lower overhead and fewer stability risks. There is some truth to that. A well-written eBPF program attached to a tracepoint can run in microseconds, and the verifier prevents the most obvious crashes that plague out-of-tree kernel modules.

But the real reason security teams adopt eBPF is that it gives them a programmable filter in the kernel. Instead of shipping every syscall event to userspace and filtering there, you can filter in the kernel, aggregate in maps, and only wake userspace when something interesting happens. That is a genuine architectural improvement over auditd, which is essentially a syscall-level firehose with a userspace parser.

For low-level researchers, eBPF is also a cheap way to instrument kernel behavior without rebuilding the kernel or fighting with ftrace’s limited scripting. You can attach a kprobe to copy_from_user, record the size and destination, and correlate that with page-fault activity. You can trace do_mmap and see exactly what the dynamic linker is doing. You can watch flush_icache_range on ARM64 and catch JIT code generation in real time.

The Isolation Model Is Not a Security Boundary

Here is where vendor claims need scrutiny. eBPF programs run in kernel context, but they are not isolated from the kernel in any hardware-enforced sense. The verifier is a software gate. If the verifier has a bug, or if the JIT compiler emits incorrect native code, the eBPF program can read or write arbitrary kernel memory. This has happened repeatedly.

On x86-64, the JIT compiler emits native code that runs with the kernel’s normal privilege level. There is no separate page table, no ring transition, and no CET shadow stack for eBPF programs unless the kernel explicitly enables it. On ARM64, the situation is similar: eBPF JIT output runs at EL1, and PAC is not automatically applied to eBPF-generated code. If you are relying on eBPF as a security boundary, you are relying on a C program that has been wrong before.

The correct mental model is that eBPF is a constrained execution environment, not a sandbox. It reduces the attack surface of kernel instrumentation, but it does not eliminate it. Anyone who tells you otherwise is selling something.

What eBPF Can Actually See

Let’s be concrete. Here is what eBPF can observe at each major hook type, and what it cannot.

Syscall Tracepoints

Attaching to sys_enter_execve gives you the filename, argv, and envp pointers. You can read the filename string with bpf_probe_read_user or bpf_probe_read_kernel, depending on the kernel version. You can record the process’s PID, UID, and cgroup. You cannot see the file’s contents, the ELF headers, or what the process will do after execve returns. You also cannot see syscalls that bypass the tracepoint, such as direct int 0x80 invocations on x86-64, which still work and still hit the syscall table but may not trigger the same tracepoint path in all kernel versions.

kprobes on Memory Management

A kprobe on do_mmap or __vmalloc shows you the requested size, flags, and protection bits. This is useful for detecting JIT spraying, where an attacker allocates executable memory and fills it with shellcode. But a kprobe only sees the function entry and exit. It does not see the page-table walk, the TLB flush, or the actual physical page allocation. On ARM64 with MTE enabled, a kprobe does not see the tag assignment unless you also probe the MTE-specific functions, which are not always exported.

LSM Hooks

LSM hooks are the only eBPF attachment points that can deny an operation. A program attached to file_open can return -EPERM and block the open. This is powerful, but it is also a policy enforcement point, not an observation point. If you use LSM hooks for monitoring, you are changing kernel behavior, and that has consequences for stability and correctness. A buggy LSM program can prevent the system from booting or lock out legitimate processes.

Network Hooks

XDP programs see packets before the network stack allocates an skb. This is the lowest-overhead packet filtering point in the kernel. You can drop, redirect, or modify packets at line rate on many NICs. But XDP does not see TCP state, connection tracking, or application-layer data without additional parsing. TC hooks see packets after the stack has done some work, which means more context but also more overhead.

Where eBPF Breaks Down

eBPF’s limitations are not always obvious from the documentation. Here are the ones that matter for security monitoring.

Verifier Complexity

The verifier is a multi-thousand-line C program that attempts to prove properties about eBPF bytecode. It has known limitations: it cannot handle loops with variable bounds, it has a maximum instruction count, and it rejects some programs that are actually safe. This means you will spend time restructuring code to satisfy the verifier, and you will occasionally hit verifier bugs that cause false positives or false negatives. The verifier is also version-dependent: a program that passes on Linux 6.1 may fail on 6.6 because the verifier’s analysis changed.

Spectre Mitigations

On x86-64, the kernel applies Spectre mitigations to eBPF programs, including retpolines and, on some CPUs, eIBRS. This adds overhead and changes the native code that the JIT emits. If you are trying to measure exact instruction counts or cache behavior, eBPF is not a clean instrument. The JIT output is also affected by the kernel’s hardening options, such as CONFIG_BPF_JIT_ALWAYS_ON and CONFIG_RETPOLINE.

ARM64 PAC and BTI

On ARM64, the kernel can enable pointer authentication and branch target identification for eBPF JIT output, but this is not universal. Some kernels disable PAC for eBPF because the verifier does not model PAC correctly. If you are researching PAC bypasses, eBPF is not a reliable way to test them. The JIT output may or may not have PAC instructions, and the verifier may reject programs that manipulate pointers in ways that PAC would otherwise protect.

Map Semantics

eBPF maps are the shared memory between eBPF programs and userspace. They are not coherent with the CPU cache in the way you might expect. On x86-64, map updates use atomic operations, but the memory ordering is not always what you want. On ARM64, the kernel uses stlr and ldar for some map operations, but not all. If you are using eBPF to detect race conditions or memory-ordering bugs, you need to understand the exact instructions the JIT emits for map access.

Practical Examples for Low-Level Researchers

Here are three concrete use cases that fit this blog’s focus.

Detecting JIT Code Generation

Attach a kprobe to bpf_int_jit_compile on x86-64 or bpf_jit_compile on ARM64. Record the program’s instruction count and the address of the JIT output. Correlate that with do_mmap calls that request PROT_EXEC. This gives you a timeline of when executable memory is allocated and when code is written to it. It does not give you the code itself, but it narrows the search space for JIT spraying.

Tracing Page-Fault Handling

Attach a kprobe to do_page_fault and record the faulting address, the error code, and the current process. On x86-64, the error code tells you whether the fault was a protection violation or a not-present fault. On ARM64, the equivalent is do_mem_abort, which gives you the fault status register. This is useful for detecting attempts to probe kernel memory from userspace, which is a common first step in privilege-escalation exploits.

Monitoring Firmware Extraction Attempts

On embedded devices, firmware extraction often involves reading from /dev/mem or a baseband-specific device node. Attach an LSM hook to file_open and filter for paths that match /dev/mem or /dev/kmem. Record the PID, UID, and parent process. This does not prevent the read, but it gives you an audit trail. If you want to block the read, you can return -EPERM from the LSM hook, but be prepared for the device to misbehave if a legitimate process needs that access.

Tooling That Does Not Suck

The eBPF tooling ecosystem is fragmented, but a few tools are worth using.

• bpftrace — a high-level tracing language that compiles to eBPF. Good for quick experiments, but the abstraction leaks when you need precise control over the generated code.
• libbpf — the C library for loading eBPF programs. This is the lowest-level stable interface, and it is what you should use for production monitoring.
• cilium/ebpf — a Go library that generates eBPF programs from Go code. Useful if you are already in a Go environment, but the generated code is not always what you expect.
• bpftool — the Swiss Army knife for inspecting loaded programs, maps, and JIT output. Use bpftool prog dump jited to see the native instructions.

For low-level work, bpftool prog dump jited is essential. It shows you the exact x86-64 or ARM64 instructions that the JIT emitted, which is the only way to verify that the verifier’s model matches reality.

Vendor Claims vs. Reality

Several commercial products claim to use eBPF for “runtime security” or “zero-trust workload protection.” The marketing usually omits the following:

• eBPF programs run with kernel privileges. A verifier bug is a kernel bug.
• eBPF does not see everything. It sees what the hooks expose, and the hooks are not comprehensive.
• eBPF overhead is not zero. The verifier adds compile-time overhead, and the JIT output adds runtime overhead. On some workloads, the overhead is measurable.
• eBPF is not a replacement for hardware-enforced mitigations. CET, PAC, and MTE operate at a different level of the stack. eBPF can observe some of their effects, but it cannot enforce them.

If a vendor claims their eBPF agent provides “complete visibility,” ask them to show the kprobe that sees a PAC authentication failure on ARM64. There is not one, because PAC failures are handled by the hardware before the kernel’s exception handler runs, and the exception handler does not expose the failed authentication context to eBPF.

What to Build Next

If you are setting up eBPF monitoring on your own systems, start with syscall tracepoints and a small set of kprobes on memory-management functions. Use bpftool prog dump jited to verify the JIT output. Do not trust the verifier blindly; read the generated native code and look for places where the verifier’s assumptions might not hold.

For this blog, the natural next step is a deep dive into the eBPF verifier’s abstract interpretation algorithm, with a focus on the register-state tracking that prevents out-of-bounds reads. That is a topic that deserves its own article, and it connects directly to the microarchitectural focus of this site.

FAQ

Can eBPF detect Spectre or Meltdown attacks?

Not directly. Spectre and Meltdown are microarchitectural attacks that exploit speculative execution. eBPF programs run after speculation has been resolved, so they cannot see the transient execution window. What eBPF can do is detect the effects of a Spectre attack, such as unusual cache-access patterns or attempts to read kernel memory from userspace. But that is indirect evidence, not direct observation.

Is eBPF safe to run on production systems?

It depends on what you mean by safe. The verifier prevents most memory-safety bugs, but it is not a formal proof. The JIT compiler has had bugs that produced incorrect native code. The kernel’s eBPF subsystem has had privilege-escalation vulnerabilities. If you are running eBPF on a production system, you are accepting a small but nonzero risk of kernel compromise. For most security-monitoring use cases, that risk is acceptable. For high-assurance systems, it may not be.

How does eBPF interact with CET, PAC, and MTE?

On x86-64, eBPF JIT output is subject to the same CET shadow-stack and IBT enforcement as other kernel code, but only if the kernel is built with those options. On ARM64, PAC and BTI for eBPF JIT output are optional and not always enabled. MTE operates at the memory-tag level and is orthogonal to eBPF; eBPF programs do not see MTE tags unless they explicitly probe the MTE-specific kernel functions. In general, eBPF does not weaken these mitigations, but it does not strengthen them either.

What is the difference between eBPF and kernel modules for security monitoring?

A kernel module has full access to kernel memory and can do anything the kernel can do. eBPF is constrained by the verifier and the hook definitions. A kernel module can crash the kernel with a single bad pointer dereference. eBPF is supposed to prevent that, but the verifier is not perfect. The practical difference is that eBPF programs are easier to load and unload, do not require kernel headers, and are less likely to cause a kernel panic. But they are also less capable. If you need to hook a function that has no tracepoint or kprobe-accessible path, a kernel module may be your only option.

eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the kernel without loading a module or touching source code. For security monitoring, that sounds like a clean win: deep visibility into syscalls, network flows, and file access, all wrapped in a verifier that promises safety. But if you’ve spent any time staring at CPU pipeline diagrams or pulling apart baseband firmware, you know that “safety” is a contract written in sand. The real question isn’t whether eBPF can spot an attack. It’s whether the monitoring infrastructure itself becomes the most reliable pivot point for someone who understands speculative execution, cache coherency, and the gap between what the verifier proves and what the silicon actually does.

This article picks apart eBPF-based security monitoring through the lens of microarchitectural attack surface, kernel memory management, and the compiler toolchains that turn C into verified eBPF bytecode. We’ll look at where the verifier’s formal model diverges from physical reality, how JIT-compiled eBPF programs interact with CPU mitigations, and why the current generation of eBPF security products may be handing attackers the very primitives they claim to detect.

The eBPF Architecture Model vs. Microarchitectural Reality

eBPF programs run inside a lightweight virtual machine in the Linux kernel. Before execution, the verifier performs static analysis to prove termination, memory safety, and the absence of out-of-bounds accesses. Then the program is JIT-compiled to native x86-64 or ARM64 instructions. Security monitoring tools—Falco, Cilium, Tracee, and various commercial EDR products—use eBPF probes attached to kprobes, tracepoints, or syscall entry points to watch system behavior.

The verifier’s safety guarantees rest on a CPU model that excludes speculative execution, cache side channels, and branch prediction state. This is the same class of abstraction failure that gave us Spectre, Meltdown, and their variants. When an eBPF program is JIT-compiled to native code, it emits real x86-64 or ARM64 instructions that interact with the branch predictor, fill cache lines, and occupy entries in the BTB (Branch Target Buffer) and RSB (Return Stack Buffer)—all shared resources on SMT cores.

The Verifier’s Blind Spot: Speculative Execution Paths

The eBPF verifier proves properties about architectural execution—the path the program takes when branch conditions resolve correctly. It does not, and architecturally cannot, reason about speculative execution paths where the CPU guesses a branch direction and executes instructions transiently. A JIT-compiled eBPF program that does a bounds check followed by a memory access looks safe to the verifier. But on a core vulnerable to Spectre v1, an attacker controlling the unchecked input can train the branch predictor to mispredict the bounds check, causing speculative access to out-of-bounds memory.

This isn’t a thought experiment. Researchers have shown that eBPF programs can be used as speculative execution gadgets. The verifier’s own analysis routines run in kernel context and can be influenced by unprivileged users—a fact documented in CVE-2021-33624 and related vulnerabilities. When you deploy an eBPF-based security monitor, you’re adding verified-but-speculatively-unsafe code to a kernel that may already be running with SMT enabled and mitigations turned off for performance.

JIT Compilation and the Transient Execution Window

The eBPF JIT compiler translates verified bytecode into native instructions. On x86-64, that means emitting actual mov, call, and ret instructions that interact with the CPU’s front-end, execution units, and memory subsystem. Each JIT-compiled eBPF program becomes a sequence of native instructions that can:

• Occupy BTB entries, potentially evicting entries used by kernel mitigation code
• Generate speculative loads that fill cache lines, creating measurable timing differences
• Execute transient instructions under misprediction that leave microarchitectural state changes

On ARM64 systems with Pointer Authentication (PAC), the situation gets more tangled. eBPF programs running with PAC enabled must handle signed return addresses. The verifier doesn’t model PAC signing or authentication—it sees abstract eBPF instructions, not the PACIASP and AUTIASP instructions the JIT compiler emits. A JIT-compiled eBPF program that corrupts a signed pointer won’t be caught by the verifier; it’ll be caught by a PAC authentication failure at runtime. But that failure generates a fault that may be observable through timing or other side channels, potentially leaking information about the corrupted pointer’s value.

eBPF Security Monitors as Attack Surface

Security monitoring tools that use eBPF typically load multiple programs into the kernel: syscall probes, network filters, file integrity watchers. Each loaded program is a potential gadget. The more comprehensive the monitoring, the larger the attack surface. This creates an uncomfortable trade-off: the very instrumentation meant to detect attacks may provide the primitives needed to construct them.

Map Leakage and Covert Channels

eBPF maps are shared memory regions between kernel and userspace. Security monitors use them to pass event data to userspace agents. While maps have access controls, the timing of map operations can create covert channels. A compromised userspace process with access to an eBPF map can observe:

• Map update frequency, revealing system call patterns of other processes
• Map lookup latency, potentially exposing kernel data structure states
• Map eviction behavior, leaking information about kernel memory pressure

These channels are low-bandwidth but can be enough to leak cryptographic key material or ASLR randomization bits. The eBPF verifier explicitly permits bounded loops since Linux 5.3, which means timing side channels through loop iteration counts are now verifier-approved in certain configurations.

Kernel Stack Probing via eBPF

eBPF programs can access limited kernel stack data through helper functions like bpf_get_stackid() and bpf_get_stack(). These helpers return stack trace information that includes return addresses—the very values that KASLR (Kernel Address Space Layout Randomization) attempts to hide. While the returned addresses are hashed or masked in some configurations, the raw values may be recoverable through repeated sampling and statistical analysis, especially on kernels with limited entropy in their KASLR implementation.

On ARM64 systems with PAC, stack return addresses are signed. But the eBPF helper returns the raw signed pointer, not the authenticated value. An attacker who can observe these signed pointers and trigger controlled PAC authentication failures can potentially forge valid signed pointers—a technique that has been demonstrated in academic research against userspace PAC but has clear kernel-space implications.

Firmware-Level eBPF: The Hidden Frontier

While most eBPF discussion focuses on the Linux kernel, eBPF is increasingly appearing in firmware contexts. UEFI firmware can include eBPF-based network drivers. Embedded baseband processors in mobile devices are exploring eBPF for packet filtering. These environments lack the mature verifier implementations of mainline Linux and often run on CPUs with different microarchitectural characteristics.

Consider a smartphone baseband processor running a lightweight RTOS with an eBPF interpreter for network packet filtering. The interpreter may not implement all verifier checks, especially those related to speculative execution. A crafted packet that triggers an eBPF program could exploit microarchitectural side channels in the baseband CPU—a CPU that typically has direct DMA access to application processor memory. This isn’t theoretical; baseband-to-AP attacks have been demonstrated using other vectors, and eBPF provides a convenient programmable interface.

Verifier Divergence Across Kernel Versions

The eBPF verifier is not a static specification; it evolves with each kernel release. A program that passes verification on kernel 5.15 may fail on 6.1 due to stricter checks, or—more concerning—a program that fails on 6.1 may pass on a vendor kernel with backported features but incomplete verifier updates. Android OEMs are notorious for shipping kernels with cherry-picked eBPF features that don’t include the corresponding verifier improvements.

This creates a fragmentation problem: security monitoring tools that rely on eBPF must target the lowest-common-denominator verifier, which means they can’t use newer safety features. Or they must maintain per-kernel-version program variants, which increases complexity and the risk of deploying a variant with insufficient safety checks.

Practical Recommendations for the Skeptical Engineer

If you’re deploying eBPF-based security monitoring, here are concrete steps to reduce the risk of your monitoring becoming the attack vector:

• Disable SMT on security-critical hosts. Simultaneous multithreading is the primary enabler of cross-thread microarchitectural attacks. If you’re running eBPF programs that access sensitive kernel data, SMT effectively gives an attacker a co-resident thread on the same physical core.
• Audit eBPF map access patterns. Monitor which userspace processes have read access to eBPF maps. A process with CAP_BPF or CAP_SYS_ADMIN that shouldn’t need map access is a red flag.
• Pin eBPF programs to specific cores. If your monitoring workload can be isolated to dedicated cores, you reduce the microarchitectural attack surface. This is especially relevant on ARM64 big.LITTLE systems where core types have different speculative execution behaviors.
• Verify the verifier. Run your eBPF programs through multiple kernel versions’ verifiers in a CI pipeline. Flag any program that passes on an older verifier but fails on a newer one—it may be exploiting a verifier bug.

FAQ

Does eBPF’s verifier guarantee that a program is safe against all attacks?

No. The verifier proves safety within its formal model, which covers memory access bounds, loop termination, and instruction validity. It does not model CPU speculative execution, cache timing, branch predictor state, or other microarchitectural behaviors. A program that passes verification can still be used as a gadget in Spectre-type attacks or leak information through timing side channels. The verifier is a necessary but insufficient safety mechanism.

How does eBPF JIT compilation affect security on ARM64 systems with Pointer Authentication?

On ARM64 systems with PAC enabled, the JIT compiler emits PACIASP and AUTIASP instructions to sign and authenticate return addresses. However, the verifier operates on eBPF bytecode and has no visibility into these PAC instructions. A JIT-compiled eBPF program that corrupts a signed pointer will trigger a PAC authentication fault, which may be observable through timing or other channels. Additionally, eBPF helper functions that return kernel pointers (like bpf_get_stack()) may expose signed pointer values that an attacker can use to forge valid PAC signatures.

Can eBPF-based security monitoring tools be used to bypass CET (Control-flow Enforcement Technology)?

Indirectly, yes. CET’s shadow stack protects return addresses, but eBPF programs that run in kernel context can access and modify memory through helper functions. If an attacker compromises a userspace process that has access to eBPF maps, they may be able to influence kernel execution flow by manipulating map contents that are consumed by eBPF programs. Additionally, eBPF programs themselves are JIT-compiled to native code and placed in kernel memory; if an attacker can locate and modify that memory (through a separate kernel vulnerability), they can bypass CET protections for the eBPF program’s execution.

What are the risks of using eBPF for security monitoring on systems with MTE (Memory Tagging Extension)?

MTE assigns 4-bit tags to memory allocations and checks them on access. eBPF programs that access kernel memory through helpers like bpf_probe_read() may interact with MTE-tagged memory in ways the verifier doesn’t anticipate. If an eBPF program reads MTE-tagged memory and the tag check fails, the resulting fault could be observable from userspace, creating a side channel. Additionally, eBPF maps themselves are kernel allocations that may or may not be MTE-tagged depending on the kernel configuration—inconsistent tagging can create information leaks between eBPF programs and other kernel subsystems.