Binary protocols hold together the quiet guts of embedded systems—firmware updates, proprietary radio links, locked-down bootloaders. Vendors ship these things assuming the obscurity of a custom wire format and some tight parsing will keep attackers out. I’ve yet to meet a binary protocol parser that didn’t break when you pointed a well-tuned fuzzer at it. The problem? Off-the-shelf tools like AFL or libFuzzer are built for flat files and syscall interfaces. They stumble hard on stateful, length-delimited, checksummed protocols. This piece walks through building a custom mutation-based fuzzer that respects the protocol’s structure enough to get past the boring checks—and then twists the semantics to trigger the kind of bugs that make a vendor’s “secure by design” claim look hollow.
Why Generic Fuzzers Miss the Mark
Coverage-guided fuzzers treat input as a flat buffer. A binary protocol parser doesn’t. It reads a length byte, grabs exactly that many bytes, checks a type field, validates a CRC. If your fuzzer flips a bit in the length field without adjusting the payload, the packet gets rejected at a boundary check. The parser’s deeper logic—the part that actually handles the command—never runs. You’re stuck fuzzing the error path, not the state machine. Worse, many embedded parsers live on bare-metal or an RTOS where you can’t just recompile with instrumentation. You need a fuzzer that speaks the protocol’s language: one that generates mostly valid packets but occasionally slips in a length that wraps an integer, a type tag that doesn’t match the payload, or a checksum that’s correct for the wrong reasons.
Modeling the Protocol as a Mutable Tree
Start by capturing traffic with a logic analyzer—I use a Saleae Logic Pro 16 for SPI and UART sessions. Parse the raw bytes into a tree of typed fields: magic, length, sequence, command ID, payload, CRC. Each field gets a type and constraints. The fuzzer doesn’t mutate raw bytes; it mutates the tree. It can replace a length field with a value that’s valid but inconsistent with the payload size. It can splice a payload from a different session. It can flip a command ID to one that’s only valid after authentication. After mutating, the tree serializer recalculates the CRC so the packet passes the first line of defense. This gets you past the boring checks and into the parser’s actual logic, where the real bugs live.
State Awareness
Most binary protocols are stateful. You can’t just fire a single mutated packet and expect to hit deep code. The fuzzer needs a model of the protocol’s state machine. I implement this as a Python class that tracks the current state and prepends the necessary setup sequence before each test case. To fuzz a flash-write command on a microcontroller bootloader, the fuzzer first sends the unlock sequence, then the erase command, then the mutated write packet. It also deliberately violates state transitions—sending a write before the unlock—to see if the parser’s state tracking has holes. Those holes are where the best bugs hide: a buffer overflow that only triggers when a command arrives out of sequence, or a use-after-free when the parser resets state mid-handshake.
Coverage Without Recompilation
On x86-64, you can get basic block coverage without touching the target binary. I run proprietary firmware inside a minimal QEMU system emulation and parse the execution trace with a Python script that maps instruction pointers to basic blocks. It’s coarse—block-level, not edge-level—but it’s enough to guide mutations. For ARM64 targets, CoreSight ETM trace works if the SoC exposes it, though many cheap microcontrollers don’t. When trace hardware is absent, I fall back to a crash monitor: a GPIO toggle or UART heartbeat that the fuzzer watches. If the heartbeat stops, the target faulted, and the fuzzer logs the last packet sent.
Side Channels as Coverage Signals
Coverage alone is a weak signal. A parser can take an error path that’s functionally correct but leaks information through timing or cache state. I instrument the fuzzer to measure response latency with high precision—using the target’s hardware timer or an external FPGA-based cycle counter—and flag any input that causes a statistically significant deviation. On x86-64, I also monitor performance counters for cache misses and branch mispredictions via perf_event_open. A spike in L1 data cache misses on a specific input often means the parser accessed a lookup table with an attacker-controlled index. That’s a classic gadget for speculative execution attacks. The fuzzer can lock onto that input and start a focused mutation campaign to turn the side channel into a covert channel or a Spectre-style leak. I’ve used this exact technique to pull firmware encryption keys from a locked-down IoT hub by watching the timing of AES-GCM tag verification over a UART console.
Differential Fuzzing Across Parser Versions
Vendors update firmware to fix bugs and often introduce new parsers that behave slightly differently. A differential setup feeds the same mutated input to two firmware versions—say, the boot ROM and the main OS driver—and compares their responses. A mismatch points to a semantic gap you can exploit. The boot ROM might accept a malformed packet that the OS driver rejects, letting an attacker inject code during early boot before the OS hardens the interface. I run this in QEMU with two separate VM instances, synchronizing input delivery and comparing register dumps at the end of each packet processing routine. The fuzzer’s grammar model ensures both parsers get identical, well-formed packets, so any divergence is a genuine parser differential, not a framing error.
A Minimal Fuzzer in Python
Here’s a sketch of the core loop. It assumes you’ve built a ProtocolTree class that can serialize to bytes, recalculate CRCs, and apply mutations from a grammar. The coverage tracker is a placeholder for your specific instrumentation.
import random
from protocol_model import ProtocolTree
from coverage_tracker import CoverageTracker
def main():
tracker = CoverageTracker()
corpus = [ProtocolTree.from_capture("seed.pcap")]
total_cases = 0
while total_cases < 100000:
parent = random.choice(corpus)
child = parent.mutate()
packet = child.serialize()
send_packet(packet)
new_coverage = tracker.get_coverage()
if new_coverage or caused_crash():
corpus.append(child)
if caused_crash():
save_crash(packet, child)
total_cases += 1
The mutation engine is where the real work happens. It includes operators like flip_bit_in_field, swap_fields, duplicate_field, set_length_to_payload_size, and set_length_to_overflow. Each operator targets a specific protocol assumption. set_length_to_overflow sets a length field to a value that, when added to the header size, wraps around a 16-bit or 32-bit integer. This reliably triggers buffer overflows in parsers that use unchecked addition to calculate buffer offsets. I’ve built up a library of these operators from years of breaking real-world firmware, and each new target usually adds one or two more.
FAQ
Why not just use AFL with a custom mutator?
AFL’s custom mutator API lets you plug in a grammar-aware mutator, but the fuzzer still treats the input as a flat buffer. For stateful protocols, you need to control the sequence of packets, not just the content of one. You also need to reset the target to a known state between test cases, which AFL’s fork-server model doesn’t handle well for embedded targets. Building a dedicated fuzzer gives you full control over delivery, timing, and state management—things that matter when you’re hunting deep bugs.
How do you handle checksums without knowing the algorithm?
If the checksum algorithm is unknown, you can often infer it by analyzing the firmware binary. Look for tight loops that XOR or accumulate bytes, or for lookup tables used in CRC calculations. If firmware analysis isn’t possible, try a differential approach: send the same packet with a valid checksum and a mutated one, and see if the target’s behavior changes. Some parsers skip checksum verification entirely for certain command types—that’s a bug in itself. I’ve also had success using symbolic execution to solve for the checksum that produces a desired parser state.
What’s the most common bug you find?
Integer overflows in length calculations, by a wide margin. A parser reads a 16-bit length field, adds it to a fixed header size, and allocates a buffer without checking for wrap-around. Send a length of 0xFFFF, the addition wraps to a small value, and the subsequent memcpy of the payload overwrites the heap or stack. The second most common is an off-by-one in the length check, where the parser allows one byte more than the buffer can hold, leading to a single-byte overflow that corrupts a saved frame pointer or a size field in an adjacent heap chunk. Both are trivial to find with a custom fuzzer that understands the protocol’s length fields.
From Crash to Code Execution
Finding a crash is just the start. The real work is figuring out exploitability. For each crash, I triage using a minimal QEMU replay that logs the faulting instruction, register state, and recent branches. If the crash is a write to a controlled address, I map the target’s memory layout and look for useful overwrite targets: function pointers, return addresses, or data that influences a later authentication check. On ARM64, pointer authentication can complicate exploitation, but many embedded implementations leave PAC disabled for interrupt handlers or boot ROM code, creating a window for code reuse attacks. The fuzzer’s output becomes the starting point for a hand-crafted exploit, and the protocol knowledge gained during fuzzer development is what makes the exploit reliable.
Building a custom fuzzer is an investment, but if you work at the boundary between software and hardware, it’s the only way to systematically uncover the flaws vendors insist aren’t there. Next time a datasheet claims a protocol is “secure by design,” run your own fuzzer against it. The results will speak for themselves.
