The Real Problem with Kernel Debugging

Most kernel debugging guides start with a cheerful assumption: you’ve got a clean VM, a stock kernel, and infinite patience. That’s not how it works on the ground. When you’re staring at a stack trace from a custom driver or a kernel module that panics under load, you need a setup that doesn’t flake out after ten minutes. I’ve burned days chasing serial port glitches and symbol mismatches. This article documents the environment I actually use—a two-machine configuration with KGDB over serial, built on Debian, because that’s what survives late-night sessions.

Choosing the Hardware and Base System

You need two physical machines. VirtualBox and QEMU have come a long way, but when you’re chasing a race condition in a network driver or testing PCIe passthrough quirks, virtualization introduces its own noise. I use a pair of old ThinkPads—one T480 as the development host, one X250 as the target. The target machine should have a real serial port or a reliable USB-to-serial adapter. Avoid cheap PL2303 clones; the FTDI-based cables handle the sustained data stream without random disconnects. Both machines run Debian 12 minimal with the x86_64 architecture. On the target, I keep the filesystem under 10 GB so that crash dumps don’t eat the disk.

Serial Connection: The Lifeline

A null-modem serial cable is the physical bridge. If your target lacks a DB9 port, a USB serial adapter on each end works—just bind the target’s adapter to a known device name with a udev rule. The rule I use places the adapter at /dev/kgdb so there’s no confusion when other USB devices show up. Test the link with picocom at 115200 baud, 8N1, hardware flow control off. Both sides must echo characters before you trust it for kernel panics.

Building a Debuggable Kernel

Stock kernels ship with optimizations that inline functions and strip debug symbols. You need to compile your own. Grab the source from kernel.org—I stick to long-term releases for stability. The configuration step is where most people get lost. Start with your distribution’s config file from /boot, then run make menuconfig. The non-negotiable options:

• CONFIG_DEBUG_INFO=y – embeds DWARF debug information.
• CONFIG_GDB_SCRIPTS=y – generates helper scripts for GDB.
• CONFIG_KGDB=y – the kernel stub for remote debugging.
• CONFIG_KGDB_SERIAL_CONSOLE=y – ties KGDB to the serial line.
• CONFIG_FRAME_POINTER=y – preserves frame pointers, making backtraces reliable even without DWARF unwinding.

Disable CONFIG_DEBUG_INFO_REDUCED and CONFIG_RANDOMIZE_BASE (KASLR) on the target. KASLR complicates symbol resolution during early boot debugs. If you need KASLR later, you can add it back once the basic flow works. Build the kernel with make -j$(nproc), install the image and modules on the target, and update the bootloader.

Configuring the Bootloader for KGDB

On the target machine, edit the GRUB configuration. The kernel command line must include parameters that tell KGDB to wait for a remote debugger. My typical entry in /etc/default/grub looks like:

GRUB_CMDLINE_LINUX="quiet kgdboc=ttyS0,115200 kgdbwait"

kgdboc specifies the serial port and baud rate. kgdbwait makes the kernel pause early in boot until a debugger attaches. Without that flag, you’d need to trigger a sysrq-g later, which is unreliable if the system is already wedged. After modifying, run update-grub. Reboot the target while the host has picocom watching the serial line; you should see the kernel pause and print a message like “Waiting for connection from remote gdb.”

When the Serial Port Isn’t ttyS0

Modern laptops often lack a real serial port, and even USB adapters show up as ttyUSB0 instead of ttyS0. Adjust kgdboc accordingly. A bigger headache: make sure the kernel’s serial driver for your adapter loads before KGDB tries to claim it. That sometimes means building the driver (usbserial, ftdi_sio) directly into the kernel rather than as modules. A common failure mode: the kernel tries to bind KGDB before the USB stack initializes, and you get a silent hang. Figuring that out once cost me a whole weekend.

Setting Up GDB on the Development Host

On your host machine, you need a cross-aware GDB or a native GDB that matches the target architecture. Since both machines are x86_64, the system GDB works. I keep a copy of the uncompiled kernel source tree in /home/zel/linux-debug/. The compiled vmlinux binary with symbols lives there. Start GDB and load the symbol file:

gdb /home/zel/linux-debug/vmlinux

Before connecting, set the serial baud rate to match the target:

(gdb) set serial baud 115200

Then attach to the target over the serial device. On my host, the serial cable appears as /dev/ttyUSB0:

(gdb) target remote /dev/ttyUSB0

GDB freezes for a moment, then reports a remote connection. The target kernel remains halted. You can now set breakpoints on kernel functions, inspect memory, and step through code. To let the target continue, use continue in GDB. To break in again, send a sysrq-g from the target (if it’s still responsive) or use the GDB interrupt sequence (Ctrl+C).

Building a Reliable Workflow

Kernel panics rarely happen at convenient times. My workflow assumes the target will die randomly. I use a small script on the host that automates the GDB connection and logs everything:

#!/bin/bash
script -c "gdb -x gdb-commands /home/zel/linux-debug/vmlinux" kgdb-session.log

The gdb-commands file contains:

set serial baud 115200
target remote /dev/ttyUSB0

With this, I can reconnect after a power cycle without re-typing. For driver work, I keep the module source on the host, build it against the debug kernel, and load it manually on the target. When the module panics, the target halts and GDB shows the exact instruction. Symbols for the module load automatically if the .ko file is in the host’s module directory, but I often use add-symbol-file with the module’s .text address from /proc/modules.

Handling Early Boot Crashes

If the kernel dies before the serial console initializes, kgdbwait won’t help. In those cases, I enable CONFIG_EARLY_PRINTK and use a hardware debugger like a JTAG probe. That’s a deeper rabbit hole, but for most driver work, KGDB covers the boot phase once the serial driver is up. Another trick: compile the serial driver directly into the kernel (not as a module) and move its initialization to an earlier stage via the console_initcall macro. It’s a bit hacky, but it gets the job done.

Symbol and Source Alignment

Nothing wastes more time than GDB reporting “No source file named” when you know the code is right there. The kernel source path embedded in the debug symbols reflects the build directory. If you built the kernel in /home/zel/linux-debug/ and then moved the tree, GDB won’t find the sources. Use the directory command in GDB to point to the correct location. For out-of-tree modules, set the source path with set substitute-path. Keeping the build directory intact on the host is the simplest approach. I snapshot the entire tree after a successful build to avoid accidental modifications.

Network Debugging Alternative

Serial debugging is slow. KGDB over Ethernet (KGDBoE) uses UDP and can be much faster, but it requires a working network stack on the target. That’s a chicken-and-egg problem if you’re debugging network drivers. I use KGDBoE only for filesystem or memory management bugs where the network is stable. The kernel command line changes to kgdbwait kgdboe=@192.168.1.10/,@192.168.1.11/, specifying the target and host IPs. On the host, GDB connects with target remote udp:192.168.1.10:6443. The speed difference is noticeable when loading large symbol tables, but a serial link remains the fallback that always works.

Troubleshooting the Common Breaks

Over years of setting this up, certain failures repeat. If the target hangs on boot without printing the KGDB wait message, check that the serial driver isn’t a module. If GDB connects but breakpoints don’t fire, verify CONFIG_DEBUG_INFO and frame pointers. If you get “Remote ‘g’ packet reply is too long,” your GDB and kernel debug stub have a mismatch—rebuild both from the same source version. And if the serial line drops characters at 115200 baud, drop to 9600. It’s painful, but it works on the worst hardware.

FAQ

Do I really need two physical machines, or can I use a single host with a VM?

You can use a VM for many scenarios, and I’ve done it for filesystem debugging. But when you’re chasing hardware-specific bugs—PCIe issues, DMA errors, or timer interrupts—a VM hides the real behavior. The two-machine setup exposes the raw hardware. If you must virtualize, pass through a real serial port to the guest and test your setup there first.

Why does my kernel keep crashing before the debugger attaches?

This usually means the crash happens in code that runs before KGDB initializes. Check your kernel config for CONFIG_KGDB_LOW_LEVEL_TRAP if your architecture supports it. Another approach: add a busy loop in the early boot function that’s crashing, rebuild, and let the debugger catch the loop. Once attached, you can step through the problematic area.

What’s the fastest way to test if my serial link works for KGDB?

Boot the target with kgdboc but without kgdbwait. Once the system is up, echo g to /proc/sysrq-trigger on the target. The system should freeze, and the serial port should output KGDB traffic. On the host, connect GDB as described. If that works, add kgdbwait for boot-time debugging.

Is it possible to debug proprietary kernel modules this way?

Yes, but with limitations. You won’t have source-level debugging unless the vendor provides debug symbols. You can still disassemble, set breakpoints on exported symbols, and inspect memory. Use objdump on the module to find function offsets. The GDB command add-symbol-file with the module’s load address lets you map symbols if you have them.

The Debugging Desert: Why Most Kernel Setups Fail Before the First Breakpoint

You’ve read the docs. Cloned the Linux source tree. Maybe you even compiled a kernel with CONFIG_DEBUG_INFO=y and launched QEMU exactly like the tutorial said. And then… nothing. KDB sits there frozen, KGDB won’t answer, or the symbols flat-out refuse to load. Welcome to kernel debugging—the graveyard where toolchains go to die and most online guides are aspirational fiction at best.

I’m Zel Mathis. I’ve built and wrecked more kernel debug environments than most folks have had kernel panics. This isn’t a cheery “hello world” module walkthrough. It’s the wiring diagram for a debug setup that survives a reboot, handles module loads, and gives you honest source-level breakpoints over a serial line or virtual socket—without some bloated IDE slapping a pretty coat of paint on the cracks. We’ll lean on GDB, QEMU, a custom-built kernel, and a mindset that expects every single component to stab you in the back.

Choose Your Weapon: The Hardware/Virtual Split

You can debug a kernel on bare metal. That means two machines, a null-modem cable, and a level of misery I wouldn’t wish on a first build. So we’re going virtual. QEMU is the obvious pick—it emulates a serial port like a champ and supports the KGDB stub baked into mainline. If you’re deep in ARM or RISC-V territory, QEMU still has your back; just swap the machine type and cross-compiler. The concepts carry straight over.

Don’t ignore VirtualBox or VMware if you already live in that world. Both can do virtual serial ports configured as host pipes or TCP sockets. The catch: VMware’s named pipe semantics don’t match QEMU’s Unix sockets, and GDB’s target remote protocol gets twitchy about timeouts. I’ll stick with QEMU here because it’s the lowest common denominator and the most likely to behave the same way on your machine.

Building the Kernel: Debug Symbols Are Non-Negotiable

Grab the latest stable or longterm release from kernel.org. Avoid your distro’s packaged debug kernel if you can—those often strip modules or compress vmlinux in ways that make GDB choke. Your .config needs at least this:

• CONFIG_DEBUG_INFO=y (or CONFIG_DEBUG_INFO_DWARF5=y for newer GCC/Clang)
• CONFIG_GDB_SCRIPTS=y (loads helper scripts for GDB)
• CONFIG_KGDB=y and CONFIG_KGDB_SERIAL_CONSOLE=y
• CONFIG_KGDB_KDB=y if you want the KDB frontend
• CONFIG_FRAME_POINTER=y (or CONFIG_UNWINDER_FRAME_POINTER on x86) for backtraces you can actually trust

Turn off CONFIG_RANDOMIZE_BASE (KASLR) for the debug kernel. Address space randomization makes breakpoint addresses a moving target. You can flip it back on later after you trust your symbol loading. Also think about setting CONFIG_DEBUG_KERNEL=y and CONFIG_DEBUG_DRIVER=y to crank up extra logging in whatever subsystem you’re hunting.

QEMU Command Line: The Hidden Knobs That Matter

Most quickstart guides hand you a QEMU invocation that boots fine but never debugs. The missing bits are the -gdb flag and a serial device that’s wired up right. Here’s a minimal but functional command for an x86_64 kernel:

qemu-system-x86_64 \
  -kernel arch/x86/boot/bzImage \
  -initrd initramfs.cpio.gz \
  -append "console=ttyS0 kgdboc=ttyS0,115200 nokaslr" \
  -serial tcp::1234,server,nowait \
  -gdb tcp::1235 \
  -m 512M \
  -nographic

Let’s pull that apart. kgdboc=ttyS0,115200 tells the kernel to use the first serial port as the KGDB I/O channel. The -serial tcp::1234,server,nowait exposes that serial console on TCP port 1234 so you can attach a terminal client (telnet localhost 1234) to watch boot messages and fiddle with KDB. The -gdb tcp::1235 creates a separate GDB stub on port 1235—this is where your debugger connects. Keeping console and GDB on different ports dodges those maddening character collisions that freeze your whole session.

If you’re on a headless server, -nographic shoves the virtual VGA onto the serial console, but you lose graphical output. For GUI debugging (say, watching a framebuffer driver), drop -nographic and add -vga std. The GDB stub still works fine.

Initramfs: Don’t Let Userland Block Your Breakpoints

A classic tripwire: you set an early breakpoint in start_kernel, but KGDB never wakes up because init hasn’t run the kgdbwait trigger. The kernel has a boot parameter kgdbwait that halts execution until a debugger attaches. Toss it into -append and the kernel stops cold after KGDB initializes, before spawning init. This is pure gold for early boot debugging.

For later stuff, you can trigger KGDB entry from sysfs: echo g > /proc/sysrq-trigger (if CONFIG_MAGIC_SYSRQ is on) or echo 1 > /sys/module/kgdboc/parameters/kgdboc_breakpoint. I reach for the SysRq method every time—it’s a hard interrupt that grabs all CPUs and drops into the stub no matter what userland is tangled up in.

GDB Configuration: Scripts and Source Mapping

Launch GDB from the kernel source directory so it can find the vmlinux file and those helper scripts:

gdb ./vmlinux \
  -ex "target remote :1235" \
  -ex "lx-symbols"

The lx-symbols command (courtesy of scripts/gdb/linux/symbols.py) teaches GDB how to load symbols for modules on the fly. Without it, stepping into a module function dumps you into raw assembly. Run lx-lsmod inside GDB to see what’s loaded and whether the symbols actually resolved.

If your source tree doesn’t match the running kernel exactly—maybe you’re debugging a distro kernel on a target machine—you can set set substitute-path /build/source /your/local/src to remap paths. But for a self-built kernel, just staying in the top-level source directory sidesteps the whole mess.

Breakpoints That Stick: Hardware vs Software

Kernel code can get patched at runtime (ftrace, alternatives, static keys), so a software breakpoint (hbreak vs break) might get overwritten or trigger a fault in read-only memory. Lean on hardware breakpoints when you can: hbreak function_name. You only get a handful (usually 4), but they survive code modifications and work in memory-mapped I/O regions. For module functions that haven’t loaded yet, GDB will whine; set a pending breakpoint with break function_name and answer “y” at the prompt.

Serial Port Shenanigans and Agent Proxies

If you’re debugging over a physical serial line (two machines tied together with USB-to-serial adapters), baud rate actually matters. KGDB runs at whatever the console is set to, but 115200 is the bare floor for tolerable stepping. You’ll also want agent-proxy (from the kgdb-agent-proxy project) to multiplex the serial line—KGDB and the console normally brawl over the same UART. Agent-proxy splits the traffic into two TCP ports: one for console, one for GDB. It’s a lightweight C program that sits on the debug host. The kernel docs mention it; almost nobody uses it until their first session hangs because a kernel log message shredded a GDB packet.

For QEMU, we don’t need agent-proxy thanks to those separate ports. For physical targets, it’s the difference between a working debug link and a brick that demands a hard reset every five minutes.

KDB: The Lightweight Alternative

Sometimes you don’t need the full GDB beast. KDB is a built-in kernel debugger that runs right on the target. It’s spartan but quick. Boot with kgdboc=ttyS0,115200 kgdbwait and hit SysRq-g to jump in. From there you can dump memory, set breakpoints (with bp), look at backtraces (bt), and poke at registers. It won’t do source-level stepping, but for crash analysis and live inspection of data structures, nothing beats it. I’ll often use KDB to corner a problem, then swap to GDB for the surgical strike.

Real-World Debugging Loop: A Worked Example

Let’s trace a common headache: a driver probe function fails and you want to know why. We’ll pretend it’s a custom PCI driver that won’t bind. Build the kernel with the driver baked in (not as a module, at first—modules add symbol-loading headaches). Boot with kgdbwait so the kernel stops before do_initcalls. Attach GDB:

(gdb) target remote :1235
(gdb) lx-symbols
(gdb) hbreak my_driver_probe
(gdb) continue

The kernel boots, runs initcalls, and slams into your breakpoint inside the probe function. Now you can step through PCI config space reads, inspect pci_dev fields, and see exactly which error path it tumbles down. If the driver is a module, you’d load it manually after boot and GDB will grab symbols when you call lx-symbols again (or you can stick it in ~/.gdbinit).

Usual failure: breakpoint never fires because the function got inlined or optimized into oblivion. Check objdump -t vmlinux | grep my_driver_probe. If it’s gone, try hbreak my_driver_probe.c:42 on a specific line. Compiler optimizations are the enemy; build with CONFIG_OPTIMIZE_FOR_DEBUGGING=y if your architecture supports it.

Netconsole and Early Panics

What if the kernel panics before the serial driver even wakes up? Netconsole can save your skin: netconsole=4444@10.0.2.15/eth0,6666@10.0.2.2/ fires log messages over UDP to a listening host before the console is initialized. Pair it with QEMU’s user-mode networking and a netcat listener: nc -u -l 6666. You won’t get interactive debugging, but you’ll see the backtrace. Then you can rebuild with earlyprintk=serial,ttyS0,115200 and take another run at it.

FAQ: The Kernel Debugging Obstacle Course

Why does GDB say “Remote ‘g’ packet reply is too long”?

This old chestnut usually means GDB connected to the wrong port (like the serial console instead of the GDB stub) or the target is spewing binary console data during the handshake. Double-check that your -gdb port is separate from -serial. For physical setups, agent-proxy is the fix.

How do I debug a kernel module that loads after boot?

Use lx-symbols in GDB after the module is loaded. You can automate it with a GDB user-defined function or just leave a pending breakpoint. When the module loads and symbols resolve, GDB will set the breakpoint automatically if you answered “y” to the pending prompt.

Can I use LLDB instead of GDB?

LLDB can connect to a KGDB stub over TCP, but it lacks those Linux-specific helper scripts (the lx-* commands). You’ll be stuck manually adding symbol files for modules and parsing memory yourself. Possible, but miserable. Stick with GDB for kernel work unless you’re poking at a macOS or FreeBSD kernel.

What’s the best way to debug a specific CPU core?

KGDB halts all cores by default, but you can use the cpu command inside KDB or info threads / thread in GDB to switch context. For per-CPU breakpoints, try conditional hardware breakpoints: hbreak function if $cpu == 2 (GDB’s $cpu convenience variable might need a script to populate).

The underground truth: kernel debugging is never a “set up once and forget” affair. It’s a sandbox that shifts under your feet with every compiler update, every new security mitigation, every QEMU version bump. The environment I’ve described here is a snapshot of what works today on a Linux 6.x kernel with GCC 13 and QEMU 8.x. Adapt it, break it, fix it—that’s the game. The only real failure is trusting a tutorial that hasn’t been tested since the Bush administration.

When you’re staring at a disassembled binary, the first thing that slaps you in the face is stack management. You see push, call, ret, and sometimes a lea that makes no sense until you know the convention. Calling conventions are the unwritten handshake between functions—how arguments get passed, who cleans the stack, and which registers survive the call. Get this wrong, and you’re reading ghosts in the assembly. For reverse engineers, mastering these conventions is like learning the dialect of the machine you’re interrogating.

Why Calling Conventions Matter in Reverse Engineering

Imagine you’ve dumped a suspicious DLL and need to trace its exports. Without knowing the convention, you can’t tell if that mov eax, [esp+4] is grabbing the first argument or leftover stack trash. Conventions dictate the binary’s shape: how the compiler weaves function prologues and epilogues, how it aligns the stack, and how it deals with return values. For an underground analyst, this is your Rosetta Stone. It lets you reconstruct function signatures, spot hand-coded assembly obfuscation, and predict side effects that debuggers might hide.

The x86 world is messy because of its history. You’ve got 32-bit conventions born in the era of slow CPUs and small caches, and 64-bit ones that the AMD architects streamlined. Each one leaves a distinctive fingerprint on the binary. If you’re doing vulnerability research or unpacking malware, you’ll see them all: cdecl in ancient Windows code, stdcall in Win32 APIs, fastcall in driver code, and thiscall in C++ objects. On Linux, the System V AMD64 ABI rules 64-bit land, while the old i386 ABI hangs around in legacy binaries.

The 32-bit Battlefield: cdecl, stdcall, fastcall, and thiscall

Let’s start with the 32-bit conventions because they’re still everywhere in legacy Windows malware and old game hacks. Each one answers three questions: argument order (right-to-left or left-to-right?), stack cleanup (caller or callee?), and register usage (which regs are volatile?).

cdecl: The Default Chaos

cdecl is the standard for C programs on 32-bit x86. Arguments go on the stack right-to-left, the caller cleans the stack, and all registers except EBP and ESP are considered volatile. This is why you see add esp, 0Ch right after a call in disassembly—the caller is popping its own arguments. For variadic functions like printf, cdecl is the only game because the caller knows exactly how many args it pushed. In the wild, you’ll spot cdecl by the call followed by stack adjustment, and the frequent use of push instructions before the call.

stdcall: Windows’ Workhorse

stdcall flips the cleanup duty to the callee. Arguments still go right-to-left, but the function itself uses ret 10h (or similar) to pop arguments and return. This is the calling convention of the Win32 API. When you see ret 4, ret 8, or ret 0Ch, you’re looking at a stdcall function, and you can immediately deduce how many DWORD arguments it takes. For a reverse engineer, this is gold—no need to trace the caller to understand the function’s signature. Many malware droppers wrap API calls with stdcall stubs, so recognizing that ret with an immediate operand is a quick win.

fastcall: Speed Over Clarity

fastcall tries to avoid stack traffic by passing the first two arguments in ECX and EDX (on Windows). The rest go on the stack right-to-left, and the callee cleans. This is common in kernel-mode code and in some performance-sensitive user-mode libraries. The disassembly hallmark is seeing arguments in ECX and EDX without an initial push. It’s easy to mistake for a thiscall if you’re not paying attention—ECX can hold a this pointer or just the first integer arg. Context from surrounding code tells you which is which.

thiscall: C++ Under the Hood

thiscall is Microsoft’s convention for C++ member functions. The this pointer goes into ECX, and the rest of the arguments are pushed right-to-left. The callee cleans the stack if the function is non-variadic (usual case); otherwise, it’s caller-clean. In Visual Studio binaries, you’ll see ecx loaded with an object pointer before the call, and often the function prologue will store ECX into a stack slot or register for later use. When you’re reconstructing C++ vtables, thiscall is your bread and butter.

The 64-bit World: System V AMD64 vs. Microsoft x64

When you jump to 64-bit, the game changes radically. The stack is only used for arguments beyond the first few, and registers are precious. There are two major conventions: the System V AMD64 ABI used on Linux and macOS, and the Microsoft x64 convention on Windows. They look similar at a glance but have critical differences that will trip you up in cross-platform analysis.

System V AMD64 ABI: The Unix Way

On Linux and macOS, the first six integer or pointer arguments go into RDI, RSI, RDX, RCX, R8, and R9. Floating-point args use XMM0–XMM7. The stack is always 16-byte aligned at a call site, and the caller cleans the stack for any overflow arguments. Return values land in RAX (and RDX for 128-bit returns). The stack has a 128-byte red zone below RSP that signal handlers can use without adjustment—a quirk that sometimes confuses new reverse engineers when they see functions accessing negative RSP offsets without a sub.

In practice, you’ll see tight code with minimal stack usage. Functions often avoid using RBP as a frame pointer, relying on debug info instead. When you’re reversing a stripped ELF binary, you have to pay close attention to register initialization before calls to infer argument counts. The prologue is usually just sub rsp, N, and the epilogue is add rsp, N; ret. No ret N here because the callee doesn’t clean up args.

Microsoft x64: The Windows Way

Microsoft’s convention uses RCX, RDX, R8, and R9 for the first four arguments. Any additional args go on the stack right-to-left. The caller must allocate 32 bytes of shadow space on the stack, even if the function takes fewer than four args—this is home space for the callee to spill registers. The stack must be 16-byte aligned, and the caller cleans the stack. Volatile registers include RAX, RCX, RDX, R8–R11, and XMM0–XMM5. Non-volatile registers (RBX, RBP, RDI, RSI, RSP, R12–R15, XMM6–XMM15) must be preserved.

For reverse engineering, the shadow space is a dead giveaway. You’ll see sub rsp, 28h even for a function with two arguments—the extra 8 bytes are for alignment. When you see a function using RBX or RSI and saving them in the prologue, you know it’s preserving non-volatile regs. The Microsoft x64 convention is rigid, which makes decompilation easier: once you learn its signature, you can mechanically reconstruct parameters.

Spotting Conventions in the Wild: Practical Tricks

You’re not going to parse every function by hand; you need heuristics. Here’s what I do when I open a binary in IDA, Ghidra, or x64dbg.

Look at the ret instruction. If it’s ret N with N > 0 and you’re in 32-bit mode, it’s almost certainly stdcall or a callee-clean convention. The immediate value divided by 4 gives you the argument count. If it’s plain ret, you could be in cdecl or 64-bit land.

Check the stack pointer after calls. In cdecl, the caller adjusts ESP. You’ll see add esp, 0Ch or pop ecx sequences. In stdcall, no adjustment follows the call. In 64-bit Windows, the shadow space means you might see a larger sub rsp than needed, and no cleanup after the call except to restore the caller’s local space.

Trace register usage before calls. In fastcall or thiscall, ECX gets loaded with something meaningful. In 64-bit Linux, RDI and RSI are the first two args—look for string pointers or integer values. In 64-bit Windows, RCX is the first arg, and it often holds a this pointer if it’s dereferenced early in the function.

Beware of obfuscation. Some packers and protectors intentionally mix conventions or insert junk stack operations. For example, a function might use stdcall-style stack cleanup but be called with cdecl adjustments—this is a sign of hand-crafted assembly or a protector trying to break static analysis. When you see mismatched conventions, you’re probably in interesting territory.

FAQ

What’s the fastest way to identify a calling convention in a disassembler?

Start at the function’s return instruction. A ret N in 32-bit code is a strong signal for stdcall or a similar callee-clean convention. If there’s no immediate and the caller adjusts the stack after the call, you’re in cdecl. In 64-bit, the absence of ret N and the presence of shadow space in Windows point to the platform’s convention. Tools like IDA or Ghidra often auto-detect, but you should verify by checking a few call sites manually.

Can a single binary use multiple calling conventions?

Absolutely. A Windows binary might use stdcall for API calls, cdecl for internal C functions, fastcall for driver communication, and thiscall for C++ objects. It’s common to see them mixed. As a reverse engineer, you need to determine the convention per function. This is especially true in malware that statically links multiple libraries or uses obfuscation that switches conventions mid-stream.

How do variadic functions affect calling conventions?

Variadic functions require the caller to clean the stack because only the caller knows how many arguments were actually pushed. On 32-bit x86, this forces the use of cdecl (or a variant where the caller cleans). On 64-bit, the conventions already have the caller cleaning the stack, so variadic functions just follow the standard ABI. However, they often use AL to pass the number of vector registers used—spotting mov al, N before a call can indicate a variadic function like printf.

Why do some 32-bit functions use ret without an immediate but still clean their own stack?

This can be a sign of a custom convention or an obfuscation trick. For instance, a function might manually pop its arguments off the stack with pop ecx or add esp, N before a plain ret. This is common in code that wants to disguise argument counts or in hand-optimized assembly where the programmer wanted to reuse popped values. When you see this, you have to trace the entire function prologue and epilogue to understand the stack frame.

Mastering calling conventions isn’t glamorous, but it’s the foundation of everything you do in reverse engineering. Once you can read the stack and register dance without thinking, you start to see the programmer’s intent behind the opcodes. That’s where the real fun begins.

The February 2025 Release: What Actually Changed

Anthropic shipped Claude 3.7 Sonnet in February 2025, and the headline everyone grabbed was the extended thinking mode. If you’ve been in this space long enough, you’ve learned to be skeptical of mode announcements. They usually sound more impressive in the press release than they feel in the actual code. This one is different, though not in the way you might expect.

Extended thinking lets the model do multi-step reasoning before committing to output, and here’s the key detail that matters: you can configure token budgets up to 128K tokens for that reasoning process. This isn’t a binary feature flip. It’s a knob you turn, which means you’re making real architectural choices about how much “thinking time” your pipeline gets to spend on each request. That’s the engineer’s problem, and it’s exactly the kind of problem worth understanding before you deploy this to production.

Also worth noting is how fast the cloud side moved. AWS Bedrock had Claude 3.7 Sonnet available within weeks of launch. That’s fast enough to signal genuine enterprise demand, and it matters for anyone already living inside the AWS ecosystem. One less integration conversation with your platform team.

The Benchmark That Actually Matters for Your Job

Look at the SWE-bench Verified leaderboard and you’ll see Claude 3.7 Sonnet scored 70.3% on autonomous coding tasks at release, putting it ahead of GPT-4o and Gemini 2.0 Pro. That’s not a small thing. Autonomous coding evaluations measure whether a model can take a GitHub issue, write actual code, run tests, and iterate without human intervention. It’s close to what production pipelines actually need.

Here’s why this matters to your career specifically: this is the benchmark that correlates with real engineering work. Not token prediction accuracy on some synthetic dataset. It’s “can this system fix the bug or not.” When you’re evaluating models for internal tooling, code generation APIs, or autonomous agents, SWE-bench performance is the one you reference in meetings. It shifts the conversation from theoretical capability to applied capability, which is where your credibility comes from as a senior engineer.

That said, benchmarks are benchmarks. They’re the floor, not the ceiling. You still need to run your own evals against your actual use cases, your codebase patterns, your error types. But having a model that performs this well on the public benchmark gives you something concrete to anchor your internal testing against.

The Latency Tax You Need to Budget For

Here’s where the pragmatism kicks in. Extended thinking mode adds 15 to 40 seconds of latency per complex query, depending on how many tokens you budget for the reasoning phase. That’s the real cost structure, and you can’t wish it away with optimization.

For batch jobs, scheduled analysis, or internal tooling, that’s acceptable overhead. You run it overnight, you get better results, everyone wins. For user-facing endpoints, customer-facing APIs, or anything with a sub-second SLA, extended thinking becomes a tactical decision rather than a default. You enable it selectively. Maybe on the retry path when standard mode gives you a low-confidence answer. Maybe on weekend processing when traffic is lighter. Maybe not at all.

This is exactly the kind of decision that separates production-hardened architectures from demo implementations. Know your latency requirements first, then evaluate whether extended thinking fits your budget. Not the other way around.

The Cost Story That Actually Determines Adoption

Developers on the Anthropic forum reported 2 to 3x higher costs per task when extended thinking is enabled versus standard mode. This brought back the cost-versus-capability debate with real numbers attached. That’s the conversation that happens in expense review meetings, and frankly, it matters more than benchmark scores when you’re pitching this to finance.

The math is straightforward: extended thinking consumes more tokens during the reasoning phase, and those tokens cost money. You’re paying for transparency into the model’s reasoning process, which is genuinely valuable for debugging, auditing, and understanding failure modes. But it’s not free, and pretending it doesn’t factor into your decision-making is how you end up with a 10x budget overrun halfway through the quarter.

If you’re building something where accuracy drives revenue directly (trading systems, medical diagnostics, fraud detection), the 2 to 3x cost multiplier might be worth it. If marginal quality improvements don’t move the needle in your use case, it’s harder to justify. Run the economics. Use actual customer impact estimates. Make the decision intentionally.

How This Shapes Your Next Deployment Decision

Claude 3.7 Sonnet with extended thinking is good enough to be a serious consideration for autonomous agents, code generation pipelines, and complex reasoning tasks. It’s not universally better than everything else for everything. That’s not how production systems work. You’re making trade-offs on latency, cost, capability, and integration complexity every single time you pick a model.

Extended thinking mode represents a philosophical shift toward paying for reasoning as a first-class resource rather than hoping the model figures things out in one pass. It’s explicit, configurable, and measurable. Those are engineer-friendly properties. Check the Anthropic Claude 3.7 Sonnet release announcement for the exact API details, then run a pilot on your actual workloads with the token budget dialed in for your latency requirements.

The best models are the ones that fit your constraints, not the ones that win the most benchmarks. If extended thinking fits your pipeline, the numbers justify it, and your customers benefit from the quality improvement, deploy it. If not, standard mode is still extremely capable. Either way, you’re making an informed decision based on production reality rather than hype.

What’s your actual experience been with extended reasoning modes in production? Hit me up if you’ve got specific use cases that either worked or fell apart. There’s always more to learn from what actually ships.

The Scale Shift Nobody Expected (Except Everyone Who Was Paying Attention)

When Rust support officially merged into the Linux kernel in late 2022 with version 6.1, the tree contained roughly 13,000 lines of Rust code. I remember the mood on the kernel mailing list felt like cautious optimism mixed with academic curiosity. “Fine, let’s try this,” seemed to be the consensus. “But it better not break anything.” Fast forward to early 2026, and we are looking at over 600,000 lines of Rust across drivers, filesystem abstractions, and core subsystem bindings. That is not a gradual adoption curve. That is a phase transition.

What strikes me most about this trajectory is what it represents operationally. You do not go from 13,000 to 600,000 lines of code in a production-critical system without something fundamental shifting in how maintainers and contributors view the tradeoff between safety guarantees and implementation friction. The cynical interpretation is that Rust advocates finally won the political battle. The honest interpretation is messier and more interesting: the kernel community collectively realized that memory safety bugs were eating their lunch, and Rust was the least terrible solution available.

When Your GPU Driver Lives in Rust

The watershed moment came with NVIDIA’s Nova GPU driver initiative. Linus Torvalds confirmed in a December 2025 kernel mailing list post that Rust driver contributions have accelerated significantly, with Nova representing the highest-profile all-Rust driver effort to date. Let me be direct: getting a GPU driver merged into the mainline kernel is already a production-readiness marathon under the best circumstances. Getting one written entirely in Rust past skeptical reviewers required technical excellence that could not be hand-waved away.

What Nova proved was not just that you could write complex device drivers in Rust. It proved that the abstractions the kernel community had developed over three years were solid enough to handle something genuinely hard. I spent a weekend reading through the Nova submission threads, and the technical feedback was real. People were not rubber-stamping it because it was Rust. They were engaging with the actual design decisions. That is how you know a technology has stopped being a religious argument and started being a tool.

The Numbers Started Talking, and They Were Loud

Here is where the conversation shifted from philosophy to pragmatism. A 2025 study from the University of Waterloo analyzing 150 kernel CVEs from 2020-2024 found that 67 percent fell into memory safety categories that Rust’s ownership model structurally prevents. Seventy years of C, and it turns out that a significant majority of security vulnerabilities in the kernel trace back to problems that Rust simply does not allow you to create in the first place. You cannot have a use-after-free bug in Rust code. You cannot have a buffer overflow through naive indexing. The language does not let you.

This is not theoretical. This is not “Rust is safer if you follow best practices.” This is structural. The compiler refuses. For kernel developers accustomed to defensive programming as a way of life, that shift registers differently when you see it backed by empirical analysis of real CVEs.

Google’s Android team reported in a 2025 blog post that the proportion of new Android OS code written in memory-safe languages reached 77 percent, with Rust accounting for the majority of systems-level additions. More importantly, memory safety vulnerabilities in Android dropped to below 24 percent of total CVEs for the first time. You can argue about correlation and causation all you like, but when a platform that ships on billions of devices shows that metric, people listen. Check their Google Security Blog on memory safety in Android for the full breakdown. The data is public.

The Abstractions Tax Started Coming Due

Now here is where I have to put on my honest face and acknowledge that nothing in systems programming is free. In late 2025, veteran C maintainer Ted Ts’o posted a detailed technical critique arguing that Rust’s abstraction layers were creating hidden performance regressions in I/O paths that benchmarks were not capturing. And he was not wrong. He was not being a Luddite. He was being precise.

The argument goes like this: Rust’s abstractions force certain patterns. Those patterns are safe, yes. But safe is not free. Compiler optimizations have to work harder. The abstraction boundaries do not always align perfectly with hardware realities. When you are talking about code paths that execute billions of times per second across millions of servers, a small hidden cost becomes audible.

This critique mattered because it shifted the conversation away from “Rust versus C as an ideology” into “where do these abstraction costs actually manifest, and are they worth the safety gains?” That is a conversation you can have empirically, where you can measure things. The kernel mailing list threads that followed were dense, technical, and remarkably civil by historical standards. People were arguing about specific performance profiles, not about whether Rust belonged in the kernel. That is progress.

What This Actually Tells Us

Looking back from 2026 at the landscape that has emerged over two years of merge cycles, a few things become clear. First, the Rust transition in the kernel is real and accelerating, but it is not displacing C wholesale. It is expanding into new problem domains where the safety guarantees justify the abstraction costs. That is the sustainable path.

Second, the technical community has genuinely grappled with the tradeoffs instead of retreating into tribal signaling. Yes, there was drama on the mailing list. Kernel development drama is not new. But the drama has been the friction of genuinely difficult engineering questions, not ideology. When I read threads about Rust abstractions and I/O performance, I am reading the same careful reasoning I would see in threads about CPU cache behavior or network stack optimization. That normalization is the most important signal.

Third, and perhaps most importantly: when you have empirical data showing that 67 percent of kernel CVEs trace to problems Rust structurally prevents, and you have a platform like Android showing real-world security improvements, you have crossed a threshold where the burden of proof flips. The question is no longer “why Rust?” It becomes “why not Rust, in this particular context?” That shift is irreversible.

For anyone still following this evolution, the practical takeaway is to stop viewing Rust in the kernel as a binary choice. Check out the Linux kernel Rust documentation and understand where it is actually deployed and why. The drama you see on the mailing list is not a sign of failure. It is a sign of a community taking difficult engineering seriously.

What has your experience been with Rust in production systems? Have you hit those abstraction costs Ted Ts’o identified, or do your workloads sit in the “safety wins” category? The conversation is far from over, and I am genuinely curious what practitioners are seeing on the ground.

The Stability Inflection Point Nobody Really Saw Coming

In 2024, something genuinely quiet but consequential happened in the observability world. OpenTelemetry’s specification for logs, metrics, and traces all hit 1.0 stability simultaneously. Not beta. Not “mostly ready.” Actual, ship-it-to-production-with-confidence stable. For those of us who remember the pre-OTel era of lock-in nightmares and rip-and-replace migrations, this was the moment when the architectural foundation finally solidified under our feet.

By early 2026, the momentum shifted from niche adoption to mainstream inevitability. The CNCF project metrics tell the story more clearly than any analyst report ever could: OpenTelemetry became the second most active project in their entire portfolio by contributor count. Only Kubernetes outpaced it. That’s not hype. That’s an entire ecosystem deciding, collectively and voluntarily, that this is the floor for how observability gets instrumented going forward.

What made this different from other “standardization” efforts that promised the world and delivered PowerPoint? The specification matured without becoming a bureaucratic nightmare. The API stayed clean. The wire protocols actually worked across vendors. Most critically, the tools started shipping real implementations instead of treating OTel as something they’d “eventually support.”

How Vendors Got Forced to Compete on Merit Instead of Lock-in

Here’s where it gets genuinely interesting from a market dynamics perspective. Datadog, one of the most observability-forward companies on the planet, hit their earnings call in Q3 2025 with a number that should have made every observability engineer sit up straight: 34 percent of their new enterprise customers arrived already instrumented with OpenTelemetry. Not asking about it. Not evaluating it. Already shipping it. Their CEO, Olivier Pomel, explicitly acknowledged that this fundamentally changed how they approached customer onboarding and pricing conversations.

Think about what that actually means. For years, the classic sales motion was: “Use our agent, our SDK, our magic pixie dust. You’re now invested.” Now you walk in the door having already made the portability investment upfront. The vendor gets evaluated on what they do with your data after it arrives, not on whether they’ll hold it hostage behind proprietary instrumentation.

That’s not a minor shift. That’s the whole economic model getting inverted. Honeycomb’s 2025 State of Observability report surveyed a thousand engineers and found that 58 percent of them cited vendor portability as their primary motivation for adopting OpenTelemetry. Not cost reduction, though that came in at 44 percent. Not performance. Portability. The ability to walk away with your telemetry intact.

What’s fascinating is how fast the major cloud providers responded to this reality. AWS, Google Cloud, and Azure all announced native OpenTelemetry pipeline support in their managed observability services throughout 2025. That’s the equivalent of every major gas station deciding they’ll pump Shell, Chevron, or Exxon fuel into the same cars. You’re not locked into the station anymore. You’re just locked into needing gas.

Getting Started Without the Paralysis

If you’re reading this thinking “okay, but where do I actually begin,” let me cut through the analysis paralysis. The beauty of OpenTelemetry hitting stability in 2024 and gaining this critical mass adoption by 2026 is that the beginner path is finally, actually, straightforward.

Start with the OpenTelemetry project documentation. Not because it’s revolutionary prose, but because the documentation now assumes you’re someone who wants to actually ship something next week, not someone reading a spec for the first time. Pick your language. Pick one signal. Traces are usually the least intimidating starting point. Get your first service instrumented. Ship it.

The Collector piece is where the real leverage lives. By late 2025, the OpenTelemetry Collector was processing over 10 billion daily spans across known public deployments. That’s a four times increase from 2023. Not because it’s trendy. Because it genuinely works as the central nervous system for telemetry routing. You instrument your code once with OTel. The Collector handles the complexity of where it goes. Want to ship to multiple backends simultaneously for redundancy or cost optimization? The Collector handles it. Want to sample dynamically based on error rates? The Collector handles it.

The practical implication: you’re no longer making a binary choice between vendors. You’re making a choice about where your Collector instances live and what you do with the data once it’s there.

The Collector as Your Sanity Preservation Layer

One of the best architectural decisions you can make right now is treating the OpenTelemetry Collector as a critical infrastructure component, not an optional optimization. This isn’t about collecting badges for your resume. This is about buying yourself future flexibility that 3-AM-you will genuinely appreciate.

If you’ve been in this industry long enough, you’ve experienced at least one observability vendor surprise. A pricing model change. A feature sunset. A performance regression. Sometimes all three in the same quarter. Every one of those moments becomes significantly less destructive if your instrumentation is already decoupled from any single vendor’s pipeline.

The architectural pattern is simple: your code ships OTel signals to a local Collector instance. The Collector batches, samples, and routes. Your backend gets clean, efficient data in whatever format it natively understands. If you need to swap backends, you reconfigure the Collector. Your code? Unchanged. That’s not a small amount of insurance.

Check out the CNCF project metrics and devstats to see which exporters and receivers are getting the most attention. Active development is a decent proxy for “this won’t be unmaintained in six months.” The Collector ecosystem is reaching the point where you can likely find a working exporter for any backend you care about.

What This Actually Means for Your Next Architecture Decision

If you’re starting a new observability initiative in 2026, choosing proprietary instrumentation as your starting point is a legacy decision. Not because proprietary agents are inherently bad. They’re often incredibly well-engineered. But you’re paying for lock-in you don’t need anymore and getting less flexibility than the open standard offers.

The inflection has happened. The ecosystem has the velocity. The documentation has matured. The vendor support is real. The cloud providers are shipping native support. The market has clearly moved. This isn’t about evangelism or ideology anymore. It’s just pragmatism.

Start small. One service. One signal. Get comfortable with how OTel actually works on your infrastructure instead of how it works in theory. The Collector handles the complexity of connecting multiple backends, so you don’t need to solve everything on day one. You can iterate toward your ideal architecture without rip-and-replace migrations.

What observability decisions are you wrestling with right now? Have you encountered situations where vendor lock-in created friction, or are you just starting to think about how OpenTelemetry fits into your infrastructure? Drop a note in the comments or grab me on whatever platform you hang out in. The most interesting observability problems rarely have clean answers, and I’m genuinely curious what’s on your team’s radar.