The Debugging Desert: Why Most Kernel Setups Fail Before the First Breakpoint

The Debugging Desert: Why Most Kernel Setups Fail Before the First Breakpoint

You’ve read the docs. Cloned the Linux source tree. Maybe you even compiled a kernel with CONFIG_DEBUG_INFO=y and launched QEMU exactly like the tutorial said. And then… nothing. KDB sits there frozen, KGDB won’t answer, or the symbols flat-out refuse to load. Welcome to kernel debugging—the graveyard where toolchains go to die and most online guides are aspirational fiction at best.

I’m Zel Mathis. I’ve built and wrecked more kernel debug environments than most folks have had kernel panics. This isn’t a cheery “hello world” module walkthrough. It’s the wiring diagram for a debug setup that survives a reboot, handles module loads, and gives you honest source-level breakpoints over a serial line or virtual socket—without some bloated IDE slapping a pretty coat of paint on the cracks. We’ll lean on GDB, QEMU, a custom-built kernel, and a mindset that expects every single component to stab you in the back.

Choose Your Weapon: The Hardware/Virtual Split

You can debug a kernel on bare metal. That means two machines, a null-modem cable, and a level of misery I wouldn’t wish on a first build. So we’re going virtual. QEMU is the obvious pick—it emulates a serial port like a champ and supports the KGDB stub baked into mainline. If you’re deep in ARM or RISC-V territory, QEMU still has your back; just swap the machine type and cross-compiler. The concepts carry straight over.

Don’t ignore VirtualBox or VMware if you already live in that world. Both can do virtual serial ports configured as host pipes or TCP sockets. The catch: VMware’s named pipe semantics don’t match QEMU’s Unix sockets, and GDB’s target remote protocol gets twitchy about timeouts. I’ll stick with QEMU here because it’s the lowest common denominator and the most likely to behave the same way on your machine.

Building the Kernel: Debug Symbols Are Non-Negotiable

Grab the latest stable or longterm release from kernel.org. Avoid your distro’s packaged debug kernel if you can—those often strip modules or compress vmlinux in ways that make GDB choke. Your .config needs at least this:

• CONFIG_DEBUG_INFO=y (or CONFIG_DEBUG_INFO_DWARF5=y for newer GCC/Clang)
• CONFIG_GDB_SCRIPTS=y (loads helper scripts for GDB)
• CONFIG_KGDB=y and CONFIG_KGDB_SERIAL_CONSOLE=y
• CONFIG_KGDB_KDB=y if you want the KDB frontend
• CONFIG_FRAME_POINTER=y (or CONFIG_UNWINDER_FRAME_POINTER on x86) for backtraces you can actually trust

Turn off CONFIG_RANDOMIZE_BASE (KASLR) for the debug kernel. Address space randomization makes breakpoint addresses a moving target. You can flip it back on later after you trust your symbol loading. Also think about setting CONFIG_DEBUG_KERNEL=y and CONFIG_DEBUG_DRIVER=y to crank up extra logging in whatever subsystem you’re hunting.

QEMU Command Line: The Hidden Knobs That Matter

Most quickstart guides hand you a QEMU invocation that boots fine but never debugs. The missing bits are the -gdb flag and a serial device that’s wired up right. Here’s a minimal but functional command for an x86_64 kernel:

qemu-system-x86_64 \
  -kernel arch/x86/boot/bzImage \
  -initrd initramfs.cpio.gz \
  -append "console=ttyS0 kgdboc=ttyS0,115200 nokaslr" \
  -serial tcp::1234,server,nowait \
  -gdb tcp::1235 \
  -m 512M \
  -nographic

Let’s pull that apart. kgdboc=ttyS0,115200 tells the kernel to use the first serial port as the KGDB I/O channel. The -serial tcp::1234,server,nowait exposes that serial console on TCP port 1234 so you can attach a terminal client (telnet localhost 1234) to watch boot messages and fiddle with KDB. The -gdb tcp::1235 creates a separate GDB stub on port 1235—this is where your debugger connects. Keeping console and GDB on different ports dodges those maddening character collisions that freeze your whole session.

If you’re on a headless server, -nographic shoves the virtual VGA onto the serial console, but you lose graphical output. For GUI debugging (say, watching a framebuffer driver), drop -nographic and add -vga std. The GDB stub still works fine.

Initramfs: Don’t Let Userland Block Your Breakpoints

A classic tripwire: you set an early breakpoint in start_kernel, but KGDB never wakes up because init hasn’t run the kgdbwait trigger. The kernel has a boot parameter kgdbwait that halts execution until a debugger attaches. Toss it into -append and the kernel stops cold after KGDB initializes, before spawning init. This is pure gold for early boot debugging.

For later stuff, you can trigger KGDB entry from sysfs: echo g > /proc/sysrq-trigger (if CONFIG_MAGIC_SYSRQ is on) or echo 1 > /sys/module/kgdboc/parameters/kgdboc_breakpoint. I reach for the SysRq method every time—it’s a hard interrupt that grabs all CPUs and drops into the stub no matter what userland is tangled up in.

GDB Configuration: Scripts and Source Mapping

Launch GDB from the kernel source directory so it can find the vmlinux file and those helper scripts:

gdb ./vmlinux \
  -ex "target remote :1235" \
  -ex "lx-symbols"

The lx-symbols command (courtesy of scripts/gdb/linux/symbols.py) teaches GDB how to load symbols for modules on the fly. Without it, stepping into a module function dumps you into raw assembly. Run lx-lsmod inside GDB to see what’s loaded and whether the symbols actually resolved.

If your source tree doesn’t match the running kernel exactly—maybe you’re debugging a distro kernel on a target machine—you can set set substitute-path /build/source /your/local/src to remap paths. But for a self-built kernel, just staying in the top-level source directory sidesteps the whole mess.

Breakpoints That Stick: Hardware vs Software

Kernel code can get patched at runtime (ftrace, alternatives, static keys), so a software breakpoint (hbreak vs break) might get overwritten or trigger a fault in read-only memory. Lean on hardware breakpoints when you can: hbreak function_name. You only get a handful (usually 4), but they survive code modifications and work in memory-mapped I/O regions. For module functions that haven’t loaded yet, GDB will whine; set a pending breakpoint with break function_name and answer “y” at the prompt.

Serial Port Shenanigans and Agent Proxies

If you’re debugging over a physical serial line (two machines tied together with USB-to-serial adapters), baud rate actually matters. KGDB runs at whatever the console is set to, but 115200 is the bare floor for tolerable stepping. You’ll also want agent-proxy (from the kgdb-agent-proxy project) to multiplex the serial line—KGDB and the console normally brawl over the same UART. Agent-proxy splits the traffic into two TCP ports: one for console, one for GDB. It’s a lightweight C program that sits on the debug host. The kernel docs mention it; almost nobody uses it until their first session hangs because a kernel log message shredded a GDB packet.

For QEMU, we don’t need agent-proxy thanks to those separate ports. For physical targets, it’s the difference between a working debug link and a brick that demands a hard reset every five minutes.

KDB: The Lightweight Alternative

Sometimes you don’t need the full GDB beast. KDB is a built-in kernel debugger that runs right on the target. It’s spartan but quick. Boot with kgdboc=ttyS0,115200 kgdbwait and hit SysRq-g to jump in. From there you can dump memory, set breakpoints (with bp), look at backtraces (bt), and poke at registers. It won’t do source-level stepping, but for crash analysis and live inspection of data structures, nothing beats it. I’ll often use KDB to corner a problem, then swap to GDB for the surgical strike.

Real-World Debugging Loop: A Worked Example

Let’s trace a common headache: a driver probe function fails and you want to know why. We’ll pretend it’s a custom PCI driver that won’t bind. Build the kernel with the driver baked in (not as a module, at first—modules add symbol-loading headaches). Boot with kgdbwait so the kernel stops before do_initcalls. Attach GDB:

(gdb) target remote :1235
(gdb) lx-symbols
(gdb) hbreak my_driver_probe
(gdb) continue

The kernel boots, runs initcalls, and slams into your breakpoint inside the probe function. Now you can step through PCI config space reads, inspect pci_dev fields, and see exactly which error path it tumbles down. If the driver is a module, you’d load it manually after boot and GDB will grab symbols when you call lx-symbols again (or you can stick it in ~/.gdbinit).

Usual failure: breakpoint never fires because the function got inlined or optimized into oblivion. Check objdump -t vmlinux | grep my_driver_probe. If it’s gone, try hbreak my_driver_probe.c:42 on a specific line. Compiler optimizations are the enemy; build with CONFIG_OPTIMIZE_FOR_DEBUGGING=y if your architecture supports it.

Netconsole and Early Panics

What if the kernel panics before the serial driver even wakes up? Netconsole can save your skin: netconsole=4444@10.0.2.15/eth0,6666@10.0.2.2/ fires log messages over UDP to a listening host before the console is initialized. Pair it with QEMU’s user-mode networking and a netcat listener: nc -u -l 6666. You won’t get interactive debugging, but you’ll see the backtrace. Then you can rebuild with earlyprintk=serial,ttyS0,115200 and take another run at it.

FAQ: The Kernel Debugging Obstacle Course

Why does GDB say “Remote ‘g’ packet reply is too long”?

This old chestnut usually means GDB connected to the wrong port (like the serial console instead of the GDB stub) or the target is spewing binary console data during the handshake. Double-check that your -gdb port is separate from -serial. For physical setups, agent-proxy is the fix.

How do I debug a kernel module that loads after boot?

Use lx-symbols in GDB after the module is loaded. You can automate it with a GDB user-defined function or just leave a pending breakpoint. When the module loads and symbols resolve, GDB will set the breakpoint automatically if you answered “y” to the pending prompt.

Can I use LLDB instead of GDB?

LLDB can connect to a KGDB stub over TCP, but it lacks those Linux-specific helper scripts (the lx-* commands). You’ll be stuck manually adding symbol files for modules and parsing memory yourself. Possible, but miserable. Stick with GDB for kernel work unless you’re poking at a macOS or FreeBSD kernel.

What’s the best way to debug a specific CPU core?

KGDB halts all cores by default, but you can use the cpu command inside KDB or info threads / thread in GDB to switch context. For per-CPU breakpoints, try conditional hardware breakpoints: hbreak function if $cpu == 2 (GDB’s $cpu convenience variable might need a script to populate).

The underground truth: kernel debugging is never a “set up once and forget” affair. It’s a sandbox that shifts under your feet with every compiler update, every new security mitigation, every QEMU version bump. The environment I’ve described here is a snapshot of what works today on a Linux 6.x kernel with GCC 13 and QEMU 8.x. Adapt it, break it, fix it—that’s the game. The only real failure is trusting a tutorial that hasn’t been tested since the Bush administration.