A Kernel Debugging Setup That Doesn’t Fight You

The Real Problem with Kernel Debugging

Most kernel debugging guides start with a cheerful assumption: you’ve got a clean VM, a stock kernel, and infinite patience. That’s not how it works on the ground. When you’re staring at a stack trace from a custom driver or a kernel module that panics under load, you need a setup that doesn’t flake out after ten minutes. I’ve burned days chasing serial port glitches and symbol mismatches. This article documents the environment I actually use—a two-machine configuration with KGDB over serial, built on Debian, because that’s what survives late-night sessions.

Choosing the Hardware and Base System

You need two physical machines. VirtualBox and QEMU have come a long way, but when you’re chasing a race condition in a network driver or testing PCIe passthrough quirks, virtualization introduces its own noise. I use a pair of old ThinkPads—one T480 as the development host, one X250 as the target. The target machine should have a real serial port or a reliable USB-to-serial adapter. Avoid cheap PL2303 clones; the FTDI-based cables handle the sustained data stream without random disconnects. Both machines run Debian 12 minimal with the x86_64 architecture. On the target, I keep the filesystem under 10 GB so that crash dumps don’t eat the disk.

Serial Connection: The Lifeline

A null-modem serial cable is the physical bridge. If your target lacks a DB9 port, a USB serial adapter on each end works—just bind the target’s adapter to a known device name with a udev rule. The rule I use places the adapter at /dev/kgdb so there’s no confusion when other USB devices show up. Test the link with picocom at 115200 baud, 8N1, hardware flow control off. Both sides must echo characters before you trust it for kernel panics.

Building a Debuggable Kernel

Stock kernels ship with optimizations that inline functions and strip debug symbols. You need to compile your own. Grab the source from kernel.org—I stick to long-term releases for stability. The configuration step is where most people get lost. Start with your distribution’s config file from /boot, then run make menuconfig. The non-negotiable options:

• CONFIG_DEBUG_INFO=y – embeds DWARF debug information.
• CONFIG_GDB_SCRIPTS=y – generates helper scripts for GDB.
• CONFIG_KGDB=y – the kernel stub for remote debugging.
• CONFIG_KGDB_SERIAL_CONSOLE=y – ties KGDB to the serial line.
• CONFIG_FRAME_POINTER=y – preserves frame pointers, making backtraces reliable even without DWARF unwinding.

Disable CONFIG_DEBUG_INFO_REDUCED and CONFIG_RANDOMIZE_BASE (KASLR) on the target. KASLR complicates symbol resolution during early boot debugs. If you need KASLR later, you can add it back once the basic flow works. Build the kernel with make -j$(nproc), install the image and modules on the target, and update the bootloader.

Configuring the Bootloader for KGDB

On the target machine, edit the GRUB configuration. The kernel command line must include parameters that tell KGDB to wait for a remote debugger. My typical entry in /etc/default/grub looks like:

GRUB_CMDLINE_LINUX="quiet kgdboc=ttyS0,115200 kgdbwait"

kgdboc specifies the serial port and baud rate. kgdbwait makes the kernel pause early in boot until a debugger attaches. Without that flag, you’d need to trigger a sysrq-g later, which is unreliable if the system is already wedged. After modifying, run update-grub. Reboot the target while the host has picocom watching the serial line; you should see the kernel pause and print a message like “Waiting for connection from remote gdb.”

When the Serial Port Isn’t ttyS0

Modern laptops often lack a real serial port, and even USB adapters show up as ttyUSB0 instead of ttyS0. Adjust kgdboc accordingly. A bigger headache: make sure the kernel’s serial driver for your adapter loads before KGDB tries to claim it. That sometimes means building the driver (usbserial, ftdi_sio) directly into the kernel rather than as modules. A common failure mode: the kernel tries to bind KGDB before the USB stack initializes, and you get a silent hang. Figuring that out once cost me a whole weekend.

Setting Up GDB on the Development Host

On your host machine, you need a cross-aware GDB or a native GDB that matches the target architecture. Since both machines are x86_64, the system GDB works. I keep a copy of the uncompiled kernel source tree in /home/zel/linux-debug/. The compiled vmlinux binary with symbols lives there. Start GDB and load the symbol file:

gdb /home/zel/linux-debug/vmlinux

Before connecting, set the serial baud rate to match the target:

(gdb) set serial baud 115200

Then attach to the target over the serial device. On my host, the serial cable appears as /dev/ttyUSB0:

(gdb) target remote /dev/ttyUSB0

GDB freezes for a moment, then reports a remote connection. The target kernel remains halted. You can now set breakpoints on kernel functions, inspect memory, and step through code. To let the target continue, use continue in GDB. To break in again, send a sysrq-g from the target (if it’s still responsive) or use the GDB interrupt sequence (Ctrl+C).

Building a Reliable Workflow

Kernel panics rarely happen at convenient times. My workflow assumes the target will die randomly. I use a small script on the host that automates the GDB connection and logs everything:

#!/bin/bash
script -c "gdb -x gdb-commands /home/zel/linux-debug/vmlinux" kgdb-session.log

The gdb-commands file contains:

set serial baud 115200
target remote /dev/ttyUSB0

With this, I can reconnect after a power cycle without re-typing. For driver work, I keep the module source on the host, build it against the debug kernel, and load it manually on the target. When the module panics, the target halts and GDB shows the exact instruction. Symbols for the module load automatically if the .ko file is in the host’s module directory, but I often use add-symbol-file with the module’s .text address from /proc/modules.

Handling Early Boot Crashes

If the kernel dies before the serial console initializes, kgdbwait won’t help. In those cases, I enable CONFIG_EARLY_PRINTK and use a hardware debugger like a JTAG probe. That’s a deeper rabbit hole, but for most driver work, KGDB covers the boot phase once the serial driver is up. Another trick: compile the serial driver directly into the kernel (not as a module) and move its initialization to an earlier stage via the console_initcall macro. It’s a bit hacky, but it gets the job done.

Symbol and Source Alignment

Nothing wastes more time than GDB reporting “No source file named” when you know the code is right there. The kernel source path embedded in the debug symbols reflects the build directory. If you built the kernel in /home/zel/linux-debug/ and then moved the tree, GDB won’t find the sources. Use the directory command in GDB to point to the correct location. For out-of-tree modules, set the source path with set substitute-path. Keeping the build directory intact on the host is the simplest approach. I snapshot the entire tree after a successful build to avoid accidental modifications.

Network Debugging Alternative

Serial debugging is slow. KGDB over Ethernet (KGDBoE) uses UDP and can be much faster, but it requires a working network stack on the target. That’s a chicken-and-egg problem if you’re debugging network drivers. I use KGDBoE only for filesystem or memory management bugs where the network is stable. The kernel command line changes to kgdbwait kgdboe=@192.168.1.10/,@192.168.1.11/, specifying the target and host IPs. On the host, GDB connects with target remote udp:192.168.1.10:6443. The speed difference is noticeable when loading large symbol tables, but a serial link remains the fallback that always works.

Troubleshooting the Common Breaks

Over years of setting this up, certain failures repeat. If the target hangs on boot without printing the KGDB wait message, check that the serial driver isn’t a module. If GDB connects but breakpoints don’t fire, verify CONFIG_DEBUG_INFO and frame pointers. If you get “Remote ‘g’ packet reply is too long,” your GDB and kernel debug stub have a mismatch—rebuild both from the same source version. And if the serial line drops characters at 115200 baud, drop to 9600. It’s painful, but it works on the worst hardware.

FAQ

Do I really need two physical machines, or can I use a single host with a VM?

You can use a VM for many scenarios, and I’ve done it for filesystem debugging. But when you’re chasing hardware-specific bugs—PCIe issues, DMA errors, or timer interrupts—a VM hides the real behavior. The two-machine setup exposes the raw hardware. If you must virtualize, pass through a real serial port to the guest and test your setup there first.

Why does my kernel keep crashing before the debugger attaches?

This usually means the crash happens in code that runs before KGDB initializes. Check your kernel config for CONFIG_KGDB_LOW_LEVEL_TRAP if your architecture supports it. Another approach: add a busy loop in the early boot function that’s crashing, rebuild, and let the debugger catch the loop. Once attached, you can step through the problematic area.

What’s the fastest way to test if my serial link works for KGDB?

Boot the target with kgdboc but without kgdbwait. Once the system is up, echo g to /proc/sysrq-trigger on the target. The system should freeze, and the serial port should output KGDB traffic. On the host, connect GDB as described. If that works, add kgdbwait for boot-time debugging.

Is it possible to debug proprietary kernel modules this way?

Yes, but with limitations. You won’t have source-level debugging unless the vendor provides debug symbols. You can still disassemble, set breakpoints on exported symbols, and inspect memory. Use objdump on the module to find function offsets. The GDB command add-symbol-file with the module’s load address lets you map symbols if you have them.