Build a Kernel Debug Setup That Doesn’t Fall Apart

The Real Reason Your Debug Environment Keeps Crashing

Most kernel debugging guides read like a sanitized corporate memo. They assume you’ve got pristine lab hardware, a support contract, and never touch anything outside a VM. That’s not how it works in the trenches. When you’re pulling apart a driver at 2 a.m. or tracing a race condition on a production box that’s been up for 400 days, you need a setup that doesn’t fold under pressure. This is the guide I wish I’d had years ago, back when I was staring at a triple-fault on a headless server with nothing but a serial cable and a bad attitude.

We’re going to build a kernel debugging environment that actually works—one that handles physical hardware, weird UEFI quirks, and the inevitable moment when kdnet decides it doesn’t feel like binding today. I’ll cover Windows and Linux because reality doesn’t care about your OS religion. You’ll walk away with a reproducible configuration, not a pile of half-baked notes.

What You’re Up Against

Kernel debugging isn’t userland. Breakpoints can hang the entire machine. Timers don’t wait for your debugger to wake up. And the tooling? It’s often built by people who expect you to read their minds. On Windows, the debugger itself (WinDbg or the newer WinDbg Preview) is powerful but encrusted with legacy. On Linux, kgdb competes with a dozen other tracing frameworks, and half the documentation assumes a Raspberry Pi or QEMU. If you’re on bare metal—especially server-grade hardware—you’ll need to get comfortable with serial consoles and network debugging because USB debugging often isn’t an option.

Windows Kernel Debugging: KDNet and Serial That Actually Connect

Let’s start with Windows because it’s where the pain is most acute. The Microsoft docs will tell you to enable test signing, run bcdedit /debug on, and hope for the best. That’s step one, sure, but it’s not the whole story. The real trick is picking the right transport.

Network Debugging with KDNet

KDNet is the default for a reason. It works over Ethernet and doesn’t require a special cable. But it has a dark side: the debugger and target must be on the same subnet, and some NICs just refuse to cooperate. Here’s the no-nonsense process:

1. On the target machine, open an admin command prompt and run: bcdedit /debug on
2. Find your NIC’s bus parameters with: bcdedit /dbgsettings net hostip:<debugger_ip> port:<port> key:<key>. Use a key like 1.2.3.4 (yes, it’s a shared secret, not a password).
3. Reboot. If the target doesn’t show up in WinDbg’s “Connect to Remote” dialog, check that the NIC driver supports NDIS debugging. Broadcom and Intel usually do; Realtek is a gamble.

On the debugger side, launch WinDbg (not Preview—the classic one handles network debugging more reliably in my experience) and go to File > Kernel Debug > Net. Enter the port and key. If it hangs at “Waiting for reconnect…”, the target’s firewall or a VLAN is probably blocking the UDP traffic. Disable the firewall entirely on both machines during setup—you can lock it down later.

Serial: When the Network Betrays You

Serial debugging is the fallback. It’s slow, but it works on anything with a COM port or a USB-to-serial adapter. You’ll need a null-modem cable or a USB serial dongle that supports active debugging (FTDI chips are your friend; Prolific can be flaky). Configure it with:

bcdedit /dbgsettings serial debugport:1 baudrate:115200

Then connect WinDbg via File > Kernel Debug > COM. The baud rate must match exactly. If you get garbled output, drop to 9600. It’s not 1995—but sometimes it acts like it.

Linux Kernel Debugging: KGDB Without the Guesswork

Linux debugging splits into two camps: those who use QEMU and those who touch real iron. I’m covering bare metal because that’s where things get interesting. KGDB is the standard, but it’s not enabled by default in most distro kernels. You’ll be compiling your own, or at least tweaking the config.

Building a Kernel with KGDB Support

Grab a kernel source (I recommend the latest longterm from kernel.org) and make sure these are set in .config:

• CONFIG_KGDB=y
• CONFIG_KGDB_SERIAL_CONSOLE=y
• CONFIG_KGDB_KDB=y (if you want the KDB frontend)
• CONFIG_DEBUG_INFO=y (for symbols)
• CONFIG_FRAME_POINTER=y (helps with stack traces)

Compile and install. This isn’t a kernel build tutorial, but if you’re reading this, you probably know make -j$(nproc) && make modules_install && make install. The critical part is the command line. Add kgdboc=ttyS0,115200 kgdbwait to your bootloader configuration. kgdbwait tells the kernel to pause and wait for a debugger connection before booting fully—essential for early init issues.

Connecting GDB

On the debug host, you’ll use gdb with the target’s vmlinux file. Connect over serial with:

gdb ./vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyS0

If you’re using a USB serial adapter, the device might be /dev/ttyUSB0. Once connected, you can set breakpoints, step through code, and inspect memory. The experience is spartan compared to WinDbg’s GUI, but it’s rock solid. One trap: if the target’s console also uses the same serial port, you’ll get interference. Either use a second serial port or redirect console output to a virtual terminal. The kgdbcon module can help, but I usually just shut up the console with console=tty1 on the kernel command line.

Proxying Debug Connections: When You’re Miles from the Server

Here’s something the textbooks skip: remote debugging over an arbitrary network. Maybe the target is in a colo or a different building, and you don’t have a direct Ethernet cable. Both Windows and Linux can be proxied.

For Windows, use virtualKD or a custom serial-to-TCP bridge. I’ve had success with socat on a Linux box acting as an intermediary: socat TCP-LISTEN:5555 /dev/ttyS0,b115200. Then connect WinDbg to the TCP socket instead of a physical port. The trick is ensuring the serial parameters match on both ends of socat.

For Linux, agent-proxy is a lesser-known gem. It multiplexes the serial connection over TCP, allowing multiple GDB clients to attach. Run it on a device near the target, then connect remotely with target remote <proxy_ip>:<port>. This avoids the “one debugger at a time” limitation of raw serial.

Symbols and Source: Don’t Debug Blind

Without symbols, you’re reading assembly with no map. On Windows, the symbol server is a lifeline. Set _NT_SYMBOL_PATH=srv*c:\symbols*https://msdl.microsoft.com/download/symbols as an environment variable. For private builds, add your PDB directory. WinDbg will fetch what it needs automatically.

On Linux, the kernel’s debug info is in the vmlinux file you compiled. If you’re debugging a module, make sure it’s built with debug info (make CFLAGS_KERNEL=-g). Then load the module’s symbols in GDB with add-symbol-file <module.ko> <address>. Finding the address requires reading /proc/modules on the target—so keep a network console open for that. Pro tip: use CONFIG_DEBUG_INFO_BTF with modern kernels to include BPF type information, which tools like bpftrace can consume without full debug symbols.

Common Breakage and Dirty Fixes

Let’s be honest: most debugging sessions start with the debugger not connecting. Here are the fixes that actually work:

• Windows KDNet timeout: Disable VMQ and RSS on the NIC in the driver properties. These offload features can eat debug packets.
• Linux KGDB hangs after “waiting for connection”: The serial port might be claimed by a console getty. Kill the getty or add kgdbcon to the kernel command line to redirect console output to KGDB.
• Symbols mismatch: If the target kernel was updated but you’re still using old vmlinux, GDB will throw cryptic errors. Always copy the exact vmlinux from the target’s /boot.
• UEFI Secure Boot blocks debugging: On Windows, you must enable test signing or disable Secure Boot. On Linux, you’ll need to sign your kernel modules if Secure Boot is on—or just turn it off in the firmware.

FAQ

Why does my network debugger disconnect under heavy load?

KDNet uses UDP, which is connectionless and can drop packets when the NIC is saturated. The target’s NIC might prioritize regular traffic over debug packets. Use a dedicated management NIC if possible, or increase the debug packet priority by disabling interrupt moderation in the NIC driver settings.

Can I debug a kernel on a machine without a serial port?

Yes, but it’s messy. On Windows, you can use USB 3.0 debugging (xHCI debug capability) if your motherboard supports it—but it requires a specific USB cable and port. On Linux, you can use a USB-to-serial adapter on the target, but you’ll need to configure the USB serial driver as a console. Netconsole is another option: it sends kernel messages over UDP, but it’s output-only, not a full debugger.

How do I set up a two-machine debug environment when I only have one physical machine?

Virtualization is your answer. Use Hyper-V on Windows or KVM/QEMU on Linux to run the target as a VM. For Windows, enable the virtual COM port or KDNet over the virtual switch. For Linux, QEMU has built-in GDB stubs: just add -s -S to the QEMU command line, and you can connect GDB to localhost:1234. It’s not identical to bare metal—timing and hardware quirks won’t show up—but it’s enough for most driver development.

What’s the best way to automate kernel debugging in a CI pipeline?

Script the whole setup. Use expect or Python’s pexpect to control GDB connections, and wrap WinDbg in a PowerShell script with the -c option for commands. For Linux, you can compile a kernel with a predefined GDB script that sets breakpoints and continues. Then capture the output and parse it for known bug signatures. The key is making the target system reproducible—use a netboot image or a disk snapshot that resets after each run.

Wrapping Up: Keep It Simple, Keep It Dirty

The best debugging environment is the one that’s up when you need it. Don’t chase the latest tools if your serial cable works. Document your setup in a local text file, not a wiki—because when the network is down, you’ll thank yourself. And remember: kernel debugging is as much about mindset as tooling. Stay patient, question every assumption, and never trust a USB adapter unless you’ve tested it yourself.

This guide is a starting point. Adapt it to your hardware, your kernel version, and your tolerance for command-line chaos. The underground knows: debugging isn’t about pretty interfaces; it’s about seeing what the machine actually does.