How to Set Up a Kernel Debugging Environment That Works

Kernel debugging gets talked about like it’s some arcane ritual reserved for the graybeards of system programming. The reality is less theatrical: it’s a skill built on a clean setup, repeatable tooling, and a stubborn refusal to let a VM crash ruin your afternoon. If you’ve been fighting with serial ports, mismatched symbols, or debugger protocols that seem to break between kernel versions, you’re hardly alone. This guide strips away the voodoo and gives you a working kernel debugging environment that holds up under real use—from driver development to rootkit analysis.

The approach here assumes you’re running Linux as your host—Debian or Arch derivatives are fine—and that you want to debug a Linux kernel inside a QEMU virtual machine. You can adapt the stack to physical machines or Windows kernel debugging (WinDbg with KDNET), but the principles stay the same: a reliable debug transport, correct debug symbols, and a kernel built with debugging in mind.

What You’re Actually Debugging

Before you touch a terminal, define your target. Are you debugging a loadable kernel module? A custom syscall? A hardware driver? A kernel exploit? The answer determines how lean or bloated your debug kernel should be. A minimal defconfig with CONFIG_DEBUG_INFO=y, CONFIG_GDB_SCRIPTS=y, and CONFIG_KGDB=y is usually enough. If you’re chasing memory corruption, turn on KASAN, LOCKDEP, and KMEMLEAK. Just remember: heavy sanitizers make the guest crawl, so toggle them only when you need to.

Build your kernel from source. Download a stable tarball from kernel.org—5.15 or 6.1 LTS are safe bets—and do a local build inside a dedicated directory. Make sure the .config includes:

• CONFIG_DEBUG_INFO_DWARF5=y (or DWARF4 if your gdb is older)
• CONFIG_GDB_SCRIPTS=y (so lx-commands work)
• CONFIG_KGDB=y and CONFIG_KGDB_SERIAL_CONSOLE=y for kgdboc
• CONFIG_FRAME_POINTER=y to keep stack traces sane

Don’t forget to set CONFIG_DEBUG_KERNEL=y as the umbrella option. A quick make olddefconfig after editing saves you from dependency hell.

Building the VM with QEMU

QEMU is the workhorse here. You’ll boot your compiled kernel, attach a debugger, and iterate fast. The guest rootfs can be a minimal Debian image built with debootstrap or a prebuilt busybox initramfs. I prefer a Debian sid image because it’s easy to drop modules into and test real-world binaries.

Create a raw disk image and install a base system:

qemu-img create -f raw debian.img 10G
sudo mount -o loop debian.img /mnt
sudo debootstrap sid /mnt
sudo chroot /mnt /bin/bash
# set root password, install gdb, ssh, etc.

Copy your compiled kernel’s bzImage and the initramfs (if you built one) into the host workspace. The QEMU invocation that makes debugging painless uses the -s flag, which is shorthand for -gdb tcp::1234:

qemu-system-x86_64 \
  -enable-kvm -cpu host -smp 4 -m 4G \
  -drive file=debian.img,format=raw \
  -kernel /path/to/bzImage \
  -append "root=/dev/sda1 console=ttyS0 nokaslr" \
  -nographic -s

Adding nokaslr is critical for predictable addresses. Without it, KASLR randomizes kernel base every boot, and your symbol offsets become useless unless you extract them at runtime.

Attaching GDB and Making It Useful

With the VM running, open GDB in the directory containing your kernel source and vmlinux:

gdb ./vmlinux
(gdb) target remote :1234

You’re connected, but raw GDB is clumsy for kernel work. Load the Linux helper scripts that live in scripts/gdb/linux/ from your kernel source tree. Add this to your ~/.gdbinit (after enabling auto-loading with set auto-load safe-path / if you’re feeling generous, or target the exact path for safety):

add-auto-load-safe-path /path/to/kernel/source/scripts/gdb/vmlinux-gdb.py
source /path/to/kernel/source/vmlinux-gdb.py

Now you get lx-ps to list processes, lx-dmesg to read the kernel log, and lx-symbols to load module symbols on-the-fly. When you insmod a driver inside the VM, run lx-symbols in GDB, and the symbols populate for that module’s address space.

Set a breakpoint on a common syscall to verify the chain works:

(gdb) hbreak sys_open
(gdb) continue

If the VM hits the breakpoint and hands control back to GDB, your setup is good. Hardware breakpoints (hbreak) are safer than software ones when the kernel is running, because they don’t modify executable memory that might be write-protected.

Serial Debugging with KGDB

Sometimes GDB over TCP isn’t practical—maybe you’re debugging a kernel panic that happens before the network is up, or you’re working on a machine with no virtualization. KGDB over a serial link is the fallback that always works, provided you have a physical or emulated serial port.

In your kernel config, enable CONFIG_KGDB_SERIAL_CONSOLE and set the console to ttyS0,115200. Add kgdboc=ttyS0,115200 to the kernel command line. On the QEMU side, expose the serial port as a pseudo-terminal:

qemu-system-x86_64 ... -serial pty

QEMU prints the pty device path. Connect GDB through that pty with:

(gdb) target remote /dev/pts/3
(gdb) set serial baud 115200

Trigger a break into the debugger from inside the VM with echo g > /proc/sysrq-trigger or by sending a SysRq-g from the host’s QEMU monitor. This method is slower than the TCP transport, but it survives early boot and network failures.

Symbol Problems and How to Fix Them

Mismatched symbols cause more wasted hours than any other debug issue. If your breakpoints never fire or GDB shows question marks for addresses, check these:

• vmlinux is from the exact same build as the running kernel. A rebuild even with identical .config can shift addresses if the toolchain changed.
• KASLR is disabled. The nokaslr boot flag is mandatory unless you extract the random base with lx-kaslr or parse /proc/kallsyms from inside the VM.
• Module addresses are stale. Run lx-symbols after every module load. For modules built out-of-tree, provide the .ko path explicitly.

When debugging a custom kernel module, build it with debug info:

make -C /lib/modules/$(uname -r)/build M=$(pwd) \
  EXTRA_CFLAGS="-g -O0" modules

Load the module in the VM, then in GDB:

(gdb) add-symbol-file /path/to/module.ko 0xffffffffc0000000

Replace the base address with the one from /sys/module/<name>/sections/.text inside the guest. The lx-symbols script does this automatically if you set up the module search path correctly.

Live Debugging Without Stopping the World

Attaching GDB stops the entire kernel. For many bugs, that’s fine. But if you’re debugging a timing-sensitive race condition or you need the system to keep handling interrupts while you inspect memory, you need a non-stop approach.

QEMU’s GDB stub supports vCont for non-stop mode, but kernel support is limited. A more practical method for live inspection is using tracepoints and ftrace in combination with GDB breakpoints. Enable the tracepoint you care about, let the system run, and only break in when a condition is met:

cd /sys/kernel/debug/tracing
echo 0 > tracing_on
echo function > current_tracer
echo "your_module:your_function" > set_ftrace_filter
echo 1 > tracing_on

Combine this with a GDB conditional breakpoint that triggers only when a specific process ID or variable state is seen:

(gdb) break do_sys_open if (strcmp(filename, "/etc/shadow") == 0)

That way you minimize the time the kernel is frozen.

Debugging Kernel Panics and Oops

When the kernel panics, the default behavior is to dump a call trace and hang. That’s useful for post-mortem analysis, but you often want to catch the panic in the debugger before the system locks up. Set CONFIG_PANIC_TIMEOUT=-1 to make the kernel wait indefinitely on panic. Then add panic=1 to the boot parameters for an automatic reboot after 1 second if you just need the trace, or omit it to stay in the panic state.

In GDB, you can set a breakpoint on panic() itself:

(gdb) break panic

When it hits, you have a live system in the exact state of failure. Inspect the stack, dump registers, and walk the call chain. The lx-dmesg command will show you the Oops message with the faulting instruction pointer.

Windows Kernel Debugging: The KDNET Shortcut

Not everyone lives in Linux. For Windows kernel debugging, WinDbg over KDNET is the modern standard. It’s faster than serial and works over Ethernet. Set up the target machine (or Hyper-V VM) with bcdedit /debug on and bcdedit /dbgsettings net hostip:192.168.1.100 port:50000 key:1.2.3.4. On the host, launch WinDbg, go to File > Kernel Debug, and enter the same port and key.

Microsoft’s public symbol server eliminates most symbol headaches. Point WinDbg to srv*https://msdl.microsoft.com/download/symbols and symbols resolve automatically. For third-party drivers, make sure the .pdb files are in the symbol path. KDNET debugging suffers from the same KASLR issue—use bcdedit /set {current} kstackpaging false and bcdedit /set nx AlwaysOff to simplify addresses during development.

Maintaining a Debug Kernel Over Time

A debugging environment rots when you ignore it. Kernel updates, toolchain upgrades, and shifting QEMU versions can break your flow. Keep a scripted provisioning process. A minimal Vagrantfile or a short setup.sh that clones your kernel config, builds the source, and launches QEMU saves you from manual recovery when something inevitably drifts.

Version-lock your debug tools. GDB 12 and 13 handle DWARF5 differently. If your kernel uses DWARF5, stay on GDB 13+. If you’re stuck on an older distribution, build GDB from source and keep it in /opt. Same for QEMU—version 7.0+ has fewer quirks with virtio and the GDB stub.

Finally, maintain a cheat sheet of the GDB commands you actually use. lx-dmesg, lx-ps, bt, info registers, x/10i $rip, and p *(struct task_struct*)my_task cover 90% of sessions. The rest is just patience.

FAQ

Why does GDB show “Cannot access memory at address 0x…” when I try to inspect a variable?

This usually means the address belongs to a module that isn’t loaded yet, or KASLR has shifted addresses and GDB’s symbol file doesn’t match. Run lx-symbols inside GDB to reload module symbols, and make sure you boot with nokaslr unless you manually extract the randomized base.

Can I debug a kernel on bare metal without a second machine?

Yes, but it’s riskier. KGDB over a USB debug cable or a real serial port works if your hardware supports it. You can also use a PCIe serial card and connect it to a USB-to-serial adapter on the same machine, but the loopback approach requires careful wiring. For most people, a VM is safer and more flexible.

My breakpoints on module code never trigger, even though the module is loaded. What’s wrong?

Module text is often page-protected, and software breakpoints that modify memory may fail silently. Use hardware breakpoints (hbreak) instead. Also verify the module’s .text address with /sys/module/<name>/sections/.text inside the guest and feed that to GDB with add-symbol-file if lx-symbols doesn’t pick it up automatically.

How do I debug early boot code that runs before the GDB stub is available?

On QEMU, you can set -S (capital S) to pause the VM at startup and attach GDB before the first instruction executes. In KGDB, add kgdbwait to the kernel command line to halt the kernel at the debug stub initialization. This lets you break into early platform setup and even architecture-specific entry points.