Intel Processor Trace (PT) is a hardware feature baked into modern Intel CPUs that captures fine-grained execution traces. For engineers working in performance tuning or security research, PT delivers a log of every taken branch, exception, and interrupt—without the heavy overhead of software-only tracers. The data is dense, the learning curve steep, and the tooling scattered across kernel drivers and userland utilities. This guide walks through setting up PT, collecting traces, decoding them, and applying the results to real analysis: finding hot paths, spotting side-channel leaks, and debugging corrupted control flow.
What Intel PT Actually Records
Intel PT encodes compressed packets into a memory region called the ToPA (Table of Physical Addresses) buffer. The hardware writes packet streams—TNT bits, target IPs for indirect branches, timing info like cycle counts. No memory values or register states, though. The decoder rebuilds the exact execution path by smashing together the binary’s static code layout with those dynamic trace packets.
On a Skylake or later core, PT can grab all threads of a process or even the whole system. You dial in the granularity: trace only user space, only kernel, or both. Address filters let you lock onto specific functions or modules so the trace size doesn’t explode when you’re chasing a single bug.
Packet Types That Matter for Analysis
- TNT: A compressed bitfield for conditional branches. One bit per branch, 0 for not-taken, 1 for taken.
- TIP: Target IP packet for indirect branches, returns, and far jumps. The decoder needs the binary to resolve targets.
- CYC: Cycle counter delta. This is how you spot latency spikes and do timing analysis.
- MODE: Updates the execution mode—16, 32, or 64-bit—when it flips.
- FUP: Flow Update Packet, fired on asynchronous events like interrupts or exceptions.
You can’t decode a raw trace without the exact binary that ran. The TIP packets only carry the low-order bits of the target address, so the decoder leans on the binary’s section layout to reconstruct the full RIP.
Setting Up Intel PT on Linux
Linux mainline has shipped perf with PT support since kernel 4.1. You need the kernel driver, the perf userland tool built with PT support, and the libipt library for decoding. Most distros give you perf that can capture traces without a fuss. Decoding usually means building libipt from Intel’s libipt repository.
Checking Hardware Support
grep intel_pt /proc/cpuinfo
If you get nothing back, your CPU lacks PT or it’s been fused off. Broadwell and newer normally have it, though some low-end SKUs skip it. Also poke around /sys/devices/intel_pt for the device node.
Capturing a Trace with Perf
The dead-simple capture targets a single command:
perf record -e intel_pt// -- my_program
That drops a perf.data file with raw PT packets and sideband data—mmap events, context switches, the works. For a long-running process, attach to a PID:
perf record -e intel_pt// -p 1234 -- sleep 10
To keep trace size under control, use address filters. Trace only my_function and whatever it calls:
perf record -e intel_pt// --filter 'filter my_function' -- my_program
Kernel tracing demands root and the --kernel flag. Careful with that—a full kernel trace can pump out gigabytes per second and make the box wobble if the buffer fills before userspace drains it.
Decoding Traces with Perf and libipt
perf script plus the PT decoder spits out human-readable output:
perf script --itrace=i0ns --ns
The --itrace options steer instruction-level decoding. i0 kills instruction output; ns adds nanosecond timestamps. For full disassembly, throw --itrace=i at it, but brace for a firehose of output. Usually you dump branch events with perf script and feed them to a visualizer or a quick custom script instead.
Using the libipt Tools Directly
ptdump and ptxed from the libipt suite give you lower-level access. Yank the raw trace out of perf.data with perf inject:
perf inject --itrace=be -o trace.dump
Then run ptdump trace.dump to see every packet. This is gold when you suspect the decoder lost sync—maybe the binary changed mid-trace, or a JIT region never got captured.
Performance Analysis with PT
Statistical profilers like perf record -e cycles sample on interrupts and give you aggregated hot spots. PT hands you exact control flow, so you can answer questions sampling can’t touch: how often a branch gets taken, the precise call sequence leading to a cache miss, or the latency of a specific indirect jump.
Finding Hot Paths with Loop Analysis
Decode the trace to basic blocks and count execution frequency. Tools like pt_filter from the processor-trace ecosystem can filter traces for specific IPs. Say you suspect a hot loop in crypto_core. Capture a trace filtered on that function, decode to blocks, and histogram the IPs. The block that shows up most is your loop body.
Layer on cycle packets (CYC) to measure per-iteration latency. The cycle packet gives wall-clock deltas between packets. Line up CYC packets with TNT/TIP, and you can annotate each branch with its elapsed cycles, flagging stalls from cache misses or branch mispredictions.
Detecting Spectre-Style Leaks
Speculative execution side channels leave fingerprints in PT traces. A Spectre v1 gadget trains the branch predictor, then accesses a secret-dependent array index. Even if the mispredicted path never architecturally commits, PT records the speculative TNT bits and TIPs if the CPU design exposes them. On some microarchitectures, PT packets for speculative paths get flushed before the buffer is written; on others, they’re suppressed only at decode time.
To test this, write a small program with a bounds-check bypass and capture a trace. Compare the decoded path against the architectural path (the one visible in perf script output). Mismatches tell you speculative execution is leaking into the trace. Early Spectre researchers used this trick to validate gadget behavior without custom microcode patches.
Security Analysis: Control-Flow Integrity and Exploit Debugging
Intel PT shines as a root-cause tool for corrupted control flow. When an exploit hijacks a return address or function pointer, the trace veers off the expected call graph. Compare the decoded trace against a known-good control-flow graph (CFG), and you can pinpoint the exact instruction where the corruption bit.
Building a CFG from the Binary
Static analysis with objdump or a disassembler like radare2 gives you the legal targets for each indirect branch. Write a script that reads the decoded PT trace and checks each TIP against the CFG. A mismatch is an anomaly. You’ve basically built a dynamic CFI verifier that runs on hardware traces.
ROP Chain Detection
Return-Oriented Programming chains show up as a sequence of return TIPs landing at ret gadgets instead of legitimate call sites. The trace decoder, running in return-compression mode, can expose these because hardware return stack buffer (RSB) mismatches generate extra packets. If the trace shows a return landing at a gadget address with no matching call before it, you’re looking at evidence of a ROP chain.
JIT Code and Dynamic Analysis
JIT engines like V8 or LuaJIT generate code at runtime. PT needs the exact binary for decoding, so for JIT regions you have to capture the generated code pages—either dump them at trace time or instrument the JIT to log code blobs. Tools like jitdump in Linux can embed JIT code into the perf.data sideband, letting the PT decoder resolve branches into JIT regions.
Automating Trace Collection for Fuzzing
Feedback-driven fuzzers like AFL chew on edge coverage. Intel PT can supply full-path coverage without binary instrumentation. Each target run under PT produces a trace; the fuzzer deduplicates traces by comparing the sequence of blocks executed. This catches bugs that depend on the exact path, not just the set of edges hit.
One practical setup: afl-fuzz spawns the target with perf record -e intel_pt// wrapping each execution. A small post-processing script decodes the trace, extracts the block tuple, and feeds it back as coverage. Overhead is higher than compile-time instrumentation, but for closed-source binaries or kernel fuzzing, it’s often the only route.
Common Pitfalls and Tuning
- Trace loss: If the ToPA buffer fills faster than userspace drains it, the hardware stops tracing and sets a flag. Always check
perf report --itrace=befor OVF (overflow) packets. Bump the buffer size or add filters. - Binary mismatch: Recompile, restart, or patch the binary between trace and decode, and the decoder loses sync. Archive the exact binary with the trace, no exceptions.
- Kernel tracing instability: Tracing kernel code can deadlock if the tracer itself takes an interrupt that generates PT packets. Use
--kernelsparingly and only on test systems. - Decoding performance: Full instruction decoding crawls. Stick to branch-only mode (
--itrace=b) for most analysis and flip to instruction mode only when you really need it.
FAQ
What CPUs support Intel PT?
Intel PT lands on Broadwell (5th gen Core) and later, but support wanders by SKU. Some Atom, Celeron, and Pentium models have it fused off. Check /proc/cpuinfo for the intel_pt flag or look for the intel_pt device in /sys/devices. Xeon Scalable and Core i5/i7/i9 almost always include it.
How much overhead does Intel PT add?
Hardware overhead usually sits at 1–5% for branch-only tracing with moderate filter settings. Full instruction tracing plus CYC packets can push 10–15%. The real bite is I/O: writing the trace buffer to disk can spike if the buffer is small and the trace rate high. Use RAM-backed buffers or perf record --snapshot mode for bursty workloads.
Can Intel PT trace kernel mode code?
Yeah, but you need root and the --kernel flag in perf record. The kernel must have PT support enabled (CONFIG_PERF_EVENTS_INTEL_PT=y). Tracing the whole kernel on a production box is asking for trouble; pin it to specific modules or functions with address filters.
How do I reduce the trace file size?
Slap on address filters to trace only the functions or libraries you care about. Try --filter 'filter my_library.so' or --filter 'start 0x400000/0x1000' to clamp address ranges. Kill cycle packets with --itrace=be if you don’t need timing data. For long runs, perf record --snapshot -e intel_pt// captures only the last N seconds before an event.
What is the difference between Intel PT and LBR?
Last Branch Record (LBR) stores only the last 4–32 branches in hardware registers. Handy for sampling short sequences, but it can’t reconstruct full execution paths. Intel PT records an arbitrarily long trace of every branch to memory, so you get complete path reconstruction. Use LBR for lightweight hotspot sampling; reach for PT when you need the exact path or timing between distant events.
