You have a binary. Could be some proprietary firmware blob, a sketchy executable someone dropped in your inbox, or a dusty piece of software where the source files bit the dust years ago. The code is gone, but you still need to figure out what this thing actually does. Reverse engineering binaries without source is a quiet obsession—equal parts tools, patience, and learning to see past the compiler’s obfuscation. This isn’t about a quick fix. It’s about reading the machine’s raw diary, page by page.
I’m Zel Mathis, and I’ve lost count of the nights hunched over disassembly listings in a hex editor, chasing calls through stripped ELF binaries, and tearing apart embedded controller dumps. What I’m going to walk through here is practical, hands-on, and assumes you’re already comfortable with low-level concepts. No hand-holding. Just the techniques that actually get results.
Setting Up Your Workspace
Before you even crack open the file, you need a clean, isolated box. Never—ever—run an unknown binary on your main machine. I spin up virtual machines with no network access by default, snapshotted so I can roll back in seconds. For static analysis, a standard Linux VM works fine, but the moment you shift into dynamic work, go air-gapped or use a dedicated sandbox. It only takes one mistake to regret it.
Your toolchain is everything. I keep three essentials loaded: a disassembler, a debugger, and a hex editor. For disassembly, Ghidra is my go-to—it’s free, handles a pile of architectures, and the decompiler is decent enough. Radare2 is lighter and scriptable if you lean that way. Debugging on Linux pretty much demands GDB with the PEDA extension; on Windows, I reach for x64dbg. And a hex editor—010 Editor or even plain old terminal-based xxd—saves you when you need to patch raw bytes directly.
First Pass: Triage and File Fingerprinting
Start by figuring out what you’re even holding. The file command on Linux spits out basics: file type, architecture, whether it’s stripped. A stripped binary means no symbol table—function names are dust, and you’ll be navigating by addresses alone. Run checksec or a similar tool to see security mitigations: PIE, stack canaries, NX. Those little flags tell you about the runtime environment and any potential exploit paths right off the bat.
Strings are your first real glimpse inside. I use strings -n 8 to pull sequences of eight or more printable characters. Scan for URLs, IPs, error messages, function names from linked libraries, or odd shell commands. I once found an entire command-and-control protocol just sitting in the strings of a malware sample. Don’t overlook the obvious—sometimes a developer left debug prints that pretty much map out the whole program logic.
Check for packers or obfuscation. Tools like UPX wrap binaries with a self-extracting stub; if you spot UPX in the strings, you can often unpack it with a simple upx -d. For custom packers, entropy analysis helps—sections with high entropy usually mean encrypted or compressed data. Binwalk is handy when you’re dealing with firmware images that mash together multiple filesystems or code chunks.
Import and Export Tables
The import table is basically a roadmap to what the binary can do. If it pulls in CreateFile, WriteFile, and registry functions, it’s doing file I/O. Network imports like socket and connect whisper “communication.” On Linux, you list dynamically linked libraries with readelf -d; on Windows, dumpbin /imports or Ghidra’s import viewer. Missing imports? The binary might be using direct system calls or resolving functions at runtime through GetProcAddress.
Exports tell you what the binary offers to the outside world. In a DLL or shared library, these are the entry points for external callers. For a standalone executable, the entry point usually points to main or _start. Finding the real main in a stripped binary can be a headache—look for the call to __libc_start_main on Linux, or trace the CRT initialization code on Windows.
Static Analysis: Mapping the Beast
Load the binary into your disassembler and let the auto-analysis churn. Ghidra will try to identify functions, cross-references, and data structures. Don’t trust it blindly—it’ll mislabel code as data and vice versa. Scroll through the listing and manually mark any obvious data regions. Keep an eye out for function prologues (like push rbp; mov rbp, rsp) to catch functions the tool missed.
Begin at the entry point and trace the initialization chain. What is it setting up? Any call to ptrace or IsDebuggerPresent screams anti-debugging. Memory allocation calls hint at dynamic data structures. Queries for system configuration reveal environmental dependencies. Build a rough call graph in your head first, focusing on the high-level flow, before you dive into the weeds.
For complex binaries, I work bottom-up on the bits that matter. Say I know the binary reads a file—I’ll find every call to file I/O functions, then trace backwards to see where the data originates and how it’s processed. This keeps me from drowning in initialization code that doesn’t matter for the goal at hand.
Data Structures and Algorithms
Compilers leave fingerprints. Loops turn into conditional jumps at the end of a block. Switch statements become jump tables. Object-oriented code shows up as vtables—arrays of function pointers. When you spot call [rax+offset], you’re probably staring at a virtual method dispatch. Common encryption algorithms give themselves away with constants: AES has its S-boxes, CRC32 has well-known polynomials. A quick web search for a mysterious constant often reveals the algorithm.
Strings and constants anchor your analysis. If you see a reference to "/etc/passwd", you know exactly what that function is up to. Cross-reference the string to find callers and piece together the context. Build a map of interesting spots and label them in your disassembler—it’s the only way to keep from losing your mind on bigger projects.
Dynamic Analysis: Watching It Run
Static analysis alone won’t answer everything. When you need to see runtime behavior, step through with a debugger. Set breakpoints on key API calls: file ops, network connections, process creation. On Linux, strace and ltrace are pure gold for a quick trace without firing up a full debugger—they show system calls and library calls, giving you a high-level trace of what the binary does.
Watch out for anti-debugging tricks. The binary might check for the TRACEME flag, hunt for breakpoint instructions (0xCC), or play timing games. Once you find these checks, you can patch them out with NOPs. For really stubborn samples, you might need a kernel-level debugger or hardware breakpoints that don’t touch memory.
Memory dumps during execution are where you’ll find unpacked code, decrypted strings, and data generated at runtime. Use your debugger’s dump feature or something like gcore to snapshot process memory at a precise moment. Then feed that dump back into your disassembler for static analysis. This is the classic move for slipping past packers—let the binary unpack itself, then freeze it in place and dissect the result.
Fuzzing and Input Analysis
If the binary chews on input—file format parsers, network protocols—fuzzing can unearth vulnerabilities and expose parser logic. Tools like AFL or libFuzzer mangle input and watch for crashes. Even without source, you can fuzz binaries using QEMU-based or hardware-assisted tracing. The crashes you get point straight to edge cases in the parsing code; reverse engineer those, and you’ll start understanding the expected format.
Common Pitfalls and How to Avoid Them
One classic blunder: trusting the disassembler’s output too much. Obfuscated binaries use tricks like overlapping instructions, where the disassembler picks the wrong alignment. Always double-check suspicious code by eyeballing the bytes manually. Another trap is ignoring the runtime environment—a binary might behave completely differently on Windows 10 versus Windows 7 thanks to API differences. Replicate the target environment as close as you can.
Don’t disappear down rabbit holes. It’s way too easy to burn hours reversing a function only to realize it’s just a standard library routine. Learn to spot library code by its structure and constants. Tools like FLIRT signatures in IDA, or Ghidra’s library identification, can automatically recognize known code. Save your brainpower for the custom logic that actually matters.
And document everything. I keep a running markdown file with findings, addresses of interesting functions, and half-baked theories. Reverse engineering is a loop—you’ll circle back to code as your understanding grows. Solid notes stop you from solving the same puzzle twice.
FAQ
What’s the first thing I should do with an unknown binary?
Run a strings analysis and file identification. Use file to get the architecture and type, then strings to pull readable text. That gives you an immediate feel for the binary’s purpose and whether it’s packed. Always do this in an isolated environment—never on your host machine.
How do I handle a binary that’s been stripped of symbols?
Lean on imports and strings to anchor your analysis. Find the entry point, trace the initialization, and look for standard library call patterns. Label functions based on what they do—if a function calls fopen and fread, name it something like read_file. Over time, you build up a custom symbol table from context.
When should I use static analysis versus dynamic analysis?
Start static to get a safe overview. Switch to dynamic when you need to see runtime behavior, unpack obfuscated code that decrypts itself, or watch system interactions. For packed malware, dynamic is often the only way to capture the unpacked payload. Ideally, you bounce between both as your picture sharpens.
What are some signs that a binary is malicious?
Look for anti-debugging calls, attempts to hide its process, direct system call invocation (skipping library wrappers), and network connections to shady domains. Encrypted strings, self-modifying code, and references to known exploit techniques are red flags. But remember, some legitimate software uses encryption and anti-debugging for DRM, so context is everything—consider the file’s source and reputation before jumping to conclusions.
