Reverse engineering can help identify flaws and errors before they become business critical software errors. Reverse engineering techniques can be used to detect the presence or absence of software elements of concern.
Acknowledgments
I would like to thank my family for their patience and support: my grandparents, Yosef and Pnina Vertzberger, my parents, Avraham and Nava Eilam-Amzallag, and my brother, Yaron Eilam. I would like to acknowledge my good friends, Adar Cohen and Ori Weitz for their friendship and support.
Contents
Applied Reversing 139
Auditing Program Binaries 243
Cracking 307
This book was written after years of working on software development projects that repeatedly required the reverse engineering of third-party code for a variety of reasons. For those of you who have not yet been exposed to the concept of software reverse engineering, a small introduction is in order.
Introduction
It occurred to me at the time that this was a remarkably powerful skill because it meant that I could quite easily get answers to any questions I had about software I was working with, even when I didn't have access to the relevant documentation or to the source code of the program in question. It is not, and it is the very purpose of this book, to teach and demonstrate reverse engineering techniques that can be applied daily to solve a wide variety of problems.
Reverse Engineering and Low-Level Software
Techniques to prevent people from reverse engineering code and a sober attempt to evaluate their effectiveness. The general principles behind modern malicious programs and how reverse engineering is used to study and neutralize such programs.
How This Book Is Organized
Reverse engineering on the .NET platform, including an introduction to the .NET development platform and its assembly language: MSIL. The legal aspects of reverse engineering: When is it legal and when is it not.
Piracy and Copy Protection: This part focuses on the reverse engineering of certain types of security-related code such as copy protec-
Many of these tools are used in the reverse engineering sessions demonstrated in this book. This chapter also demonstrates a real reverse engineering session on a real-world malicious program, which is exactly what malware researchers often have to go through to study malicious programs, evaluate the risks they pose, and learn how to remove them.
Who Should Read this Book
Appendices: The book has three appendices that serve as a powerful reference when trying to decipher programs written in Intel IA-32 assembly language. Far beyond a simple assembly language reference guide, these appendices describe the common code fragments and compiler idioms emitted by popular compilers in response to typical code sequences, and how to identify and decipher them.
Tools and Platforms
The primary operating system used in this book is Microsoft Windows, and for good reason. Windows is the most popular environment for reverse engineering, and not just because it is the most popular operating system in general.
What’s on the Web Site
Its lovely open source alternative Linux, for example, is far less relevant from the reverse point of view, precisely because the operating system and most of the software that runs on top of it is open source. There is no point in flipping open source products - just read the source code, or better yet, ask the original developer for answers.
Where to Go from Here?
Still, in some cases, large-scale reverse engineering projects can greatly benefit from some of these expensive products.
Reversing 101
3 This chapter provides some background information on reverse engineering and the various topics discussed in this book. Then there is a short introduction to the reverse-engineering process and the tools of the profession.
What Is Reverse Engineering?
We start by defining reverse engineering and its different types of applications in software, and then move on to demonstrate the relationship between low-level software and reverse engineering. Finally, there is a discussion of the legal aspects of reverse engineering, attempting to classify when reverse engineering is legal and when it is not.
Foundations
Traditionally, reverse engineering has been about taking prepackaged products and physically taking them apart to reveal their design secrets. In many industries, reverse engineering involves examining the product under a microscope or taking it apart and figuring out what each part does.
Software Reverse Engineering: Reversing
Not so long ago, reverse engineering was actually a fairly popular hobby that was practiced by a large number of people (although it wasn't called reverse engineering). Remember how, in the early days of modern electronics, many people were so fascinated by modern devices like radios and television sets that it became common to take them apart and see what was going on inside them.
Reversing Applications
In general, there are two categories of reverse engineering applications: security related and software development related.
Security-Related Reversing
Since the algorithm is a secret, flipping can be seen as a way to break the algorithm. Look for a bug in the algorithm that can be used to extract the key or the original message.
Reversing in Software Development
With open source software, having open access to the program's source code means that some vulnerabilities and security holes can be discovered very early, often before malicious programs can take advantage of them. Those who lack the purchasing power to convince a large corporation to give them access to the product's source code must either take the company's word that the product is well-built, or resort to returns.
Low-Level Software
To become a competent reverse engineer, you need to develop a solid understanding of low-level software and low-level programming. In order to successfully extract information during a reversal session, reversers must understand the various aspects of low-level software.
Assembly Language
That's a misconception: machine code and assembly language are two different representations of the same thing. This is a relatively simple operation because the textual assembly language is simply another representation of the object code.
Compilers
Since assembly language is a platform-specific matter, we need to choose a specific platform to focus on while studying the language and practicing turning. Much of this book is devoted to the art of deciphering machine-generated assembly language.
Virtual Machines and Bytecodes
What this means is that the resulting object code, when translated into assembly language by a disassembler, is essentially a machine-generated assembly language program. This is an important point because this code is not always easy to understand, even when compared to a human-made assembly language program - machines think differently than humans.
Operating Systems
Customers must obtain a virtual machine that is compatible with both their specific bytecode language and their specific platform. Decommissioning programs written in bytecode-based languages is a completely different process that is often much simpler compared to the process of reversing the source executables.
The Reversing Process
This book focuses primarily on the reverse engineering of source executable programs created by source machine code compilers.
System-Level Reversing
Code-Level Reversing
The Tools
Let's take a quick look at the different types of tools you'll be dealing with.
System-Monitoring Tools
Disassemblers
Debuggers
Reversers can step through the disassembled code and essentially "see" the CPU as it executes the program one instruction at a time. As with source-level debugging performed by software developers, reversers can install breakpoints at points of interest in the disassembled code and then examine the state of the program.
Decompilers
Tracing means that the program executes one line of code and then freezes, allowing the user to observe or even change the program's state. Because developers have access to the source code of their program, debuggers present the program in source code form, allowing developers to set breakpoints and trace through source lines, even though the debugger is actually working with the machine code underneath.
Is Reversing Legal?
Interoperability
Competition
Finally, the more appropriate (and ethical) type of reverse engineering in a competitive product situation is one where the reverse engineering is only applied to small parts of the product and is only used to gather information, not code. In these cases, most of the product is developed independently without any use of reverse engineering, and only the most complex and unique areas of the competing product are reverse engineered and reimplemented in the new product.
Copyright Law
Trade Secrets and Patents
The Digital Millenium Copyright Act
Regulation: DRM technologies can be bypassed in order to regulate material accessible to minors online. So, a theoretical product that allows unsupervised and unsupervised Internet browsing can be repurposed for the purpose of controlling the use of the Internet by minors.
DMCA Cases
License Agreement Considerations
Code Samples & Tools
Conclusion
25 This chapter provides an introduction to low-level software, a critical aspect of reverse engineering. We then proceed to introduce low-level software and show how the fundamental concepts of high-level software map to the low-level domain.
High-Level Perspectives
Program Structure
Static libraries are added to a program during its build, and they become an integral part of the program's binaries. Clients are other components in the program that request the object's services, but are not interested in any of its implementation details.
Data Management
Developers are usually only made aware of the simplified data flow described by the high-level language. The most significant disadvantage of arrays is the difficulty of adding and removing elements in the middle of the list.
Control Flow
The difference lies in the logical arrangement of the items: in a tree structure, items are arranged hierarchically, which greatly simplifies the search for an item. Like the root item, each item in the lower levels of the hierarchy also has two links to lower nodes (unless it is the lowest item in the hierarchy).
High-Level Languages
On the other hand, C supports the common high-level functions found in other higher-level languages. Because Java programs run on a virtual machine (VM), the process of reversing a Java program is quite different from reversing programs written in compiler-based languages such as C and C++.
Low-Level Perspectives
The process of obfuscating .NET programs and the effects of the various obfuscation tools are discussed in Chapter 12.
Low-Level Data Management
Registers are used to store the most immediate data, while stacks are used to store slightly longer-term data. A stack is used to store parameters and an instruction pointer for each procedure call.
Assembly Language 101
One of the problems is that most high-level conditional statements are simply too long for low-level languages, such as assembly language, and so they are broken into sequences of operations. The key to understanding these sequences, the correlation between them, and the high-level statements from which they arise is understanding the low-level control-flow constructs and how they can be used to represent high-level control-flow instructions.
Registers
EBP Can be used as a generic register, but is mainly used as a stack base pointer. A stack frame can be defined as the stack area of the current function, which is located between the stack pointer (ESP) and the base pointer (EBP).
Flags
There are arithmetic instructions that test operands for certain conditions and set processor flags based on their values. Then there are instructions that read these flags and perform various operations depending on the values loaded into the flags.
Instruction Format
Suppose you have a variable called bSuccess in the high-level language, and you have code that tests if it is false. It is generally not possible to test the value of a variable and act on that value in a single statement.
Basic Instructions
Memory addresses are denoted using parentheses, so that. ebx) means "address pointed to by EBX.". For example, if the result of the subtraction is zero, the Zero Flag (ZF) is set, indicating that the two operands are equal.
A Primer on Compilers and Compilation
This means that the EDI value will be multiplied by the EBX value and the result will be written back to EDI. The bottom line is that understanding the processes that compilers go through and how they "perceive" code will help in the ultimate deciphering of their output.
Defining a Compiler
Sometimes it differs so much from the original code structure of the program that it becomes difficult to determine the original intentions of the software developer. While it's true that the following sections could be considered optional, I still recommend reviewing them at some point if you're not familiar with basic translation concepts.
Compiler Architecture
Optimization tools often change the structure of the code to make it more efficient while preserving its meaning. A good optimizer identifies such statements and moves them to an area outside the loop to improve code efficiency.
Specific Compilers
Intel has, not surprisingly, focused on making this compiler generate highly optimized IA-32 code that takes into account the specifics of the Intel NetBurst architecture (and other Intel architectures). The Intel compiler also supports the advanced SSE, SSE2, and SSE3 extensions offered in modern IA-32 processors.
Execution Environments
Software Execution Environments (Virtual Machines)
Another prominent feature is runtime type safety: because virtual machines have accurate data type information in the program being executed, they can verify that type safety is maintained throughout the program. Unlike conventional binary programs, in which each instruction is decoded and executed by hardware, virtual machines perform their own decoding of program binaries.
Hardware Execution Environments in Modern Processors
The Intel NetBurst microarchitecture is the current execution environment for many of Intel's modern IA-32 processors. To improve the processor's predictive capabilities, IA-32 processors use a branch trace buffer (BTB), which records the results of the most recent branch instructions processed.
Windows Fundamentals
If you feel completely comfortable with operating systems in general and the Windows architecture in particular, feel free to skip this chapter. If you feel you need additional information on certain topics discussed in this chapter, I have listed some additional resources at the end of this chapter.
Components and Basic Architecture
Brief History
Features
Portable Unlike the original Windows product, Windows NT was written in a combination of C and C++, which means it can be recompiled to run on different processor platforms. Multiprocessing The Windows NT kernel is multiprocessing, which means it is better suited for high-performance computing environments such as large data center servers and other CPU-intensive applications.
Supported Hardware
The Windows NT File System (NTFS) also supports an ACL for each individual file and supports the encryption of individual files or entire volumes. Compatible Windows NT is fairly compatible with older applications and can also run 16-bit Windows applications and some DOS applications.
Memory Management
Each object in the system has an associated access control list (ACL) that determines which users are allowed to manipulate it. Old applications run in a special isolated virtual machine where they cannot compromise the rest of the system.
Virtual Memory and Paging
Later, when the flushed pages are accessed, the processor will generate a page fault (because their page table entries are invalid), and the system will know that they have been paged out. In reality, this only works when applications are not actively using more memory than is physically available, because in such cases the system will have to move data back and forth between physical memory and the hard drive.
Working Sets
Page faults have a bad reputation because any program or system crash is usually accompanied by a message informing us of an unhandled page fault. In most cases, the system handles such page faults as part of its normal operations.
Kernel Memory and User Memory
A good example of a legitimate page error is when a page has been paged to the paging file and is being opened by a program. Because the page's page table entry is invalid, the processor generates a page fault, which the operating system resolves by simply loading the contents of the page from the paging file and resuming the program that originally caused the error.
The Kernel Memory Space
Terminal Services Session Space This memory area is used by the kernel-mode component of the Win32 subsystem: WIN32K.SYS (see the section on the Win32 subsystem later in this chapter). The page table area is simply a virtual memory map of the currently active page tables.
Section Objects
The internal kernel uses the PTE system space to map device driver executables and to store kernel stacks (there is one for each thread in the system). The system uses file-backed partition objects for various purposes, including loading executable images.
VAD Trees
Like any other paged area of memory, a section supported by a page file can be paged to a page file if necessary. File backed A file backed section object is added to a physical file on the hard disk.
User-Mode Allocations
If it is writable, any changes made to the data while the object is being mapped in memory will be written back to the file. A file-backed section object is a convenient way to access a file because instead of using cumbersome APIs like ReadFile and WriteFile, a program can directly access the data in memory using a pointer.
Memory Management APIs
In Win32, a section object is called a memory-mapped file and can be created using the CreateFileMappingAPI. A section object can be mapped into the user-mode address space using the MapViewOfFileExAPI, and can be unmapped using the UnmapViewOfFileAPI.
Objects and Handles
VirtualProtect This function sets a memory region's protection options, such as whether the block is readable, writable, or executable (newer versions of Windows actually prevent execution of non-executable blocks). All objects use a standard object header that describes the basic object properties, such as its type, reference count, name, and so on.
Named objects
Along with the object pointer, each handle entry also contains an access mask that determines the types of operations that can be performed on the object using that specific handle. The object's access mask is a 32-bit integer that is split into two 16-bit access flag words.
Processes and Threads
BaseNamedObjects This directory is where all conventional Win32 named objects, such as mutexes, are stored. Win32 APIs can never directly access the object in this directory—they must use symbolic links (see below).
Processes
Threads
An interesting aspect of the Windows architecture is that the kernel is preemptive and interruptible, meaning that a thread can normally be interrupted while running in kernel mode, just as it can be interrupted while running in user mode. Not surprisingly, there are some components or areas of code that cannot be interrupted (think what would happen if the scheduler itself was interrupted...), but these are usually very short sections of code.
Context Switching
Each thread is assigned a quantum, which is the maximum amount of time a thread can run continuously. When a thread's quantum goes up, it is suspended and the system allows other threads to run.
Synchronization Objects
The process of suspending and resuming the thread is completely transparent to the thread - the kernel saves the state of all CPU registers before suspending the thread and restores this state when the thread resumes. If more than one thread is waiting, they will each receive ownership of the mutex in the original order in which they requested it.
Process Initialization Sequence
Any thread that attempts to acquire a mutex while it is already owned by another thread will enter a wait state until the original thread releases the mutex or until it exits. When the maximum number is exceeded, a thread requesting ownership of the semaphore will enter a wait state until one of the threads releases the semaphore.
Application Programming Interfaces
The initialization process consists of calling each DLL entry point with the DLL_PROCESS_ATTACH constant. After all DLLs are initialized, LdrpInitialize calls the real thread initialization routine, which is the BaseProcessStart function from KERNEL32.DLL.
The Win32 API
GDI APIs are implemented in the GDI32.DLLand and include low-level graphics services such as those for drawing a line, rendering a bitmap, and so on. USER APIs are implemented in the USER32.DLL module and include all higher-level GUI-related services such as window management, menus, dialog boxes, user interface controls, and so on.
The Native API
Kernel APIs (also known as BASE APIs) are implemented in the KERNEL32.DLL module and include all non-GUI related services, such as file I/O, memory management, object management, process and thread management, and so on. Kernel APIs are used to create and work with kernel-level objects such as files, sync objects, and so on, all of which are implemented in the system's object manager discussed earlier.
System Calling Mechanism
The reason you would want to go through the system call mechanism when calling an API from kernel mode is to "prove" to the API being called that you are actually calling it from kernel mode. This is why operating systems use a special mechanism to switch from user mode to kernel mode.
Executable Formats
In the previous implementation, the invocation of int 2e would store the current value of the EIP and EFLAGS registers. SYSENTER, on the other hand, stores no state information, so by calling into SystemCallStub, the operating system records the address of the current user-mode stub on the stack so that it later knows where to return.
Basic Concepts
When you have a pointer in the executable header, it will always be in the form of a relative virtual address (RVA). Sections are necessary because different areas in the file are handled differently by the memory manager when a module is loaded.
Section Alignment
A common division is to have a code section (also called a text section) that contains the executable's code and a data section that contains the executable's data. The code section contains the executable's code, and the data sections contain the executable's initialized data, meaning they contain the contents of any initialized variable defined anywhere in the program.
Dynamically Linked Libraries
This means that if you just open the executable as a regular file and try to access it, you may run into problems with the RVAs not pointing to the right place. This is because RVAs are calculated using the alignment of the file's section (which is actually its in-memory alignment) rather than using the file's alignment.
Headers
All of these headers are defined in the Microsoft Platform SDK in the WinNT.H header file. Additionally, it should be noted that most of the interesting content in a PE header is actually located in the DataDirectory, which is a series of additional data structures stored within the PE header.
Imports and Exports
Directories
This is the input that gets the current pointer to the imported function at runtime, when the module is loaded. Import Address Table (IAT) Contains a list of entries for A 32-bit list of each function imported from the pointers of the current module.
Input and Output
The I/O System
I/O element such as a network interface, a high-level network protocol, a file system, or a physical storage device. On the other hand, if this filter exists at the network interface level, it will receive low-level network protocol headers such as TCP, IP, and so on.
The Win32 Subsystem
Because USER and GDI are both legacy components that were carried over from older versions of Windows, they do not use the kernel object manager discussed earlier. In reality, the Win32 subsystem does not always allow more than one process to access the same objects; the specific type of behavior object.
Structured Exception Handling
The exception list member is the one of interest; it is the head of the current thread's exception handler list. This sequence simply adds an _EXCEPTION_REGISTRATION_RECORD entry to the current thread's exception handler list.
Reversing Tools
There are no all-in-one recovery tools available (at least not at the time of writing). This chapter describes the different types of tools that are available and provides recommendations for the best products in each category.
Different Reversing Approaches
Offline Code Analysis (Dead-Listing)
Live Code Analysis
A linear sweep simply goes instruction by instruction, meaning any data in the middle of the code could potentially confuse the disassembler. The boxes are connected by arrows that show the flow of the code based on whether the conditional jump is met or not.
ILDasm
The idea is that the debugger provides a detached view of the currently running function and allows the user to step through the detached code and see what the program is doing at each line. While the code is being stepped through, the debugger usually displays the state of the CPU's registers and a memory dump, usually showing the currently active stack area.
User-Mode Debuggers
The following sections describe the advantages and disadvantages of user-mode and kernel-mode debuggers and provide an overview of the most popular tools in each category. PEBrowse offers several informative views of the process, such as a detailed view of the currently active memory heaps and the allocated blocks within them.
Kernel-Mode Debuggers
Once loaded, SoftICE hooks up the system's keyboard driver and essentially monitors keystrokes on the system. Having the system hard drive in a single file on the host really simplifies management and backup.
Patching Tools
Hex Workshop
In addition to being a patching tool, Hex Workshop is also an excellent program for data reverse engineering because it supports the translation of data into organized data structures. Unfortunately, Hex Workshop is not free; it can be purchased at www.bpsoft.com.
Miscellaneous Reversing Tools
Executable-Dumping Tools
PEBrowse Professional is a great PE debugging tool that can also be used as a disassembler (the name may sound familiar from our previous discussion about debuggers - it's not the same product, PEBrowse Professional doesn't offer any live debugging capabilities). PEBrowse Professional can output all PE-related headers, both as raw data and as structured header information.
Applied Reversing
This is possible because reverse engineering provides the best insight into third-party code—it takes you beyond the documentation. In this chapter, I'll show a relatively extreme case where turning techniques are used to learn how to use undocumented system APIs.
Beyond the Documentation
I've chosen a relatively complex set of APIs from the original Windows API, and I'm going to break down the functions in that API to the point where you fully understand what that function does and how to use it. I consider this an extreme case because in many cases one has a level of documentation – it just tends to be insufficient.
Reversing and Interoperability
Laying the Ground Rules
This means that you are primarily just reading lists of assembly languages and trying to decipher them, rather than running programs in the debugger and stepping through them. The whole point is that you're going to look at raw assembly language code as it's presented to you in a real reverse session, and try to extract the information you're looking for from that code.
Locating Undocumented APIs
Although in many cases you'll want to combine both approaches, I've decided to just use the offline (dead list) analysis because it's easier to implement in the context of a written guide. The advantage of using dead lists is that you will be able to follow everything I do just by reading the code lists on the page and analyzing them with me.
What Are We Looking For?
Since the generic table API is big enough as it is, I'll just ignore those functions for the purpose of this discussion. The following reverse session dives into each of the important features of the Generic Table API and demonstrates its inner workings.
RtlInitializeGenericTable
This allows easy access to the parameters passed on the stack regardless of the current value of ESP while the function is running (ESP is constantly changing as the function pushes parameters to the stack while other functions are being called). This is much the same as before: the rest of the structure is initialized.
RtlIsGenericTableEmpty
The function loads ECX with the value of the first parameter (which should be the root data structure from before), and sets EAX to 0. What this function effectively does is it checks if offset +0 of the data structure is 0, and if it the function returns TRUE.
RtlGetElementGenericTable
This means that offset +4 in the root structure points to the last element in the list. The element is closer to the end of the list than the last // element found.
