Asynchronous procedure calls (APCs) are used to implement many important operations such as asynchronous I/O completion, thread suspension, and process shutdown. Unfortunately, they are undocumented from a kernel perspective.
The offi cial driver development documentation simply includes a short section acknowledging that APCs exist and that there are different types. However, for common reverse engineering tasks, it is not necessary to understand all the underlying details. This section explains what APCs are and how they are commonly used.
APC Fundamentals
Generally speaking, APCs are functions that execute in a particular thread context.
They can be divided into two types: kernel-mode and user-mode. Kernel-mode
APCs can be either normal or special; normal ones execute at PASSIVE_LEVEL, whereas special ones execute at APC_LEVEL (both execute in kernel mode). User APCs execute at PASSIVE_LEVEL in user mode when the thread is in an alertable state. Because APCs run in thread context, they are always associated with an ETHREAD object.
Concretely speaking, an APC is defi ned by the KAPC structure:
1: kd> dt nt!_KAPC
+0x000 Type : UChar +0x001 SpareByte0 : UChar +0x002 Size : UChar +0x003 SpareByte1 : UChar +0x004 SpareLong0 : Uint4B
+0x008 Thread : Ptr32 _KTHREAD +0x00c ApcListEntry : _LIST_ENTRY +0x014 KernelRoutine : Ptr32 void +0x018 RundownRoutine : Ptr32 void +0x01c NormalRoutine : Ptr32 void +0x014 Reserved : [3] Ptr32 Void +0x020 NormalContext : Ptr32 Void +0x024 SystemArgument1 : Ptr32 Void +0x028 SystemArgument2 : Ptr32 Void +0x02c ApcStateIndex : Char +0x02d ApcMode : Char +0x02e Inserted : UChar
This structured is initialized by the KeInitializeApc API:
KeInitializeApc
NTKERNELAPI VOID KeInitializeApc(
PKAPC Apc, PKTHREAD Thread,
KAPC_ENVIRONMENT Environment, PKKERNEL_ROUTINE KernelRoutine, PKRUNDOWN_ROUTINE RundownRoutine, PKNORMAL_ROUTINE NormalRoutine, KPROCESSOR_MODE ProcessorMode, PVOID NormalContext
);
NTKERNELAPI BOOLEAN KeInsertQueueApc(
PRKAPC Apc,
PVOID SystemArgument1, PVOID SystemArgument2, KPRIORITY Increment );
Callback prototypes
typedef VOID (*PKKERNEL_ROUTINE)(
PKAPC Apc,
PKNORMAL_ROUTINE *NormalRoutine, PVOID *NormalContext,
PVOID *SystemArgument1, PVOID *SystemArgument2 );
typedef VOID (*PKRUNDOWN_ROUTINE)(
PKAPC Apc );
typedef VOID (*PKNORMAL_ROUTINE)(
PVOID NormalContext, PVOID SystemArgument1, PVOID SystemArgument2 );
typedef enum _KAPC_ENVIRONMENT { OriginalApcEnvironment, AttachedApcEnvironment, CurrentApcEnvironment, InsertApcEnvironment
} KAPC_ENVIRONMENT, *PKAPC_ENVIRONMENT;
N O T E This defi nition is taken from http://forum.sysinternals.com/
howto-capture-kernel-stack-traces_topic19356.html. While we cannot guarantee its correctness, it has been known to work in experiments.
Apc is a caller-allocated buffer of type KAPC. In practice, it is usually allocated in non-paged pool by ExAllocatePool and freed in the kernel or normal routine.
Thread is the thread to which this APC should be queued. Environment determines the environment in which the APC executes; for example, OriginalApcEnvironment means that the APC will run in the thread’s process context (if it does not attach to another process). KernelRoutine is a function that will be executed at APC_LEVEL in kernel mode; RundownRoutine is a function that will be executed when the thread is terminating; and NormalRoutine is a function that will be executed at PASSIVE_LEVEL in ProcessorMode. User-mode APCs are those that have a NormalRoutine and ProcessorMode set to UserMode. NormalContext is the parameter passed to the NormalRoutine.
Once initialized, an APC is queued with the KeInsertQueueApc API. Apc is the APC initialized by KeInitializeApc. SystemArgument1 and SystemArgument2 are optional arguments that can be passed to kernel and normal routines.
Increment is the number to increment the run-time priority; it is similar to the PriorityBoost parameter in IoCompleteRequest. Where is the APC queued?
Recall that APCs are always associated with a thread. The KTHREAD structure has two APC queues:
0: kd> dt nt!_KTHREAD
+0x000 Header : _DISPATCHER_HEADER +0x018 SListFaultAddress : Ptr64 Void +0x020 QuantumTarget : Uint8B
…
+0x090 TrapFrame : Ptr64 _KTRAP_FRAME +0x098 ApcState : _KAPC_STATE +0x098 ApcStateFill : [43] UChar +0x0c3 Priority : Char +0x288 SchedulerApc : _KAPC
…
+0x2e0 SuspendEvent : _KEVENT 0: kd> dt nt!_KAPC_STATE
+0x000 ApcListHead : [2] _LIST_ENTRY +0x020 Process : Ptr64 _KPROCESS +0x028 KernelApcInProgress : UChar
+0x029 KernelApcPending : UChar +0x02a UserApcPending : UChar
The ApcState fi eld contains an array of two queues, storing kernel-mode and user-mode APCs, respectively.
Implementing Thread Suspension with APCs
When a program wants to suspend a thread, the kernel queues a kernel APC to the thread. This suspension APC is the SchedulerApc fi eld in the KTHREAD structure; it is initialized in KeInitThread with KiSchedulerApc as the normal routine. KiSchedulerApc simply holds on the thread’s SuspendEvent. When the program wants to resume the thread, KeResumeThread releases this event.
Unless you are reverse engineering the Windows kernel or kernel-mode rootkits, it is unlikely that you will run into code using APCs. This is primarily because they are undocumented and hence not commonly used in commercial drivers. However, APCs are frequently used in rootkits because they offer a clean way to inject code into user mode from kernel mode. Rootkits achieve this by queueing a user-mode APC to a thread in the process in which they want to inject code.
Exercises
1. Write a driver using both kernel-mode and user-mode APCs.
2. Write a driver that enumerates all user-mode and kernel-mode APCs for all threads in a process. Hint: You need to take into consideration IRQL level when performing the enumeration.
3. The kernel function KeSuspendThread is responsible for suspending a thread. Earlier you learned that APCs are involved in thread suspension in Windows 8. Explain how this function works and how APCs are used to implement the functionality on Windows 7. What is different from Windows 8?
4. APCs are also used in process shutdown. The KTHREAD object has a fl ag called ApcQueueable that determines whether an APC may be queued to it.
What happens when you disable APC queueing for a thread? Experiment with this by starting up notepad.exe and then manually disable APC queueing to one of its threads (use the kernel debugger to do this).
5. Explain what the following functions do:
■ KiInsertQueueApc
■ PsExitSpecialApc
■ PspExitApcRundown
■ PspExitNormalApc
■ PspQueueApcSpecialApc
■ KiDeliverApc
6. Explain how the function KeEnumerateQueueApc works and then recover its prototype. Note: This function is available only on Windows 8.
7. Explain how the kernel dispatches APCs. Write a driver that uses the dif- ferent kinds of APCs and view the stack when they are executed. Note:
We used the same method to fi gure out how the kernel dispatches work items.