3. The kernel function KeSuspendThread is responsible for suspending a thread. Earlier you learned that APCs are involved in thread suspension in Windows 8. Explain how this function works and how APCs are used to implement the functionality on Windows 7. What is different from Windows 8?
4. APCs are also used in process shutdown. The KTHREAD object has a fl ag called ApcQueueable that determines whether an APC may be queued to it.
What happens when you disable APC queueing for a thread? Experiment with this by starting up notepad.exe and then manually disable APC queueing to one of its threads (use the kernel debugger to do this).
5. Explain what the following functions do:
■ KiInsertQueueApc
■ PsExitSpecialApc
■ PspExitApcRundown
■ PspExitNormalApc
■ PspQueueApcSpecialApc
■ KiDeliverApc
6. Explain how the function KeEnumerateQueueApc works and then recover its prototype. Note: This function is available only on Windows 8.
7. Explain how the kernel dispatches APCs. Write a driver that uses the dif- ferent kinds of APCs and view the stack when they are executed. Note:
We used the same method to fi gure out how the kernel dispatches work items.
+0x002 Number : Uint2B +0x008 DpcListEntry : _LIST_ENTRY +0x018 DeferredRoutine : Ptr64 void +0x020 DeferredContext : Ptr64 Void +0x028 SystemArgument1 : Ptr64 Void +0x030 SystemArgument2 : Ptr64 Void +0x038 DpcData : Ptr64 Void
Each fi eld’s semantic is as follows:
■ Type—Object type. It indicates the kernel object type for this object (i.e., process, thread, timer, DPC, events, etc.). Recall that kernel objects are defi ned by the nt!_KOBJECTS enumeration. In this case, you are dealing with DPCs, for which there are two types: normal and threaded.
■ Importance—DPC importance. It determines where this DPC entry should be in the DPC queue. See also KeSetImportanceDpc.
■ Number—Processor number on which the DPC should be queued and executed. See also KeSetTargetProcessorDpc.
■ DpcListEntry—LIST_ENTRY for the DPC entry. Internally, the inser- tion/removal of DPCs from the DPC queue operate on this fi eld. See KeInsertQueueDpc.
■ DeferredRoutine—The function associated with this DPC. It will be executed in arbitrary thread context and at DISPATCH_LEVEL. It is defi ned as follows:
KDEFERRED_ROUTINE CustomDpc;
VOID CustomDpc(
_In_ struct _KDPC *Dpc, _In_opt_ PVOID DeferredContext, _In_opt_ PVOID SystemArgument1, _In_opt_ PVOID SystemArgument2 )
{ ... }
■ DeferredContext—Parameter to pass to the DPC function.
■ SystemArgument1—Custom data to store in the DPC.
■ SystemArgument2— Custom data to store in the DPC.
■ DpcData—A pointer to a KDPC_DATA structure:
0: kd> dt nt!_KDPC_DATA
+0x000 DpcListHead : _LIST_ENTRY +0x010 DpcLock : Uint8B
+0x018 DpcQueueDepth : Int4B +0x01c DpcCount : Uint4B
As you can see, it keeps accounting information about DPCs. The data is stored in the DpcData fi eld of the KPRCB structure associated with the DPC. DpcListHead is the head entry in the DPC queue (it is set during KPRCB initialization) and DpcLock is the spinlock protecting this struc- ture; each time a DPC is queued, the DpcCount and DpcQueueDepth are incremented by one. See also KeInsertQueueDpc. It can be instructive to analyze KeInsertQueueDpc in assembly; pay attention to the KPRCB access and head/tail list insertion.
The DPC usage pattern in code is simple: Initialize the KDPC object with KeInitializeDpc and queue it with KeInsertQueueDpc. When the processor IRQL drops to DISPATCH_LEVEL, the kernel processes all DPCs in that queue.
As mentioned earlier, each CPU core keeps its own queue of DPCs. This queue is tracked by the per-core KPRCB structure:
0: kd> dt nt!_KPRCB
+0x000 MxCsr : Uint4B +0x004 LegacyNumber : UChar +0x005 ReservedMustBeZero : UChar +0x006 InterruptRequest : UChar ...
+0x2d80 DpcData : [2] _KDPC_DATA +0x2dc0 DpcStack : Ptr64 Void +0x2dc8 MaximumDpcQueueDepth : Int4B +0x2dcc DpcRequestRate : Uint4B +0x2dd0 MinimumDpcRate : Uint4B +0x2dd4 DpcLastCount : Uint4B +0x2dd8 ThreadDpcEnable : UChar +0x2dd9 QuantumEnd : UChar +0x2dda DpcRoutineActive : UChar 0: kd> dt nt!_KDPC_DATA
+0x000 DpcListHead : _LIST_ENTRY +0x010 DpcLock : Uint8B +0x018 DpcQueueDepth : Int4B +0x01c DpcCount : Uint4B
The two notable fi elds are DpcData and DpcStack. DpcData is an array of KDPC_DATA structures whereby each element tracks a DPC queue; the fi rst ele- ment tracks normal DPCs and the second tracks threaded DPCs. The function KeInsertQueueDpc simply inserts the DPC into one of these two queues. The relationship can be illustrated as shown in Figure 3-7.
KPRCB KDPC KDPC KDPC
Type Type Type
DpcData[0]
DpcData[1] DpcListEntry DpcListEntry DpcListEntry
DeferredRoutine DeferredRoutine DeferredRoutine
…
… … …
…
… …
…
Figure 3-7
DpcStack is a pointer to a block of memory to be used as the DPC routine’s stack.
Windows has several mechanisms to process the DPC queue. The fi rst mecha- nism is through KiIdleLoop. While “idling,” it checks the PRCB to determine if DPCs are waiting and if so to call KiRetireDpcList to process all DPCs. This is why sometimes these two functions appear on the stack while executing a DPC. For example:
0: kd> kn
# Child-SP RetAddr Call Site
00 fffff800`00b9cc88 fffff800`028db5dc USBPORT!USBPORT_IsrDpc 01 fffff800`00b9cc90 fffff800`028d86fa nt!KiRetireDpcList+0x1bc 02 fffff800`00b9cd40 00000000`00000000 nt!KiIdleLoop+0x5a
The second mechanism occurs when the CPU is at DISPATCH_LEVEL. Consider the following stack:
0: kd> kn
# Child-SP RetAddr Call Site
00 fffff800`00ba2ef8 fffff800`028db5dc USBPORT!USBPORT_IsrDpc 01 fffff800`00ba2f00 fffff800`028d6065 nt!KiRetireDpcList+0x1bc 02 fffff800`00ba2fb0 fffff800`028d5e7c nt!KyRetireDpcList+0x5
03 fffff880`04ac67a0 fffff800`0291b793 nt!KiDispatchInterruptContinue 04 fffff880`04ac67d0 fffff800`028cbda2 nt!KiDpcInterruptBypass+0x13 05 fffff880`04ac67e0 fffff960`0002992c nt!KiInterruptDispatch+0x212 06 fffff880`04ac6978 fffff960`000363b3 win32k!vAlphaPerPixelOnly+0x7c 07 fffff880`04ac6980 fffff960`00035fa4 win32k!AlphaScanLineBlend+0x303 08 fffff880`04ac6a40 fffff960`001fd4f9 win32k!EngAlphaBlend+0x4f4
09 fffff880`04ac6cf0 fffff960`001fdbaa win32k!NtGdiUpdateTransform+0x112d 0a fffff880`04ac6db0 fffff960`001fdd19 win32k!NtGdiUpdateTransform+0x17de 0b fffff880`04ac6ed0 fffff960`001fded8 win32k!EngNineGrid+0xb1
0c fffff880`04ac6f70 fffff960`001fe395 win32k!EngDrawStream+0x1a0
0d fffff880`04ac7020 fffff960`001fece7 win32k!NtGdiDrawStreamInternal+0x47d 0e fffff880`04ac70d0 fffff960`0021a480 win32k!GreDrawStream+0x917
0f fffff880`04ac72c0 fffff800`028cf153 win32k!NtGdiDrawStream+0x9c 10 fffff880`04ac7420 000007fe`fd762cda nt!KiSystemServiceCopyEnd+0x13
This long stack indicates that win32k.sys was handling some graphics operation request from the user, and then the USB port driver’s DPC routine—which has nothing to do with win32k—is executed. What probably happened is that while win32k.sys was handling the request, a device interrupt occurred that caused the CPU to operate at device IRQL; and then the IRQL is eventually lowered to DISPATCH_LEVEL, which causes the DPC queue to be processed.
The third mechanism is through a system thread created during processor initialization. KiStartDpcThread creates a thread (KiExecuteDpc) for each pro- cessor, which processes the DPC queue whenever it runs. For example:
0: kd> kn
# Child-SP RetAddr Call Site 00 fffff880`03116be8 fffff800`028aadb0 nt!KiDpcWatchdog
01 fffff880`03116bf0 fffff800`028aac4b nt!KiExecuteAllDpcs+0x148 02 fffff880`03116cb0 fffff800`02b73166 nt!KiExecuteDpc+0xcb
03 fffff880`03116d00 fffff800`028ae486 nt!PspSystemThreadStartup+0x5a 04 fffff880`03116d40 00000000`00000000 nt!KiStartSystemThread+0x16
Recall that the thread dispatcher runs at DISPATCH_LEVEL, and code running at this IRQL cannot be interrupted by other software IRQLs (i.e., those below DISPATCH_LEVEL). In other words, if there is an infi nite loop in the DPC routine, the processor associated with it will spin forever and the system will practically
“freeze”; in a multi-processor system, it may not freeze but the processor executing the DPC will not be usable by the thread dispatcher. In addition, the DPC routine cannot wait on any kind of dispatcher objects because the dispatcher itself oper- ates at DISPATCH_LEVEL; this is why functions such as KeWaitForSingleObject and KeDelayExecutionThread cannot be called in DPC routines.
N O T E Windows has a DPC watchdog routine that detects DPCs run- ning over a certain time period and bugchecks with code DPC_WATCHDOG_
VIOLATION (0x133). You can query the watchdog timer value by calling KeQueryDpcWatchdogInformation.
Some rootkits use DPCs to synchronize access to global linked lists. For example, they may remove an entry from the ActiveProcessLinks list to hide processes; because this list can be modifi ed at any time by any processor, some rootkit authors use a DPC along with another synchronization mechanism to safely operate on it. In one of the exercises, you will be asked to explain why some authors succeed at this while others fail (machine bugchecks).
Exercises
1. Where and when is the DpcData fi eld in KPRCB initialized?
2. Write a driver to enumerate all DPCs on the entire system. Make sure you support multi-processor systems! Explain the diffi culties and how you solved them.
3. Explain how the KiDpcWatchdog routine works.