CXPInspector: Hypervisor-Based, Hardware-Assisted System Monitoring

Carsten Willems, Ralf Hund, Thorsten Holz

TR-HGI-2012-002, Ruhr-Uni­ver­si­tät Bo­chum, Horst Görtz In­sti­tut für IT-Si­cher­heit (HGI), November 2012


Abstract

In the last few years, many di erent techniques were introduced to analyze a given binary executable. Most of these techniques take advantage of Virtual Machine Introspection (VMI), the process of analyzing the state of a virtual machine from the outside. On the one hand, many approaches are based on system emulators which enable a tight control over the program execution. Unfortunately, such approaches typically induce a huge performance overhead. On the other hand, there are approaches based on hypervisors. Early implementations were hampered by the missing virtualizability of the x86 instruction set architecture: since the memory management unit (MMU) itself was not virtualized, memory separation needed to be enforced in software with the help of so called shadow page tables, an approach that again induced performance overhead. However, processor vendors have recently added hardware support for MMU virtualization and modern CPUs o er so called Two-Dimensional Paging to overcome such performance bottlenecks.

In this paper, we study how this processor feature can be utilized to implement a binary analysis framework. More speci cally, we introduce an approach to monitor code execution based on the concept of Currently eXecutable Pages (CXP), i.e., we precisely control which memory pages are currently executable to enable the interception of intermodular function calls and their corresponding returns. When an interception occurs, we apply VMI to deduce runtime information such as function parameters. To demonstrate the practical feasibility of the proposed approach, we implemented CXPInspector, a framework for binary analysis on 64-bit machines and Windows 7. In several case studies, we present di erent application scenarios for CXPInspector: rst, we demonstrate how the kernel rootkit TDSS/TDL4 can be analyzed in an automated way. Second, we show how our tool can be used for transparent and ecient performance pro ling in a case study with the Apache webserver.

[pdf]

Tags: analysis, Malware