Using Memory Management to Detect and Extract Illegitimate Code for Malware AnalysisCarsten Willems1, Thorsten Holz1, Felix Freiling2

1Ruhr-University Bochum, Germany2University of Erlangen, Germany

Annual Computer Security Applications Conference (ACSAC) 2012

左昌國Seminar @ ADLab, NCU-CSIE

• Introduction• Related Work• Model and Definitions• Approach• Implementation• Application to the Analysis of PDF Documents• Detection Evaluation• Extraction Evaluation• Conclusion

Outline

2

• Exploiting Software• The ultimate aim is to perform malicious computation

• Executing illegitimate code (a.k.a. shellcode)

• Countermeasures in OSs• Data Execution Prevention (DEP)• Address Space Layout Randomization (ASLR)

• Above countermeasures do not help in the analysis of illegitimate code

Introduction

3

• CWXDetector• A tool for the analysis of malware for the Windows OS • Dynamic analysis for detecting and extracting illegitimate code

• Using memory management techniques

• For analysis• CWXDetector is not meant to protect a system• But to monitor and analyze the illegitimate code

Introduction

4

• Limitations• Dynamic analysis

• Could get incomplete results• Incapable to detect malicious code embedded in arbitrary data (only

those being executed)• Malicious computation does not always imply the existence of

illegitimate code• Return Oriented Programming (ROP)• JIT-spraying

Introduction

5

• Preventive Measures• Detection of Illegitimate Code• Extraction of Illegitimate Code

Related Work

6

• Microsoft EMET tool (Enhanced Mitigation Experience Toolkit)• http://

blogs.technet.com/b/srd/archive/2012/05/15/introducing-emet-v3.aspx

• CFI

• Cons:• Self-modifying or dynamically created code• No assistance to further analysis

Related Work – Preventive Measures

7

• Static signatures• “sled component” searching• Heuristics-based approaches

• Cons:• Not generic• Have to be extended when new anti-detection come up

Related Work – Detection of Illegitimate Code

8

• OllyBone• OmniUnpack• Renovo

• Cons:• OllyBone: debugger-driven malware analysis• OmniUnpack: rely on signatures, FP• Renovo: requiring emulation environment

Related Work – Extraction of Illegitimate Code

9

• Attacker Model• A remote attacker provides some malicious piece of data to exploit

a vulnerability application• Resulting in the execution of shellcode• Not single staged full-ROP/JIT-sprayed attacks

• Illegitimate code (ILC)• Code that is not legitimate (would not execute if functions properly)

Model and Definitions

10

• Enforcing an Invariant• When a vulnerability is exploited, the control flow is redirected to

one of the following locations:• ILC on the stack (buffer overflow)• ILC in the heap (heap-spraying)• ILC in a static data area (exploiting a static data buffer)

• The approach enforces the following invariant• All ILC resides in non-executable memory• all execution attempts of ILC will result in page fault exception

Approach

11

• Trusted Files and Functions• All existing files are trusted before analysis.• All created or modified files are untrusted during later operation.• Trusted memory modification functions

• Trusted callers

Approach

12

• Memory Protection Modifications• Only trusted callers can allocate executable memory.• Only trusted callers can modify existing memory to being

executable.• Only trusted files can be mapped into executable memory.• all attempts that violate these rules are intercepted (hooked) and

the resulting memory regions becomes non-executable.

• Exception:• If the target memory belongs to a mapped trusted file (writable), the

executable right has to be removed.• Enforcing W ^ X

Approach

13

• Custom Page Fault Handler• When page fault triggered

• Check if it is related to the system• If so,

• Dumping memory• Modifying the memory to executable and continuing

• Multi Version Dumping• Different versions of each executed page may be created• Compare each dumped record

Approach

14

• CWXDetector• For x86 version of Windows XP

• Memory Function Hooks• NtAllocateVirtualMemory• NtProtectVirtualMemory• NtMapViewOfSection

• Checking the Caller• Trace the user-mode call stack

• Custom Page Fault Handler• Hook MmAccessFault

• Check if the fault was cause by execute operation• Check if the fault address resides in user-space

• Additional• NtCreateFile• NtCreateProcess

Implementation

15

• 32bit Windows XP SP2• Adobe Acrobat Reader versions 6.0.0, 7.0.0, 7.0.7, 8.1.1,

8.1.2, 8.1.6, 9.0.0, 9.2.0, and 9.3.0• Foxit Reader version 3.0.0

• 1. installed the customized page fault handler and the system hooks

• 2. started the particular viewer application• 3. disabled DEP for the viewer application• 4. opened the PDF document• 5. enforced the invariant: new allocated memory and

modified memory

Application to the Analysis of PDF Documents

16

• 6. if the execution of ILC was detected, dumped the memory page to a file, created a log entry, and modified the related PTE to being executable.• Check the dumped file for patterns marked as “PATTERN”

• 7. if a new process was created by the PDF viewer• marked as “PROCESS” and prevent the process from spawning

• 8. if a dialog window was shown• marked as “DIALOG” and simulated a user input to close the

window• 9. “CRASH”• 10. time out “NOTHING”

Application to the Analysis of PDF Documents

17

• The log file contains information about:• All attempts to allocate executable memory invoked by untrusted

callers• All attempts to modify existing memory to being executable invoked

by untrusted callers• All attempts to execute memory that contains ILC• All created files• All created processes• All shown user dialog windows

Application to the Analysis of PDF Documents

18

• Every PDF file ended up with a tuple (d, c)• d: whether illegal code was detected• c: {PATTERN, CRASH, PROCESS, DIALOG, NOTHING}• (d, c) > (d’, c’) iff either d had detected ILC and d’ not, or

(if d = d’) c > c’• PATTERN > CRASH > PROCESS > DIALOG > NOTHING

Application to the Analysis of PDF Documents

19

• Determining Trusted Callers• Identifying all the functions from all trusted files that are used to

produce executable memory• Loader-related function in ntdll.dll• Tested benign PDF documents and manually inspected the function

calls

Application to the Analysis of PDF Documents

20

• Benign PDF Sampleset• Retrieved URL of the TOP 5000 sites from alexa• Queried Google for the first 10 PDF documents on each site• Using tool pdfid, selected all documents which contained

JavaScript, OpenActions or some other extended PDF features• Uniformly picked random samples from other files to 7,218

samples• The benign set on the system

• No ILC execution detected FP 0%

Detection Evaluation

21

• Malicious PDF Sampleset• 7,278 known malicious PDF documents form a well-known AV

vendor• all their valid PDF samples from Jan. 2011

Detection Evaluation

22

Detection Evaluation

23

• 15 “CRASH”:• All performed invalid memory accesses (before executing ILC)

• 33 “PROCESS”:• Using regular build-in features to create IE or OE with special

crafted parameters• 295 “DIALOG”:

• Invalid embedded JavaScript code• Social engineering

• 154 “NOTHING”:• Required specific environment

Detection Evaluation

24

Detection Evaluation

25

Detection Evaluation

26

• Adobe Acrobat Reader 9.0.0• 4,869 samples• Quality:

• Valid x86 instructions (code ratio)• Contained strings

• Partitions

Extraction Evaluation

27

• Analyzed all 4,869 PDF samples and got 2 versions of partitions• Initial partition• Final partition

• Code ratio determined by IDA Pro• Valid strings

Extraction Evaluation

28

Extraction Evaluation

29

• Initail partitions• 7,807 strings• 1,866 URLs

• Final partitions• 8,676 strings• 2,280 URLs

Extraction Evaluation

30

• A generic and automatic method to detect and extract illegitimate code

• Effective in supporting malware analysis• Also good detection rates

Conclusions

31