ANALYSIS
By Andy Asmoro
2-1551-014
MASTER’S DEGREE in
INFORMATION TECHNOLOGY
FACULTY OF ENGINEERING AND INFORMATION TECHNOLOGY
SWISS GERMAN UNIVERSITY The Prominence Tower
Jalan Jalur Sutera Barat No. 15, Alam Sutera Tangerang, Banten 15143 - Indonesia
August 2018
STATEMENT BY THE AUTHOR
I hereby declare that this submission is my own work and to the best of my knowl- edge, it contains no material previously published or written by another person, nor material which to a substantial extent has been accepted for the award of any other de- gree or diploma at any educational institution, except where due acknowledgement is made in this thesis.
Andy Asmoro
Student Date
Approved by:
Dr. Maulahikmah Galinium, S.Kom., M.Sc
Thesis Advisor Date
Charles Lim, B.Sc, M.Sc., CHFI, EDRP, ECSA, ECSP, ECIH, CEH, CEI
Thesis Co-Advisor Date
Dr. Irvan Setiadi Kartawiria, S.T.,M.Sc Dean of Faculty of Engineering and Informa- tion Technology
Date
ABSTRACT
UNPACKING CODE PATTERN FROM PACKED BINARY EXECUTABLE USING EXECUTION UNIT PATTERN BASED SEQUENCE ALIGNMENT
ANALYSIS
By
Andy Asmoro
Dr. Maulahikmah Galinium, S.Kom., M.Sc , Advisor
Charles Lim, B.Sc, M.Sc., CHFI, EDRP, ECSA, ECSP, ECIH, CEH, CEI , Co-Advisor
SWISS GERMAN UNIVERSITY
Software packer has been used effectively to hide the original code inside a binary executable of any malware, making it more difficult for existing signature-based anti- virus software to detect malicious code inside the executable. The objective of this thesis is to develop a new framework using memory Analysis and DNA Sequence that could identify packer type and implement an extraction method of unpacking code from packed malware. By using this method we can identify packer type based on the sim- ilarity with at least 90% similiarity result from the same packer. This show that using this method can extract unpacking code pattern, identified packer type and determine the best algorithm of sequence alignment.
©Copyright 2018 by Andy Asmoro All rights reserved
DEDICATION
I would like to dedicate this research project to my beloved country, Indonesia. I believe this thesis research can contribute to the advancement of science and technology in Indonesia, no matter how subtitle.
ACKNOWLEDGEMENT
I would like to express my deepest gratitude to Dr. Maulahikmah Galinium, S.Kom., M.Sc and Charles Lim, B.Sc, M.Sc., CHFI, EDRP, ECSA, ECSP, ECIH, CEH, CEI as my thesis advisor for the time, support, advice, and guidance given throughout this research project and the completion of this thesis report. It is because of their priceless contributions that this thesis report and the whole research project can arrive at this point.
I would like to thank all of my friends for their companionship, and to the countless number of people who have helped me throughout this research project, either directly or indirectly.
Last, but the most important, I would like to thank my whole family for the countless moral supports throughout my life. It is because of their guidances that I become the person as who I am today. It is because of their affections that I become as happy as I am today.
Contents
Statement by the Author . . . 2
Abstract . . . 3
Dedication . . . 5
Acknowledgement . . . 6
Contents . . . 11
List of Figures . . . 13
List of Tables . . . 13
1. Introduction . . . 14
1.1 Background . . . 14
1.2 Problem Statement . . . 17
1.3 Research Objective . . . 17
1.4 Research Question . . . 18
1.5 Hypothesis . . . 18
1.6 Scope & Limitation . . . 18
1.7 Significance of Study . . . 18
1.8 Thesis Structure . . . 19
2. Literature Review . . . 20
2.1 Malware . . . 20
2.1.1 Malware Definition . . . 20
2.2.1 Malware Type . . . 22
2.3 Malware Stealth Protection . . . 24
2.3.1 Static Stealth Protection . . . 24
2.3.2 Dynamic Stealth Protection . . . 29
2.4 Code Packing Technique . . . 31
2.5 Packer Detection . . . 35
2.5.1 Entropy-Based Static Analysis . . . 35
2.5.2 Reverse Engineering . . . 36
2.5.3 Emulator . . . 36
2.5.4 Simulator . . . 37
2.5.5 Debugger . . . 37
2.5.6 Dynamic Binary Instrumentation . . . 37
2.6 Malware Detection . . . 38
2.6.1 Handling a Packed Malware . . . 39
2.7 Malware Analysis . . . 40
2.7.1 Static Analysis . . . 41
2.7.2 Dynamic Analysis . . . 42
2.7.3 Tools for Dynamic Analysis . . . 43
2.8 Comparison of static and dynamic analysis . . . 44
2.9 Execution Unit Pattern . . . 45
2.10 DNA . . . 46
2.10.1 Biological DNA . . . 46
2.10.2 Digital DNA Sequence . . . 47
2.11 Sequence Alignment . . . 48
2.11.1 Sequence Alignment Concept . . . 48
2.11.2 Sequence Alignment Method . . . 49
2.11.3 Multiple Sequence Alignment Method . . . 50
2.12 Sequence Alignment and Similarity Algorithms . . . 51
2.13 Related Works . . . 56 2.13.1 Previous Work on Malware Analysis using Sequence Alignment 56
quences . . . 57
2.13.3 Previous Work on Packed Malware Analysis . . . 57
2.13.4 Previous Works on Extract Hidden Code From Packed Executa- bles . . . 58
2.13.5 Previous Works on DNA sequences based on frequent patterns . 60 2.14 Our Approach and Position of our study . . . 61
2.15 Summary . . . 62
3. Methodology . . . 63
3.1 Environment Architecture . . . 63
3.2 Research Framework . . . 63
3.3 Get Dataset Samples . . . 64
3.4 Emulation Recording Creation . . . 65
3.5 Instruction Code Extraction . . . 66
3.6 Unpacking Code Extraction and Processing . . . 67
3.6.1 Behavioral Blocking Sequence Extraction . . . 68
3.6.2 Packer Similarity Calculation . . . 69
3.7 Evaluation . . . 69
3.8 Validation . . . 70
4. Experimental Results . . . 71
4.1 Experiment Setup . . . 71
4.1.1 Hardware Setup . . . 71
4.1.2 Software Setup . . . 71
4.1.3 Software Packers . . . 72
4.1.4 Dataset . . . 72
4.2 Benign Experiment Results . . . 74
4.2.1 Recording Creation of Training Dataset . . . 74
4.2.2 Instruction Code Trace . . . 75
4.2.3 DNA Sequence Creation . . . 76
4.4 Analysis . . . 79
4.4.1 Benign Analysis . . . 79
4.4.2 Malware Analysis . . . 80
5. Conclusion . . . 85
5.1 Recommendation . . . 85
5.2 Future Work . . . 86
6. Intel Skylake Instruction Code (Fog, 2011) . . . 88
REFERENCES . . . 102
