DETECTING NEW NETWORK SECURITY THREATS USING DNS AND NETFLOW TRAFFIC

(1)

DETECTING NEW NETWORK SECURITY THREATS USING DNS AND NETFLOW TRAFFIC

By Rinkel Hananto

11302014

BACHELOR’S DEGREE in

Information Technology

Faculty of Engineering and Information Technology

SWISS GERMAN UNIVERSITY The Prominence Tower

Jalan Jalur Sutera Barat no. 15, Alam Sutera Kota Tangerang, Banten 15143

Indonesia

August 2017

Revision after the Thesis Defense on 19 July 2017

(2)

DETECTING NEW NETWORK SECURITY THREATS

USING DNS AND NETFLOW TRAFFIC Page 2 of 88

Rinkel Hananto

STATEMENT BY THE AUTHOR

I hereby declare that this submission is my own work and to the best of my knowledge, it contains no material previously published or written by another person, nor material which to a substantial extent has been accepted for the award of any other degree or diploma at any educational institution, except where due acknowledgement is made in the thesis.

Rinkel Hananto

____________________________________________

Student Date

Approved by:

Charles Lim, M.Sc., ECSA, ECSP, ECIH, CEH, CEI ____________________________________________

Thesis Advisor Date

Ir. Heru Purnomo Ipung, M.Eng

____________________________________________

Thesis Co-Advisor Date

Dr. Ir. Gembong Baskoro, M.Sc

____________________________________________

Dean Date

(3)

Rinkel Hananto

ABSTRACT

DETECTING NEW NETWORK SECURITY THREATS USING DNS AND NETFLOW TRAFFIC

By Rinkel Hananto

Charles Lim, M.Sc., ECSA, ECSP, ECIH, CEH, CEI, Advisor Ir. Heru Purnomo Ipung, M.Eng, Co-Advisor

SWISS GERMAN UNIVERSITY

Uncontrolled network traffic in organizations could lead to many malicious threats, such as data breach, server compromised, server availability, and others. Many network security threats can be detected by monitoring and analyzing network traffic. One of the emerging threats is Domain Name System (DNS) Distributed Denial of Service (DDoS) attack, which ﬂood the authoritative DNS server with large amount of DNS request. Monitoring and understanding the traffic data could prevent such attack.

Therefore, we present a technique for detecting DDoS attack by correlating DNS and NetFlow traffic. The idea is to show that NetFlow can be used as the first DDoS indicator and then DNS is used to evaluate and verify the DDoS. We propose to model the ratio DNS NXDOMAIN response and Information Entropy feature using statistical approach. The traffic is under anomaly condition if the traffic is outside from the standard deviation threshold. We discovered low volume and high volume DDoS attack using statistical approach during the experiment. Attackers’ botnet utilizes DNS to do DDoS called DNS water torture attack or random subdomain attack. The results of the experiment can be used to prevent the attack such as domain blacklist.

Keywords: Botnet, DNS, DDoS, Information Entropy, NetFlow, Network Anomaly Detection, Network Security Threats, Traffic Correlation

(4)

Rinkel Hananto

(5)

Rinkel Hananto

DEDICATION

I dedicate this to my mother, my father and nine TWICE members; Minatozaki Sana, Hirai Momo, Im Nayeon, Kim Dahyun, Myoui Mina, Park Jihyo, Son Chaeyoung, Chou Tzuyu and Yoo Jeongyeon, who makes life worth living.

(6)

Rinkel Hananto

ACKNOWLEDGEMENTS

I would like to deliver my sincere gratitude to my advisor, Mr. Charles Lim, my co advisor Mr. Heru Ipung Purnomo, and Dr. Lukas for the limitless support and time during my thesis research, for their patience, motivation, and immense knowledge.

Their knowledge and guidance help me in all the time of research and writing of this thesis and related researches.

I would like to thank PT Cyberindo Aditama (CBN) for allowing me the opportunity to conduct my research and work as an intern in the company. I speciﬁcally present my gratitude to Rommy Kuntoro, Achmad Yahya Sjarifuddin, Royke Kaligis, and Anderson Lumbantobing for guiding and helping me during the internship and research period at CBN. I would also like to thank the team in Network Operations Center department for cooperating and aiding me with the knowledge and resources needed during the period.

I would also like to thank you to Hansen Chitrahadi and Jason Yapri for mentioning my name in his thesis acknowledgements.

(7)

Rinkel Hananto

TABLE OF CONTENTS

STATEMENT BY THE AUTHOR ... 2

ABSTRACT ... 3

DEDICATION ... 5

ACKNOWLEDGEMENTS ... 6

TABLE OF CONTENTS ... 7

LIST OF FIGURES ... 10

LIST OF TABLES ... 12

CHAPTER 1 - INTRODUCTION ... 13

1.1 Background ... 13

1.2 Problem Statement ... 14

1.3 Research Objectives ... 15

1.4 Research Questions ... 15

1.5 Research Scope ... 15

1.6 Significance of Study ... 15

1.7 Document Structure... 16

CHAPTER 2 - LITERATURE REVIEW ... 17

2.1 The Internet ... 17

2.1.1 Internet Security Threats ... 18

2.1.2 Internal Network Security Threats ... 19

2.2 Domain Name System (DNS) ... 20

2.2.1 Anomaly DNS Traffic... 22

2.2.2 Network Security Threats Related to DNS ... 24

2.3 NetFlow ... 28

2.3.1 Anomaly NetFlow Traffic... 30

2.3.2 Network Security Threats Related to NetFlow ... 31

(8)

Rinkel Hananto

2.4 Log Data Correlation ... 33

2.5 DNS and NetFlow Correlation ... 34

2.6 DNS Approach ... 35

2.6.1 Statistical Approach ... 35

2.6.2 Feature Based Approach ... 35

2.7 NetFlow Approach ... 36

2.7.1 Statistical Approach ... 36

2.7.2 Approach Regarding Packet Sampling ... 37

2.8 Threshold for Statistical Approach ... 38

2.8.1 Static Threshold ... 38

2.8.2 Dynamic Threshold ... 38

2.9 Related Works ... 39

2.9.1 Network Security Threats via DNS Traffic ... 39

2.9.2 Network Security Threats via NetFlow Traffic ... 40

2.9.3 Summary ... 43

CHAPTER 3 - RESEARCH METHODS ... 46

3.1 Research Methodology ... 46

3.2 Research Framework ... 47

3.3 Data Collection ... 48

3.4 Pre-process ... 48

3.5 Analysis ... 49

3.6 Evaluation... 50

CHAPTER 4 - EXPERIMENTAL RESULTS ... 51

4.1 Data Collection ... 51

4.2 Pre-process ... 53

4.2.1 DNS Pre-process ... 53

(9)

Rinkel Hananto

4.2.2 NetFlow Pre-process ... 54

4.3 Analysis ... 55

4.3.1 Dynamic Threshold Calculation ... 57

4.4 Evaluation... 59

4.4.1 Evaluation on 14 March 2017 ... 59

4.5 Summary of Analysis and Evaluation ... 73

CHAPTER 5 – CONCLUSION AND RECOMMENDATIONS ... 76

5.1 Conclusion ... 76

5.2 Recommendations ... 77

5.3 Future Works ... 77

GLOSSARY ... 78

REFERENCES ... 79

APPENDIX ... 84

CURRICULUM VITAE ... 87