http://researchspace.auckland.ac.nz ResearchSpace@Auckland
Copyright Statement
The digital copy of this thesis is protected by the Copyright Act 1994 (New Zealand).
This thesis may be consulted by you, provided you comply with the provisions of the Act and the following conditions of use:
• Any use you make of these documents or images must be for research or private study purposes only, and you may not make them available to any other person.
• Authors control the copyright of their thesis. You will recognise the author's right to be identified as the author of this thesis, and due acknowledgement will be made to the author where appropriate.
• You will obtain the author's permission before publishing any material from their thesis.
To request permissions please use the Feedback form on our webpage.
http://researchspace.auckland.ac.nz/feedback
General copyright and disclaimer
In addition to the above conditions, authors give their consent for the digital copy of their work to be used subject to the conditions specified on the Library Thesis Consent Form and Deposit Licence.
Toward Empirical IP Host Traffic Measurement in Passive Network Measurement
Dong Jin LEE
A thesis submitted in fulfilment of the requirements for the degree of DOCTOR OF PHILOSOPHY in Computer Science,
The University of Auckland, 2009
ii
iv
A BSTRACT
An IP flow represents a group of packets that share the same attribute such as their source address. The ever-growing network traffic produces an enormous number of flows. Recent studies attempt to simplify and mine flows in order to understand the network’s behaviour. The traditional technique of packet aggregation to 5-tuple flows provides understanding of the flows themselves, but fails to capture an understanding of the aggregated end-point that generates flows: the IP host.
This thesis describes the design, development and analysis of a measurement method that identifies an IP host from network traffic. A conceptual model of IP host aggregations has been designed to summarize traffic: from 5-tuple to 2-tuple and finally to 1-tuple IP host. Using the framework, various observations and analyses have been conducted at the host level, including empirical distributions and behaviour relationships.
Several host characteristics and applications are examined from real-world network data, such as characterizing host interaction variability and identifying hosts that are potentially significant.
vi
A CKNOWLEDGEMENT
I am grateful to Associate Professor Nevil Brownlee (supervisor) for his valuable comment and constructive criticism. His contribution played a significant role shaping ideas allowing me to reach to this point.
I am also grateful to Professor Brian E. Carpenter (co-supervisor) and Dr. Ulrich Speidel (ex co-supervisor) for various network measurement discussions.
I thank Andrew Moore and Wei Li from Cambridge Computing Lab (The University of Cambridge) for welcoming me at Cambridge, and motivating my research pursuit. I thank Richard Nelson and Perry Lorier from WAND (The University of Waikato) for providing network resources and helpful information regarding them.
T ABLE OF C ONTENTS
1. Introduction ... 1
1.1. Problem Statement and Motivation ... 2
1.2. Contributions and Overview ... 5
1.3. Background ... 7
1.3.1. History ... 7
1.3.2. Network Management ... 9
1.3.3. Network Measurement ... 10
1.3.4. Passive Measurement Technique ... 12
1.3.5. Monitoring and Measurement Examples ... 14
1.4. Network Traffic Flow ... 16
1.4.1. IP flow Tuple Granularity ... 16
1.4.2. Flow Characteristics ... 17
1.5. Empirical and Theoretical Models ... 19
1.6. Summary ... 22
2. Host Conceptual Model ... 23
2.1. Generic Host Behaviour ... 23
2.2. Aggregation ... 25
2.2.1. Flow [5-tuple] ... 26
2.2.2. Interaction [2-tuple] ... 27
2.2.3. Host [1-tuple] ... 28
2.3. Tuple consideration ... 30
2.3.1. Masqueraded IP host ... 30
2.3.2. Insignificant IP host ... 31
2.3.3. Untraceable IP host ... 31
2.4. Conceptual Implementation ... 32
viii
2.4.1. Bidirectional Traffic Meter ... 34
2.5. Expiry Timeout ... 35
2.6. Observation ... 36
2.7. Selected Hosts Observation ... 39
2.8. Variation ... 40
2.9. Summary ... 42
3. One-way and Two-way Flows ... 43
3.1. Introduction and Motivation ... 43
3.2. Examples of One-way and Two-way Flows ... 44
3.3. One Day Observations ... 46
3.4. Distributions ... 50
3.5. One-way Flow Lifetime and Volume ... 53
3.5.1. One-way Flow Packet Inter-Arrival Time ... 54
3.6. Two-way Flow Lifetime and Volume ... 57
3.6.1. Two-way Flow Flags and RTT ... 59
3.7. Summary ... 61
4. IP Host Measurement ... 63
4.1. Introduction and Motivation ... 63
4.2. Host Arrivals ... 64
4.3. Host Re-entry ... 69
4.4. Per-Host, Interaction and Flow Observation ... 73
4.5. Host Lifetime ... 78
4.6. Host Attribute Correlations ... 81
4.7. Summary ... 87
5. Host Interaction Variation ... 89
5.1. Introduction and Motivation ... 89
5.2. Related Study ... 91
5.3. Host and Interaction ... 92
5.3.1. Observations of Host Interaction and Distributions ... 93
5.4. Interaction Variations ... 97
5.5. Correlations ... 102
5.6. Application Examples ... 104
5.6.1. Significant Variations ... 104
5.6.2. Variation Change ... 106
5.7. Summary ... 108
6. Extracting Significant IP Hosts ... 109
6.1. Introduction and Motivation ... 109
6.2. Related Study ... 111
6.3. Concept of Significant Host ... 113
6.3.1. Attribute Consideration ... 114
6.3.2. Host Score and Rank ... 115
6.4. Host Score Observations ... 117
6.5. Score and Rank Variations ... 121
6.6. Selecting significant hosts ... 125
6.7. Attribute Weights ... 127
6.8. Summary ... 129
7. Conclusion ... 131
7.1. IP Host Measurement ... 131
7.2. Host Application ... 132
7.3. Future Work ... 133
References ... 135
Appendix A. Network traces ... 143
Appendix B. Auckland Network Traffic ... 144
x
Appendix C. Traffic Meter ... 145
L IST OF F IGURES AND T ABLES
Figure 1.1: active and passive measurement example ... 11
Figure 2.1: abstract host behaviours ... 23
Figure 2.2: an illustration of the IP host conceptual model ... 25
Figure 2.3: flow (5-tuple) illustration ... 26
Figure 2.4: interaction (2-tuple) illustration ... 27
Figure 2.5: host (1-tuple) illustration ... 28
Figure 2.6: an example of IP hosts observed by the traffic meter and their tables ... 33
Figure 2.7: an illustration of bidirectional flow matching ... 33
Figure 2.8: expiry timeout measurement ... 37
Figure 2.9: an example of host’s number of flows and interactions ... 39
Figure 3.1: time-series plots of flows over the one day period ... 47
Figure 3.2: time-series plots of hosts over the one day period, Auck-06 ... 48
Figure 3.3: CDF plot of flow and volume lifetime distribution, Auck-06 ... 51
Figure 3.4: CDF plot of host ratio of one-way vs two-way ... 52
Figure 3.5: one-way average flow size distribution vs lifetime, Auck-03 ... 53
Figure 3.6: per-flow (one-way) packet inter-arrival time vs lifetime ... 55
Figure 3.7: two-way average size vs lifetime flow distributions, Auck-03 ... 57
Figure 3.8: two-way flow and RTT distributions ... 58
Figure 4.1: Poisson and exponential distribution with average rate ¸(p;f;i;h), Wits-06 ... 65
Figure 4.2: IAT distributions (24 hours), Wits-06 ... 66
Figure 4.3: re-entry and idle time of hosts ... 70
Figure 4.4: repetition ratio and ratio distribution of hosts, Auck-06 ... 72
Figure 4.5: CDF and CCDF plot showing flow, interaction and host size distribution .. 74
Figure 4.6: CDF and CCDF plot of host, frequency and volume distribution ... 76
Figure 4.7: CDF and CCDF plot showing lifetime distribution ... 79
Figure 4.8: time-series correlation coefficient over one-day period ... 85
Figure 4.9: time-series correlation coefficient with different conditions, Auck-06 ... 86
Figure 5.1: illustration of host interaction characteristic ... 92
Figure 5.2: example of host interaction variation ... 94
Figure 5.3: interaction distributions of two busy (DNS and Web) servers ... 96
Figure 5.4: CDF plots of three coefficients of variation (CoV) ... 98
Figure 5.5: CDF plots separated into 7 interaction sets ... 100
xii
Figure 5.6: CCDF plots separated into 7 interaction sets ... 101
Figure 5.7: scatter plots of Bell-I for hosts interaction set [33-128]. ... 103
Figure 5.8: correlation coefficient for three CoVs ... 103
Figure 5.9: distance (scaled) from the centroid ... 105
Figure 5.10: variation change over time, Auck-07 ... 107
Figure 6.1: abstract host behaviour significance ... 114
Figure 6.2: time-series score plot for selected hosts A to H, Auck-03 ... 118
Figure 6.3: time-series score plot for selected hosts A to L, Auck-06 ... 120
Figure 6.4: observation of host score/rank variation over three/one-hour period ... 123
Figure 6.5: selecting significant hosts based on the score and variation ... 124
Figure 6.6: observation of attribute weight criteria ... 128
Figure C.1: a screenshot of HIF traffic meter ... 145
Table 2.1: example of an IP host represented by various attributes ... 29
Table 3.1: summary of three Auckland network traces (one day) ... 45
Table 4.1: example of an IP host represented by various attributes ... 81
Table 4.2: correlation coefficient summary, Auck-06 ... 83
Table A.1: measured network traces ... 143
Style of the thesis
Footnotes are used for two purposes in the thesis: 1) web resources and 2) other information that aids the paragraph. Two font types are used: 1) Times New Roman font for majority of the sections including Abstract, Introduction, Body and Conclusion. 2) Helvetica font for chapter headers, diagrams of figure/plot and highlighting certain names in the paragraph.
