Design and Development of Intrusion Detection System

I have followed the norms and guidelines given in the institute's ethical code of conduct. In our opinion, the thesis is worth considering for the award of the doctorate in philosophy according to the institute's rules.

List of abbreviations

Overview

IDS taxonomy based on attack detection

Signature based IDSs

Limitations of existing IDS, Motivation and contribution of the Thesis 3

Anomaly based IDSs

Limitations of existing IDS, Motivation and contribu- tion of the Thesis

ARP based Attacks

Attack through ICMP messages

TCP Low rate DoS attack

Attack based on NDP

Organization of the thesis 15
Organization of the thesis

Limitations of Existing IDS, Motivation and Contribution of Thesis 5. switch) [9] and software-based solutions have also been proposed and are flexible from static IP-MAC pairs. The correctness of the LTL formulas and the system model can be determined before proceeding to detector design.

Active DES based detection of ARP related attacks

Introduction

If the communicating host does not know the destination MAC address, it sends an ARP request to the broadcast domain asking for the MAC address corresponding to the destination host's IP address. The destination host identifies that the ARP request is intended for its IP address and therefore sends back its MAC address in a unicast ARP reply packet.

ARP based attacks and existing countermeasures 19

ARP based attacks and existing countermeasures

Attacks Based on ARP

ARP Spoofing
ARP MiTM Attack
ARP DoS
MAC Flooding

When a MiTM is performed, the attacker gets in the middle of the communication path of the two target computers. Gateway DoS: When the gateway IP address is mapped to a non-existent MAC, systems fail to connect to the gateway.

ARP based attacks and existing countermeasures 21

Existing Detection Mechanisms of Attacks based on ARP

Static ARP Entries[8]
Security Features[9]
Software based solutions
Signature and anomaly based IDS
Modifying ARP using cryptographic techniques
Active techniques for detecting ARP attacks

An IDS using active detection for ARP attacks sends probe packets in addition to observations in changes of IP-MAC pairs. In this way, the IDS can distinguish real MiTM attackers from all systems with IP forwarding.

ARP based attacks and existing countermeasures 23

Proposed Active DES based IDS for ARP related attacks

Active Probing Technique

Proposed Active DES based IDS for ARP related attacks 25

In case of a match in the source IP and mismatch in the source MAC address (i.e. RQPIPS =AUTHTIPS[i] andRQPMACS,AUTHTMACS[i]) the packet is detected to be spoofed without explicit use of the DES -detector and no ARP probes are sent. For such a free request, ARP probe is sent to the RQPIPS without verifying in the tables, because such requests are made when a host comes up with a new IP-MAC pair that needs to be reverified by the detector.

Proposed Active DES based IDS for ARP related attacks 27

An ARP probe response can be determined if RSPIPD is the IP IDS and RSPMAC is the MAC IDS. Output: RSP Intimate Receive, ARPPRSP Probe Response, ARP Probe Request PRQP and DetectedDTD sent, AUTHTinSPOOFT updated.

Proposed Active DES based IDS for ARP related attacks 29

An Example

Also, it is assumed that the authenticated table and the forged table are empty (at this time). Now the IDS can know that the response made in packet sequence 1 is false (since it already has IP(B)-MAC(D)) and the IDS can generate an alert (and also trace the attacking MAC(D)).

Proposed Active DES based IDS for ARP related attacks 31

Probe Request PRQP sent at time T to verify the Source IP-MAC Pair of RSP. IP-MAC same as that of RSP DTD If Source IP-MAC Pair RSP already verified.

Active DES modeling

Active DES modeling 33

State explosion of DES
Adapted Active DES modeling

Each IP-MAC pairing can be modeled by assigning values to corresponding model variables, thus avoiding state explosion. The active model DES is a tuple Σ,X,X0,V,C,=i, where Σ is the set of events,Si is the set of states,X0 ⊆Xis the set of initial states,Vis is the set of model variables,Cis is the set of clock variables and= is the set of transitions.

Figure 2.4: State Explosion in System Model

Active DES modeling 35

Model with measurability

We use the symbol E to denote the measurement equivalence of finite traces and that of transitions, with slight misuse of the notation. The projection operatorP, the inverse projection operatorP−1 and the concept of measurement equivalenceE of finite traces can be naturally extended to traces∈ =w.

Active DES modeling 37

Model with controllability
Failure Modeling

Since the errors are assumed to be permanent, there is no transition from any inxFi state to any inxN state or from any inxFiFj state to any inxFi state.

F i -diagnosability)

The Diagnoser
Active DES modeling 39

DES Modeling of ARP spoofing

Diagnoser for the DES model of ARP request spoofing

Experimentation and Result 47
Experimentation and Result

Testbed Architecture

The overview of the active DES model for ARP under normal and request spoofing cases is as follows. In the normal case (Figure 2.6), the model goes from x1 to statex2 upon observation of a.

Figure 2.5: Active DES model for diagnosis of ARP

IP(B) MAC(B) IP(E) MAC(E) PS 4:PRSP IP(B) MAC(D) IP(E) MAC(E)

Detection Rate, Accuracy and Resource Overhead
Conclusion 53
Conclusion

This is because of cases like attacker spoofing himself etc. which cannot be detected. Figure 2.11 and Figure 2.12 also compare the CPU and memory utilization of the proposed IDS with [17].

Partial Diagnosis of DES based detection for ICMP related attacks

Introduction

For ICMP-related attacks, there is no indicator event available for routes where there is a problem such as congestion. Thus, this framework of limited diagnosticity [20] is adopted to model and design IDS for ICMP-related attacks.

ICMP based attacks and existing schemes to mitigate it 57

ICMP based attacks and existing schemes to mitigate it

Attacks using ICMP Messages

Destination Unreachable

This error message can be forged by the attacker and can be used to stop a host connection to the destination network, resulting in DoS. An attacker can forge this error message to prevent the source from communicating with the destination host.

ICMP based attacks and existing schemes to mitigate it 59

Source Quench
ICMP Redirect
ICMP Router Discovery Messages
ICMP Echo request and reply
Existing Schemes for Detecting ICMP Attacks

Signature-based Detection Systems

Another serious attack is possible by flooding the target host with a huge amount of ICMP echo messages to launch DoS attacks. All hosts on that network will receive such an ICMP echo request and therefore respond with an ICMP echo reply to the victim.

ICMP based attacks and existing schemes to mitigate it 61

Router Configurations
Extensions to ICMP Messages
Cryptographic Techniques

To the best of our knowledge, there is no previous work reported on the modeling of RI detector IDSs. The proposed technique requires the IDS to run on a single host instead of modifying each network host/router.

IDS for ICMP NHU attack using the Failure Detection Theory of DES 63

IDS for ICMP Network / Host unreachable (NHU) attack using the Failure Detection Theory of DES

Network Architecture

The proposed scheme is based on the assumption that the attacker cannot prevent the genuine host from sending a response to the probes initiated by the IDS. Handlers: The handlers send probe packets when required and inform about encountered events to the other module ie. the detector module.

IDS for ICMP NHU attack using the Failure Detection Theory of DES 65

Active Probing Technique

Proposed Algorithms

For the sake of optimization, two global variables – CONGESTION STATE and ICMP COUNTER – are used in all described algorithms. CONGESTION STATE is set to beUnspeci f ieda if the congestion check module was not called during the congestion period.

IDS for ICMP NHU attack using the Failure Detection Theory of DES 67

These values are then substituted into the formula for estimating bandwidth usage [56] and the result is calculated Bandwidth Usage. Input: Host IP address, ICMP packet type, ICMP packet code, SNMP TABLE, CONGESTION STATE.

IDS for ICMP NHU attack using the Failure Detection Theory of DES 69

IDS for ICMP NHU attack using the Failure Detection Theory of DES 71

An Example

An example is used in this subsection to illustrate the handling of forged NHU packets by upstream operators. Furthermore, this example also highlights the difference in ICMP packet sequences (after active probing) in the case of spoofing compared to the normal scenario.

IDS for ICMP NHU attack using the Failure Detection Theory of DES 73

DES Modeling

So initially the ICMP events under normal and spoofed conditions are modeled using state event based DES models and then a DES detector is designed. RepresentationGN: hXN,X0N,ΣN,=N,VNiis used for a DES model under normal conditions andGFi: hXFi,X0Fi,ΣFi,=Fi,VFiiis used for a DES model under fault (attack)Fi condition.

IDS for ICMP NHU attack using the Failure Detection Theory of DES 75

Detector

After transitionτ02 there are two possibilities; (i) the probe response of the authentic host arrives (τ05) in the form of an Echo Reply or (ii) the probe response (τ03) arrives in the form of another NHU packet. Now, in the first case, it becomes obvious that the NHU starter pack has been tampered with.

IDS for ICMP NHU attack using the Failure Detection Theory of DES 77

This means that there is a path in both the normal and attack models that produce the same transition record. If there is a cycle in the detector that is not determined by the attack, no decision can be made in the finite delay after the attack has occurred, since the detector can loop forever in that cycle.

Figure 3.8: Detector for DES Model of Figure 3.6 and Figure 3.7

IDS for ICMP NHU attack using the Failure Detection Theory of DES 79

I-detector

In this section, the concept of partial diagnosis is briefly explained and a partial detector (I-detector) for the NHU attack is constructed. Before a detailed description of the procedure, the concept of partial diagnosis is explained as stated in [20].

Figure 3.9: I-Detector for DES Model of Figure 3.6 and Figure 3.7

IDS for ICMP NHU attack using the Failure Detection Theory of DES 81

Figure 3.6 shows that τ5 corresponds to the fact thatNHUPRS arrived within the specified time for the IPRQP and no congestion was detected in the network afterwards. In this case, the attack cannot be detected within a limited delay even though it has already occurred.

Partial diagnosis Using Reduced I-detector

Partial diagnosis Using Reduced I-detector 83

Reduced I-detector
Construction of RI-detector

Whenever the attack model moves to a statex0 where x0 is the source state of a transitionτ0i, theRI detector moves to the intermediate statemi. Atmi, if the attack model moves to a statex0 where x0 is the destination state of transitionτ0i, theRI detector moves to the final statemf.

Partial diagnosis Using Reduced I-detector 85

The output value of the transition is N because the detector cannot acknowledge the occurrence of a fault at this point. This causes the detector to transition from statem1 to mf issuing Ya, because the occurrence of a fault can be acknowledged after an AD transition.

Experimentation and Results 87

Complexity Analysis

Experimentation and Results

Backtrack 4 respectively. The machine D with Backtrack 4 acts as the attacker machine and machine C is set as the IDS. The graph in Figure 3.11 shows the amount of traffic during normal use, with and without the IDS.

Memory Utlization of IDS

Conclusion 91

Conclusion

Stochastic DES Based Detection and Mitigation of Induced Low rate

TCP-targeted DoS attack

Introduction

Therefore whenever a segment is lost, it is assumed that there is congestion in the network. In case of loss of a segment, the slow start phase starts again and the lost segment is retransmitted.

Induced Low rate TCP-targeted DoS attack and countermeasures 95

Induced Low rate TCP-targeted DoS attack and coun- termeasures

Let the first segment have sequence numbers #, which represent the beginning of the data bytes being sent (in that segment). In the first segment, 1 MSS data bytes are sent from notos no+1×MSS−1.

Figure 4.1: Simple example of Induced Low Rate TCP attack

Stochastic DES 97

Stochastic DES

Stochastic DES 99

Model under Measurability
Failure Modeling

A transition τ = hx,x+, σ,check(V),assign(V),pi, where C(x) , C(x+), is called an error transition, indicating the first occurrence of an error in the set C(x+) - C(x). If, for example, a transitionτ= hx,x+, σ,check(V),assign(V),occurs whereC(x)= {N}andC(x+) ={Fi}, then it suggests that errorFi has occurred.

Stochastic DES 101

The fault diagnosis problem is to determine whether faultFi has occurred within a finite number of nFi (where nFi ∈ N), for example, of transitions after the occurrence of τ1Fi or τ2Fi. Transitionτ04 causes the difference between normal and faulty states and is mandatory for fault detection.

Stochastic DES 103

Stochastic Diagnoser

For the system state xl ∈ z and another state xm ∈ z+, the transition probability τ ∈ a (from xl toxm in model G) is represented by the l,m element Φa. For any other state of the system inz0, the value in the probability matrix is the sum of the probabilities of the traces from the initial state of the system to the considered state of the system; asz0 has the initial state of the system and its non-measurable range, so these traces comprise only non-measurable transitions.

Stochastic DES 105

The probability matrix associated with the initial D-transition0 (whose destination is 2×1) is [1 0.1] which is obtained as follows. ii) The O-output ngaz0 transitions are obtained as follows. The probability matrix associated with D-transition1 is Φa1=. iii) The destination state D corresponding to toa1 is obtained as follows.

Figure 4.3: Stochastic diagnoser for DES of Figure 4.2

Stochastic DES 107

Diagnosability analysis

In the current example we show that the probability of motion in z0,z1 (these are Fi-uncertain D-states) decreases as the length of system traces increases after failure. The D-track probabilityγ is a matrix product of probability matrices associated with the D-transitions in the track.

Application of Stochastic DES for detecting Induced Low Rate TCP attack 109

Proposed scheme: Application of Stochastic DES for detecting Induced Low Rate TCP attack

SEND RCV HANDLER()” module and if a certain Fi state is reached, the alarm is generated indicating the detection of the attack. It receives data as (i) TCP segments from the buffer that have not yet been sent (this is possible since SEND RCV HANDLER() is part of modified TCP); (ii) L1 and L2 are two fixed positive numbers; (iii) TCP Acknowledgments;. iv) TEST FLAG indicating when to detect the attack by sending a segment with fewer data bytes.

Figure 4.4: Block Diagram of the proposed scheme

Application of Stochastic DES for detecting Induced Low Rate TCP attack 111

If SEG NO does not match the RND1 event, TSN is sent to diagnostics and the TCP segment is sent. If a segment is received that is an acknowledgment from the receiver, event TA is passed to the diagnostic.

Application of Stochastic DES for detecting Induced Low Rate TCP attack 113

V={ips,pts,ipd,ptd,seq no,seq no1,RND2,TEST FLAG}is the set of model variables. In the modelling, we assume that a TCP connection has been established, and then τ0 takes place.

Table 4.1: Transitions of the DES model of Figure 4.5

Application of Stochastic DES for detecting Induced Low Rate TCP attack 115

The model variable seq #1 contains the value of the sequence number of the segment used to detect the attack (ie, it has a few data bytes less). So the model variable no contains the sequence number of the last TCP segment that contained MSS data bytes, and seq no.1 stores the sequence number of the segment with fewer data bytes.

Application of Stochastic DES for detecting Induced Low Rate TCP attack 117

Note: If model variables were not used, there would have been transitions (and states) fromx0, each representing a combination of IPS−PTS−IPD− PTD−seq that is not present in the TCP segment. Failure Modeling: One-prime states and transitions correspond to failure (or attack), eg, C(x00)={Fi}and τ01 is a Fi transition.

Figure 4.6: Illustration of state explosion problem without model variables

Application of Stochastic DES for detecting Induced Low Rate TCP attack 119

Application of Stochastic DES for detecting Induced Low Rate TCP attack 121

An Example of Attack Detection

The attacker thinks that the next segment will have the sequence number as 1101 and the MSS data bytes will arrive. Now let's assume that the attacker correctly guesses RND1 and RND2 and sends an acknowledgment with the acknowledgment number as 1171.

Figure 4.8: Attack detection example scenario

Experimentation and Results

We have also discussed assigning model variables in the transitions at the model level. Activating D-transitions and assigning model variables in D-transitions are the same as any of the model transitions contained in the corresponding D-transition.

Figure 4.9: Test Bed setup for the experiments

Experimentation and Results 125

Tables 4.2, 4.3 and 4.4 show detection speed and throughput for different ranges of the random numbers (L1 and L2). In some of the cases, the attacker can guess the random numbers leading to false negatives (ie lower detection rate).

Figure 4.10: Throughput of TCP flow: Normal and Attack scenario

Conclusion

LTL in DES based detection of NDP related attacks

Introduction

IPv6 address identifies a host in the link and can be used to communicate with other hosts on the link. It can be noted that in all the DES frameworks discussed in the last three chapters, modeling of the normal and attack conditions is done manually, which is assumed.

Introduction 131

Model variables are associated with each transition, where they can be assigned values and checked for equality to fire the corresponding transition. Further, it can be concluded that all possible attack scenarios can be detected by IDS.

NDP Related Attacks and Existing Counter Measures 133

NDP Related Attacks and Existing Counter Measures

NDP Related Attacks

Neighbor Solicitation / Advertisement Spoofing
Duplicate Address Detection (DAD) attack

For example, let there be three hosts in a link A,B and D with IP address as IP(A),IP(B) and IP(D), respectively; let the MAC address be MAC(A), MAC(B) and MAC(D) respectively. In IPv6, when a new node wants to come up in the network, before using the IP address, it verifies if there is another node using the same IP address or it is fighting for the same address; this is called a duplicate address detection (DAD) effort.

NDP Related Attacks and Existing Counter Measures 135

Existing Detection Mechanisms

Therefore, WinSEND may not be suitable as it is Windows only and requires patching of all hosts on the network. The basic idea used in active IDS for NDP attacks involves sending test packets to hosts in the LAN in addition to observations (such as changes in IP-MAC pairs).

LTL Based FDD in DES 137

LTL Based FDD in DES

Introduction to LTL

After that, several operations are performed on the Buchi automaton and the system state transition model to generate the detector.

LTL Based FDD in DES 139

Construction of the DES Detector in LTL framework

Notion of Pre-diagnosability and Diagnosability

Lis a label function Si →2AP that labels each state with the set of atomic propositions that are true in that state. The language generated by Md, defined as a series of transition traces, is denoted by lang(Md) ⊆ R∗.

LTL Based FDD in DES 141

Thus, if the model is not diagnosable a priori, it can never be diagnosable, as additional constraints are introduced that do not allow for the monitoring of unobservable events. The model can be pre-diagnosed against the specification f = Gp1 because the false state trace is ((s0,s1)k,sω2), where (s0,s1)k denotes visit0 states and s1 a finite number of times, i.e. k≥1 and ωdenotes visiting the country an arbitrary number of times starting with ω≥1.

Figure 5.1: Example of Pre-diagnosability

LTL Based FDD in DES 143

Algorithm for Testing Diagnosability and Building the Detector

In the first step, the non-defective behavior of the system is represented using an FSA called Buchi Automata, which is obtained automatically from the LTL specification. An input sequence is accepted by the Buchi automaton if there exists a run of the automaton which visits (at least) one of the final states infinitely often.

LTL Based FDD in DES 145

This formal notion implies that a transition tM from q to q0 is added to T2 if and only if there is a path in T1fromqtoq0 in which all the transitions except the last one are caused by unobservable events and the observable equivalent of the last transition istM . Thus the language generated by L20 is all unobservable extension of observable equivalent language generated by L1.

LTL Based FDD in DES 147

Proving Correctness of a Specification

Decomposition Rules

In this sub-section we discuss a procedure to prove the correctness of the LTL formula (developed from a specification of the English language) using appropriateness; scheme is proposed in [71] and discussed here briefly. To verify that it must be checked whether there is an assignment of true in the sub-formulas of the specification so that the overall specification is true.

LTL Based FDD in DES 149

Construction of Reduced Tableau
An Example

A formulaGsp2 or sp1∪sp2 or Fsp2 is satisfiable in a pre-state if there is a path in reduced tableau leading from the pre-states to a node containing the formulasp2. From the reduced tableau, we can determine whether a specification is satisfiable using the following definition.

LTL Based FDD in DES 151

IDS for NDP attacks Using LTL based DES Framework

Assumptions

IDS for NDP attacks Using LTL based DES Framework 153

Packet handler of the IDS

It has three fields, Source IP–SPOOFTIPS, Source MAC–SPOOFTMACS, and the time when the IDS–SPOOFTTi receives the invocation/advertisement packet. The DAD table has four fields, Destination IP–DADTIPT, Source MAC–DADTMACS, Time the packet was received by IDS–DADTtimeo f Received and Probe Send Time (NS) –DADTtimeo f Send.

IDS for NDP attacks Using LTL based DES Framework 155

The details (ie hIPrand,NSPDAD1MACS,−,timeo f sendings) of sending the probe are added to the DADT, a PRNSPDAD event is generated, and EXPHAND() is called, passing the random IP as a parameter. Motivation: This probe is sent to verify the presence of an attacker performing a DAD attack.

IDS for NDP attacks Using LTL based DES Framework 157

The NSP's target IP is already present in DADT and is received within the probe dispatch timeout.

IDS for NDP attacks Using LTL based DES Framework 159

If so, it checks if the authenticity of the NAPDAD IP-MAC pair has already been authenticated (by looking at AUTHT/SPOOFT). If NAPDAD is not a response to NSPDAD1, then it is checked to see if it is a response to any DAD probe (PRNSPDAD) sent earlier.

IDS for NDP attacks Using LTL based DES Framework 161

Neighbor Solicitation Spoofing attack

LTL based DES Modeling of NS spoofing

IDS for NDP attacks Using LTL based DES Framework 167

Since NSPIPS-NSPMACS is spoofed, the IP-MAC pair in the probe response from the real host will be different. The PRNAP event causes transition t3 to go to state4;. iii) The NA Probe Response (PRNAP) arrives from the attacker, which will of course have the same IP−MAC pair as that of NSPIPS−NSPMACS.

IDS for NDP attacks Using LTL based DES Framework 169

Proof of Correctness of the LTL Specification: Normal condition for NS packets under probing

Before proceeding with the diagnostic construction process, the specification must be shown to be correct. As already discussed, a reduced tableau of the specification is constructed to verify correctness.

Construction of DES detector for NS spoofing

IDS for NDP attacks Using LTL based DES Framework 173

Since the only condition labeled F1 in T1 is one where p6 is true, testing for GFp6 is sufficient to test the possibility before diagnosis.

Figure 5.10: NuSMV Code snippet for testing pre-diagnosability of system

An Example to illustrate detection of NS spoofing