6.2 Evaluation
6.2.2 Benign Applications Test
88
encryption-type ransomware based on the observed system behaviour (unlinking files) rather than the user perceived behaviour, such as ransom notes or displaying a perpetual window, which may result in a different classification. Further, in our evaluation, we were unable to find any match for the Rename & Unlink File pattern as this behaviour was likely tied to a specific variant of WannaLocker.
The results shown in this evaluation have validated the feasibility of our discov- ered malicious behavioural patterns for detection of encryption-type ransomware.
Additionally, we have shown the feasibility of our streaming approach for detecting malicious patterns by achieving similar successful results to our offline approach.
a second layer of FSM as mentioned in Section 6.1.2.2, which captured the sequence of behaviours.
After re-evaluating with the inclusion of layer 2 FSM, we encountered a much higher false positive rate of 4.2% (21 out of 500) with unchanged true positive rates. The increase in false positive rate was caused by the combination of the suspicious pattern directory search and unlinking user file, which was present in 17 out of 21 of falsely classified benign applications. This issue occurred because the initial directory search pattern matched all folders within the user directory.
This included theAndroid folder where application-specific files were stored. The unlinking user file pattern also had the same issue where any file within the user directory was considered a match. To alleviate this issue, we restricted theDirec- tory Search pattern to exclude the Android folder. This alteration significantly reduced the false positive rate to 1% (5 out of 500) whilst retaining the 100% true positive rate.
This method, however, can potentially produce false negatives, as applications may store valuable data for the user within the application-specific folders or users can also store their own files within the folder. To observe this, we tested the new pattern on 6 different encryption-type ransomware (from different families). Each sample was observed for 5 minutes in an emulated environment with trap files stored within the Android directory. The extension of time from 2 minutes to 5 minutes is to extensively evaluate each application and ensure that all behavioural patterns were captured. In this test, 5 out of 6 ransomware encrypted the files within the Android directory except for Wannalocker, which did not encrypt files within the Android folder. These results posed an issue as the exclusion of the Android directory limited the scope of our detection process. The Android di-
90
rectory is often used to store application related files. For example, gaming save files. Some ransomware samples might not consider specific directories to encrypt, but rather the file extensions. Hence, if there is a file extension of interest in the Android directory, then encryption will still occur.
To mitigate this issue without compromising on the detection rate, we observed the differences in behaviour between benign and encryption-type ransomware, specifically the behaviour of directory search. We noticed that with encryption- type ransomware, a directory search occurred for multiple folders within the user directory to ensure a widespread effect. However, for benign applications this search was less frequent, except for specific applications, such as cache-cleaning applications. To evaluate this theory, the directory search pattern was altered to detect directory searches that occurred two or more times in separate directories.
With this alteration, the false positives rates were reduced to 0.4% (2 out of 500) with 100% true positives. This was a 250% reduction in false positives compared to the methodology of excluding Androiddirectory without compromising on the scope, and accuracy of our detection. Hence, we utilised this methodology in our detection system.
Utilising the Altered Directory Search method, two false positives were de- tected. These two false positives consisted of search unlink sequences. This was likely caused by the applications accessing the same user directory multiple times (i.e., Android directory) and unlinking application related files. As the systems calls were abstracted into tokens, the detection system was unable to identify fine- grain details, such as different user directories being accessed (i.e., if the same user folder was accessed twice, it would be considered a directory search pattern). This is one of the known limitations of our proposed streaming approach.
6.2.2.2 Cache-Cleaning Applications
As previously detailed in Chapter 5, Section 5.2.1.1, specific benign applications, such as cache-cleaning applications could produce behaviours, which can poten- tially be deemed as malicious if the context is not known (e.g., unlinking junk files within the user directory). Hence, we separately evaluated two cache-cleaning applications to evaluate the efficacy of our approaches. By utilising the offline methodology mentioned in Chapter 5, Section 5.1, one of the cache-cleaning ap- plication resulted in a false positive. There were four total malicious patterns matched, and all four of those patterns were linked to Read User File. From the examination of the patterns file and system call log file, these four patterns were reading the contents of the user created files (i.e., pre-existing files, not created by the application), which would be deemed as malicious behaviour as it is unusual for most benign application to be reading the contents of user created files.
Table 6.3 contains a summary of our results, which utilised the offline approach.
ThePercentage column shows the percentages of true negatives and false positives detected for all benign samples evaluated. The Sample Size column denotes the numerical value of true negatives and false positive samples detected, while Ta- ble 6.4 provides an overview of the true negatives and false positives of 502 benign applications for the streaming approach with the 4 aforementioned alterations.
Additionally, the evaluation results for cache-cleaning application have also been included.
We can see that the false positive rates of our streaming approach have notice- ably improved (using the Altered Directory Search method) compared to the offline approach. This was due to the introduction of a layer 2 FSM, which observed se-
92
quences of behaviours, thus further distinguished the differences between a benign and malicious application behaviour. Additionally, based on our observations, we made incremental alterations to the patterns based on the behaviours exhibited by benign and malicious applications to identify the best-fit method for our approach.
The false positive rates show that detecting ransomware and malware in general through behaviours exhibited in system calls is feasible.
Table 6.3: Summary of all benign applications evaluated using offline approach
Benign Samples Percentage Absolute Number Sample Size
True Negative 98.6% 495
502
False Positive 1.4% 7
Table 6.4: Summary of benign evaluation with the streaming approach using afore- mentioned methods
Methodology True Negative False Positive Sample Size
Layer 1 Evaluation 489 (97.8%) 11 (2.2%) 500 Layer 2 Evaluation 479 (95.8%) 21 (4.2%) 500 Restricting User Directory 495 (99%) 5 (1%) 500 Altered Directory Search 498 (99.6%) 2 (0.4%) 500 Incl. Cache-cleaning application 498 (99.2%) 4 (0.8%) 502