EventId Content
Log Message Content
< . . .Preamble . . .>
List of variable parameters occurring in the log message Log
message line number
EventTemplate ParameterList LineId
Log Key - unique log key representing the log event template Various Metadata
Event Template - template that describes the static part of the log message
Figure 6.2: Data structure and format of the data passed from the the Data Miner to the Inference Engine. Mandatory fields are also illustrated
The rows of the data structure in Figure 6.2 represent individual log messages from a given log file while the columns represent data fields describing the log message. The mandatory data fields required for this data interface, as shown in Figure 6.2, are:
• LineId - contains the line number of the log message
• Content - contains the raw log message content
• EventId - contains the log event key corresponding to the log message content
• EventTemplate - contains the log event template corresponding to the log message content
• ParameterList - contains a list of variable parameters extracted from the log message content
Depending on the structure of the log messages, additional data fields may be generated as described in Chapter 4). These are shown as the Preamble in Figure 6.2. These fields contain metadata information about the log messages, e.g. timestamps, source file names, and may be used for further processing or analysis but are not considered by the Inference Engine in this study.
6.2.2 External Interfaces
There are two major external interfaces of the ALFAF as shown in Figure 6.1. This includes the Input Data Interface to the Log Files of System Under Test and a User Output Interface to theEnd-User.
Input Data Interface
The ALFAF is designed to be tunable and adaptable to be able to perform automated log file analysis on log files produced by different systems. Most systems generate log files using raw text and these files are stored on disk. The ALFAF assumes this of all log files provided for analysis.
The Input Data Interface is a functional interface that provides access to the log files to be analysed to the ALFAF. The framework may be deployed on hardware forming part of the actual system under test that generates the log files, or it may be deployed on a separate, stand-alone system. To accommodate either scenario, the Input Data Interface specifies that access to log files be provided by the following:
• path to directory - an absolute path to a directory that contains the log files to be analysed by the ALFAF. Regardless of where the framework is deployed, this directory shall be accessible withread file privileges
• log file name - the file name, including extension, of the log file that is to be analysed or ingested by the framework. This log file shall exist in the directory specified above.
User Output Interface
The objective of the ALFAF is to perform automated log file analysis, with the aim of identifying possible anomalous events, errors and failures that have occurred during system operation, to provide information that operators and/or developers may use to assist in their efforts to debug the system failure.
The User Output Interface is a data interface that describes the data format, structure and required data fields of the output generated by the framework, that may be used by end-users to assist in debugging. The ALFAF generates two outputs after analysing a log file. These are the Debugging Report and the Suspicious Lines Report. Both reports are structured in pandas DataFrames and are stored as CSV files. The structure of these reports is shown in Figure 6.3.
It should be noted that all data fields are applicable to all row entries in the data structure - the separation in Figure 6.3 is for visualisation purposes.
LineId
Event Template - template that describes the static part of the log message
< . . .Preamble . . .>
Various Metadata
Log Key - unique log key representing the log event template Log
message line number
Log Message Content
Content EventId EventTemplate
Is this log message an anomaly?
anomaly candidate_keys
Possible valid log keys for this log message, based on the input sequence actual_next_key
Sequence of log keys used to determine if this log message is an anomaly
Log key of this log message - as it appears in the log file ParameterList
List of variable parameters occurring in the log message
log_key_sequence
continues
Figure 6.3: Data structure and format of the Debug and Suspicious Lines Reports As shown in Figure 6.3, the data structure is based on the parsed log data structure shown in Figure 6.2, with additional data fields. Each row of the data structure again represents a single log message from a given log file. The additional fields are as follows:
• log key seq - contains the input sequence of log messages, represented as log event keys, used to determine whether a particular log message is anomalous or not
• actual next key - contains the actual log key that appeared in a given line in the log file
• candidate keys - contains a list of the likeliest log messages, represented as log keys, that follow the given input sequence of log messages
• anomaly - contains a flag that specifies whether a particular log message is anomalous or
not. 1 represents an anomalous event, 0 represents a normal event.
Both the Debugging Report and the Suspicious Lines Report share the same data structure shown in Figure 6.3. The only difference between the two reports is that the Suspicious Lines report only contains the lines that have been flagged as anomalous, while the Debugging Report contains all lines contained in the entire log file. This approach enables faster debugging as operators or developers can consult the Suspicious Lines Report to identify which lines, as indicated by their line numbers in the original log file, are anomalous, and then cross-reference these with the Debugging Report to analyse the events leading to, and following, the anomalous event.