A Prescriptive Analytical Logic Model for Error Analysis on Enterprise Level of Software Application
Hoo Meng Wong1*, Sagaya Sabestinal Amalathas2, Sayan K. Ray3, Harry Murdani4
1 Taylor’s University, Subang Jaya, Selangor, Malaysia
2 University of Southampton, Nusajaya, Johor, Malaysia
3 Manukau Institute of Technology, New Zealand
4 CISSP, CCSK
*Corresponding Author: [email protected]
Accepted: 15 June 2022 | Published: 30 June 2022
DOI:https://doi.org/10.55057/ijarti.2022.4.2.5
_________________________________________________________________________________________
Abstract: Despite its multiple periods, Information Technology (IT) continues to accelerate its growth. Today, the great bulk of corporate operations rely heavily on software programs.
Particularly for businesses that rely on the software application to handle a high volume of commercial transactions. As the volume of daily business transactions increases, it is undesirable to extend the downtime window associated with a malfunctioning software application. Regardless of the variety of variables or causes that can result in a severe error in a software application and consequent downtime. If the fundamental cause of a software application's unavailability can be reliably determined within the service level agreement (SLA) or even lower, the downtime timeframe can be reduced. However, the principal cause of the error could be within the software application layer or could be caused by an external factor. This complicates the root cause analysis procedure, much more so when dealing with an enterprise software application with many levels of security (such as a client tier, a web tier, an application layer, middleware, and database tier). In these circumstances, accurately identifying the root cause becomes far more challenging, as root cause analysis requires more than one event recording file (i.e., log file). With this degree of complication, the duration of the root cause analysis activity may be prolonged. The total amount of time required increases, and the rise in total analysis time suggests that it may take longer to restore the software service for users. It is essential to have a logic model capable of precisely identifying the error's root cause. The significance of the Prescriptive Analytical Logic Model (PALM) lies in the fact that it includes not only supervised learning for error detection and optimal treatment of the identified root cause but also AHP for prioritizing which errors to resolve first. The optimum treatment is to reduce the time required for root cause analysis procedures and increase their accuracy by resolving the legitimate error with the highest priority.
Keywords: Analytic Hierarchy Process, Log File Analysis, Logic Model, Supervised Learning _________________________________________________________________________
1. Introduction
Organizations in today's information technology era continue to rely heavily on software applications to survive, grow and even boost business productivity [1] [2]. If the software application fails, it will have a significant impact on the organizations. Identifying the root cause of a software application's failure is tough in any context, but it becomes significantly more complex when the application is hosted in a multi-tier environment. Simply examining
the software application log file is insufficient for discovering the root cause of a software application's true failure point [5]. Even if all interested parties' log files are collected and one error is detected in one log file and another error is discovered in another log file, it is impossible to determine the relationship between these two errors. This is a significant issue and one that could be investigated.
2. Background
It is tough to specify how much time a person should dedicate to the root cause analysis. This is because it is still relying on the individual's skills and job experience. The precision and trustworthiness of the root cause analysis results are crucial since they lead to the resolution of the underlying issue. Any improper judgment on root cause analysis is made will result in rework. The rework is expensive as it entails effort and time. The Root Cause Analysis (RCA) performed by humans is usually time-consuming [4]. In addition, software application unavailability can result in financial losses to business operations [3]. This is because when a software application experiences an error during execution, users are unable to do daily transactions while the root cause analysis is conducted.
3. Motivation
Although the analysis itself may be time demanding, the opportunity to reduce or eradicate the core causes of multiple reoccurring problems/problem patterns is well worth the effort [9].
Having said that, the question remains whether real errors can be accurately recognized after spending so much effort on root cause research. On the other hand, the execution of a software application requires sufficient server resources. If the server does not have sufficient resources, the software application's events cannot be accurately recorded in the log file. However, even if server resources are adequate, increased information in logging events might have a negative influence on the speed of the software application [2] [3]. With this context in mind, it becomes clear why the current research is necessary. It attempts to increase the precision with which root causes are recognized and the duration of root cause activities. Following the identification of the root cause, certain activities are carried out. These operations are undertaken during the bug fix development, testing, and deployment. The bug fix is referring to the resolution designed to resolve the recognized root cause. All of these activities take place in a sequence.
Each activity is conditional on the prior task being accomplished. Reducing the duration of the yellow bar (i.e. root cause analysis) permits the complete software application downtime to get shorter.
Figure 1: The Illustration of time on Software Application Downtime
4. Potential Contributions
The objective is to provide a logic model, PALM. The specified algorithm will be used to design and implement this logic model. The method consists of an error pattern recognition technology and a decision-making procedure for determining the root cause based on the relationship between the observed error occurrences in all linked log files. The appropriate resolution will be recommended in light of the determined root cause. PALM analyzes all log file events across the log files of all involved parties. Rather than focusing exclusively on the software application log file, this technique provides a broader scope for identifying the true root cause. Even though the logic model does not forecast faults, it takes proactive measures to shorten the duration of root cause analysis and to prevent human error when discovering real errors.
5. Log File Analysis
There have been several tools and techniques for analyzing software application error logs introduced to the market. The scope and focus, on the other hand, differ considerably.
i. Troubleshooting real-time software application failures by utilizing logic analyzer debug macros [6].
ii. Translating error logs to be more readable [9].
iii. Concentrating exclusively on mistake detection in software applications during their development and maintenance phases [10].
iv. Examining error logs to forecast future failures; is referred to as online failure prediction [7].
Apart from (ii), the standard error logs analysis activity is limited to the study of software application error logs. These log events are gathered from either software application files or database tables (depending on whether the log is stored in a file or a database table). The strategy is advantageous for debugging and profiling reasons when the software application problem occurs within the software application boundary [8]. It is pioneered to use of a formal syntax for identifying distinct data relations in heterogeneous tables [8]. The work attempted to cover as much as possible of the investigation into some important theories and outlined the main objectives and methodologies of the log file analysis. Additionally, multiple case studies
were attempted to cover in the same paper, each of which highlighted a different feature of software application log file analysis. The important point was realized: what if the root cause stretched beyond the software application's boundaries, and the root cause analysis effort was wasted on examining only software application errors.
6. Supervised Learning
The use of machine learning to predict software application errors was proposed [12] as a method of preventing software application failures. During the software development life cycle, the focus was entirely on detecting the error within the software application boundary.
The benefit of machine learning is that it may be combined with supervised learning to identify error keywords during root cause analysis. This issue identification method can be expanded to include error terms outside the software application's bounds. By incorporating machine learning into a tool for analyzing infrastructure-based system data [11]. After evaluating the incoming events, the system discovered the normal behavior of each system and produced a database of event structures based on the incoming events. It claimed it could improve error detection intelligence by evaluating error patterns based on database-stored data. This tool's approach can be reviewed and incorporated into an algorithm for root cause analysis.
The classification component of Machine Learning was emphasized since supervised learning anticipated the output based on specified qualities as a label applied to newly presented input data [14]. Moreover, Linear Regression was a subtype of Supervised Learning. The subcategory of regression was a predictive statistical technique for determining the relationship between dependent and independent variables. Linear regression expected a simple outcome, which means that if it does not hold in one example, it will hold in another. In this prediction's outcome, there is no third possibility. Both input and output variables are required to be linear for the linear regression to be able to determine the relationship between them with precision [129].
Given an example that when the humidity increased in a place with a tropical climate, the air held much more water, which eventually led to a high risk of precipitation [15]. However, it is difficult to create a linear relationship in the context of a software application operating in a multi-tier system. This is so that end users can access the software application through the Internet. When a Web browser displays a failure page, this is not always indicative of an unsuccessful software application: (i) The Web server's server resources may be depleting; (ii) The messaging queue (MQ) is full and no additional transactions can be processed in the backend; (iii) The backend agents are not responding to requests sent from the Web container;
and (iv) The database's network connectivity is intermittent. Even though linear regression may not be optimal for decision-making in the context of a software application working in a multi-tier system, supervised learning is still an effective method for identifying valid errors based on the labels provided.
7. Analytic Hierarchy Process
Despite the use of machine learning to forecast software application errors was offered [12] as a technique for preventing software application failures. The decision-making is more toward prediction and the focus was fully on detecting the error within the software application boundary. On the other hand, the Analytic Hierarchy Process (AHP) is particularly helpful when you are deciding with no clear best choice. There may be several criteria involved in the decision-making process. AHP was developed by Thomas L. Saaty stated in R. W. Saaty
(1987). The AHP combines math and psychology to compare numerous possibilities and select the best one. It accomplishes this by employing a notion known as pairwise comparisons. The procedure of comparing two criteria concurrently. Instead of assessing multiple criteria simultaneously, only two are evaluated at a time. This makes the decision easier to accomplish.
AHP may be utilized to evaluate software application defects and those connected with a software application failure. When the root cause of a software application error is identified, utilizing AHP to examine feasible options helps facilitate the organized resolution of the software application error. Indeed, supervised learning’s linear regression even though it may not be optimal for decision-making in the context of a software application functioning in a multi-tier system, nonetheless, is still an effective way of recognizing genuine errors based on the labels provided. Therefore, PALM gives the ability for error detection through supervised learning, and the decision-making process is by the AHP.
8. PALM
The algorithm of PALM has three important features to make the logic model unique.
• It incorporates supervised learning (SL) to facilitate keyword searching on error detection in the log file of software applications.
• It incorporates AHP to identify the highest priority error found in the involved log files.
• It incorporates AHP to decide the preferred resolution for the identified error.
The complete algorithm of PALM is shown as follows: -
Table 1: The Algorithm of PALM No. AHP SL Algorithm of PALM
1 With the granted, the “time” field of the first error event found in the software application log file will be utilized as the linkage to retrieve log errors across all involved log files based on this defined time. These log events are referred to as log data.
2 Identify whether the newly reported software application error is a first-time occurrence or re-occurrence by cross-checking the database which is associated with the PALM.
3 Identify possible log data and select the necessary log data for analysis under the defined software application error classification.
4 * Allocate weight to each possible software application error based on the Analytic hierarchy process (AHP).
5 * Shortlist the software application error under the highest weight.
6 Analyze the selected log data for shortlisted software application errors and define possible resolution options.
7 * Allocate weight to each possible resolution option based on AHP.
8 * Shortlist the preferred resolution option under the highest weight.
9 Deploy the preferred resolution option to fix the software application error under the predefined condition.
10 Store the analysis result and resolution action into a database that is associated with the PALM for future reference and knowledge base activities.
By referring to the algorithm of PALM that is numbered 4, 5, 7, and 8 (with the Note *) in the above table. These activities in the algorithm play an important role in the AHP process. The activity details are explained respectively as follows: -
• Activity number 4 helps to identify the possible software application errors and to filter out all the false alarms. On those possible software application errors, each error will be further identified by its error characteristic and categorized into a specific error category under a predefined software application error category list.
• Activity number 5 will perform assigning the weight to each possible software
application error based on the error category and the impact level of each error within the category. With the assigned weight on each error, it would easily shortlist the highest priority of software application error as the crucial error to be fixed.
• Activity number 7 handles the process of assigning weight to each possible resolution after the analysis activities have been conducted in section 6. This is because possible circumstances can happen when two similar resolutions are selected, but we need to identify which is the most suitable resolution that can be applied to resolve the software application error.
• Activity number 8 helps to identify the final preferred resolution to be applied for the crucial error after evaluating the weight, and this action will isolate multiple
resolution actions to be applied to the crucial error to be fixed.
With the AHP process, the PALM under the complex analysis area will conduct the important actions on filtering errors and identify which is the crucial error to be fixed. It also helps to identify which is the preferred resolution action to be applied on the crucial error to improve effectiveness in resolving the root cause.
9. Reduction of Human Mistakes
The more humans involved, the more human errors are possible. It is usual for each of us to commit inadvertent human errors [17]. As a result, a mature PALM is here to limit human error by minimizing human involvement in root cause analysis activities. PALM relies heavily on learning cycles throughout the training phase. Following incremental training, a mature PALM is deployed into a live (actual) environment. PALM's objective is to search and extract legitimate errors from the associated log files after filtering out alerts and warnings. This action is taken to avoid misunderstandings (concerning alerts and warnings) and to avoid wasting human work manually identifying each fault discovered in the associated log files. PALM can assist in determining the relationship between the mistakes and propose the preferred course of action for resolving the issues based on the obtained valid errors. This significantly reduces the amount of time spent by humans performing root cause analysis manually and improves the accuracy of recognizing real errors in the log files involved.
The debate falls on the immature PALM during the training phase. This is because PALM will not understand what are valid errors, and what alerts and warnings can be safely ignored.
Furthermore, PALM does not even know what suggestion to be provided if a valid error is identified. Therefore, during the training phase, PALM is still required high human involvement to provide feedback as the education to ensure the PALM can understand what it should. Indeed, no logic model can have any intelligence without any learning cycle. The same goes for humans, the intelligence does not come along from the time of birth.
The training phase was conducted with the following arrangement. There are 7 tests in total.
In these 7 tests, there are 5 tests with customized log files and 2 tests with real log files (without any customization). The meaning of customization here is to trim down the log file size on each involved log file manually on each test. Although this is time-consuming, it is worth ensuring a good comparison between manual error detection and PALM error detection. Please note that for all 7 tests, the number of errors across all the involved log files is manually checked before any PALM execution.
Table 2: The Average & Percentage of Valid Errors Detection & Identification Error Detected
No.# Log Files Total Error Manually PALM
% Manually
% PALM
1 Customized 22 22 22 100 100
2 Customized 30 28 30 93 100
3 Customized 26 23 26 88 100
4 Customized 27 25 27 93 100
5 Customized 28 25 28 89 100
6 Actual 32 27 32 84 100
7 Actual 35 26 35 74 100
Average 25 29 89 100
Each of the seven test results demonstrates how easy it is for humans to ignore a valid error after spending hours examining the log events line by line. Indeed, completing the reading action on each log file is time expensive and requires a longer length. As a result of the seven test results, the size of the log files is increasing, but the accuracy of manually recognizing valid errors is dropping. This is because, on average, human error detection and identification can discover and identify just 25 legitimate errors. Whereas, on average, PALM is capable of detecting and identifying all 29 genuine errors.
10. Ease of Decision-Making
AHP is a simple method for allocating weight to each attribute. Please keep in mind that the attributes correspond to the environment's software applications. The weight assigned to each attribute is either 1 (meaning more significant than) or 0. (means less important than). The evaluation will begin with the left row vs the vertical column, in the left-to-right direction.
Additionally, the AHP table is referred to as the Pairwise Comparison Chart. The "Total"
column will display the final priority of each attribute based on the Pairwise Comparison. Thus, by referencing the number in the AHP table's "Total" column, the ranking of the attributes represents the significance of each error and its priority order.
There are two critical reasons why AHP should be used as the decision-making procedure for PALM. The following detail is provided: -
i. The initial supporting reason – Simple to comprehend. When the AHP table contains all the associated software applications (as attributes), it enables humans to readily comprehend the Pairwise Comparison process (by comparing one attribute to another).
Whereas, without such an AHP table, it is difficult for a human to determine the priority of any software application involved. Eventually, either bias or a wild guess will be used to determine the priority of the relevant software programs.
ii. The second justification is that it is simple to rework. AHP's flexibility enables humans to rework the Pairwise Comparison if a new characteristic is required in the AHP table
at a later time. It will not take long to determine the ultimate priority of each character in the "Total" column if the AHP process stages are rigorously followed.
After the PALM is developed based on the defined algorithm, the configuration file contains the value in the “Total” column of the AHP table. Any rework on Pairwise Comparison will only require updating either the value assigned to the existing attributes or adding additional attributes along with their value inside the configuration file. The PALM provides such convenience and flexibility to humans, part of the contribution is from the AHP process itself.
When a genuine error has two distinct solutions, the conventional approach is to either adjust the priority of each solution differently. If, however, both solutions have the same priority, the second AHP is activated to determine the preferred option. However, during the training period of PALM, it is difficult or even impossible to state categorically that such a situation occurred.
As a result, all genuine errors discovered in the specified live (actual) environment have a single resolution, which is stored in the configuration file.
11. Conclusion
PALM is a logic model comprised of an algorithm for determining the existence of a genuine error in a complicated environment involving several parties. Additionally, if the purpose is accurate error identification and decision-making for the chosen resolution, the same method can be customized to match other industries. PALM seeks to accomplish three critical objectives. The objectives are to minimize human error when reading through a large number of log files, to increase accuracy by pinning down the real root cause, and to minimize time spent by humans performing root cause analysis tasks. Indeed, PALM is the logic model that corresponds to the research issues, and it is necessary to demonstrate that the algorithm contained within PALM is possible.
When PALM detects an error during the log file scanning process, it refers to its configuration file to do a filtering exercise that allows it to securely ignore known alarms and warnings. As a result, PALM will focus exclusively on faults that are considered legitimate. PALM has two decision-making sections. The first decision-making area is determining which software program error should be fixed first, and the second is determining how the determined fault should be resolved. PALM makes decisions about valid errors and suggested resolutions using AHP. Priority is allocated to each participating software program before conducting the pairwise comparison under AHP. The rationale for using AHP in decision-making is straightforward. This is because AHP is meant to compile a metric table of all the relevant things (in this case, the software applications). When a comparison procedure is used to determine the highest priority among the participating software programs, this metric table can be comprehended by a human.
The design and development of the software application is a proof of concept (POC) for the PALM. Before PALM is mature, it must undergo training throughout the training period. This is because no intelligent system is capable of comprehending everything when it begins with zero. Thus, during the training phase, PALM must determine which errors are legitimate and which alarms and warnings may be safely ignored. The desired resolution is determined and applied for each valid mistake. Additionally, if more than one valid error is identified, PALM must comprehend the relationship between the valid errors. By analyzing the relationship between valid errors discovered during concurrent log scanning, it is possible to find the true root cause using the AHP decision process specified in the configuration file. When a genuine error is resolved, unless there are many resolutions, the AHP process is activated to choose the
ultimate resolution. During the training phase, all of this knowledge becomes intelligent for the PALM. As a result, the Logic Model only interacts with humans during the training phase before maturing. PALM's configuration files include the intelligence.
References
[1] Edward Mercer, 2015. How Technology Affects Business Operations.
OpposingViews.com. http://science.opposingviews.com/technology-affects-business- operations-1659.html (Accessed on 28th November 2015)
[2] SuccessFactors, 2015. Using Technology to Increase Your Business Productivity.
https://www.successfactors.com/en_us/lp/articles/using-technology-to-increase-your- business-productivity.html (Accessed on 28th November 2015)
[3] Alex Bloom, 2014. The Cost of Application Downtime is Priceless. StatusCast.
http://www.statuscast.com/cost-application-downtime-pricess/ (Accessed on 28th November 2015)
[4] Management Logic, 2012. Root-Cause Analysis. http://www.management-
logic.com/toolbox/sales/Root-Cause%20Analysis/Index.html (Accessed on 28th November 2015)
[5] Gunjan Khanna, Ignacio Laguna, Fahad A. Arshad and Saurabh Bagchi. 2007. Distributed Diagnosis of Failures in a Three-Tier E-Commerce System. 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007). DOI:
10.1109/SRDS.2007.16.
https://engineering.purdue.edu/dcsl/publications/papers/2007/arshad-Diagnosis.pdf [6] David B. Stewart, Ph.D. 2012. Troubleshooting real-time software issues using a logic
analyzer. InHand Electronics, Inc. http://www.embedded.com/design/debug-and-
optimization/4236800/Troubleshooting-real-time-software-issues-using-a-logic-analyzer (Accessed on 12th December 2015)
[7] Felix Salfner and Steffen Tschirpke. 2015. Error Log Processing for Accurate Failure Prediction. www.usenix.org.
https://www.usenix.org/legacy/event/wasl08/tech/full_papers/salfner/salfner_html/
(Accessed on 12th December 2015) [8] Jan Valdman. 2001. Log File Analysis.
https://www.kiv.zcu.cz/site/documents/verejne/vyzkum/publikace/technicke- zpravy/2001/tr-2001-04.pdf (Accessed on 18th January 2017)
[9] Stephen G. Eick, Michael C. Nelson, and Jeffery D. Schmidt. 1994. Graphical Analysis of Computer Log Files. Communications of the ACM.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.43.4832&rep=rep1&type=pdf (Accessed on 12th December 2015)
[10] Wendy W. Peng and Dolores R. 1993 Wallace. Software Error Analysis. NIST Special Publication. http://www.geocities.ws/itopsmat/SoftwareErrorAnalysis.pdf (Accessed on 12th December 2015)
[11] Unomaly. 2019. Root cause analysis: Using the right tools for the job.
https://unomaly.com/product/root-cause-analysis/ (Accessed on 21st February 2019) [12] Lukasz Cmielowski. 2017. Adoption of machine learning to software failure prediction.
IBM Watson Data. https://medium.com/ibm-data-science-experience/adoption-of- machine-learning-to-software-failure-prediction-e8d85ed0338f (Accessed on 2nd April 2019)
[13] Klass, Lina. 2018. Machine Learning - Definition and application examples. MM International. https://www.spotlightmetal.com/machine-learning--definition-and- application-examples-a-746226/?cmp=go-aw-art-trf-SLM_DSA-
20180820&gclid=Cj0KCQjwxYLoBRCxARIsAEf16-vBblTDXGp3-
EuK0AsxQu90ckOqGbTyXEvjQBWSA3D3MHB51TxDhqUaAlPDEALw_wcB (Accessed on 21st February 2020)
[14] Wilson, Aidan (2019). A Brief Introduction to Supervised Learning. Towards Data Science. https://towardsdatascience.com/a-brief-introduction-to-supervised-learning- 54a3e3932590 (Accessed on 21st February 2020)
[15] Aidan Milan (2019). What is high humidity and how does it affect the weather (and your hair)? Metro News. Associated Newspapers Limited.
https://metro.co.uk/2019/06/25/high-humidity-affect-weather-hair-10057932/ (Accessed on 24th February 2020)
[16] R. W. Saaty (1987). THE ANALYTIC HIERARCHY PROCESS-WHAT IT IS AND HOW IT IS USED. Pergamon Journals Ltd.
https://www.sciencedirect.com/science/article/pii/0270025587904738/pdf?md5=4050e2 84cabc0a9f733b31ca9c8ff63f&pid=1-s2.0-0270025587904738-main.pdf&_valck=1 [17] Graham Edkins. 2016. Human Factors, Human Error and the role of bad luck in Incident
Investigations. https://www.linkedin.com/pulse/human-factors-error-role-bad-luck- incident-graham-edkins
[18] Ginette Collazo. 2021. 5 of the Most Common Types of Human Error in the Workplace.
Human Error Solutions. https://humanerrorsolutions.com/5-of-the-most-common-types- of-human-error-in-the-workplace/
[19] Hoo Meng W., Amalathas S.S. (2019) A New Approach Towards Developing a Prescriptive Analytical Logic Model for Software Application Error Analysis. In:
Silhavy R., Silhavy P., Prokopova Z. (eds) Intelligent Systems Applications in Software Engineering. CoMeSySo 2019 2019. Advances in Intelligent Systems and Computing, vol 1046. Springer, Cham. https://doi.org/10.1007/978-3-030-30329-7_24
[20] Wong H.M., Amalathas S.S. (2020) The Error Analysis for Enterprise Software Application Using Analytic Hierarchy Process and Supervised Learning: A Hybrid Approach on Root Cause Analysis. In: Silhavy R., Silhavy P., Prokopova Z. (eds) Software Engineering Perspectives in Intelligent Systems. CoMeSySo 2020. Advances in Intelligent Systems and Computing, vol 1295. Springer, Cham.
https://doi.org/10.1007/978-3-030-63319-6_59
[21] Wong H.M., Amalathas S.S., Zitkova T. (2020) A Prescriptive Analytical Logic Model Incorporated with Analytic Hierarchy Process for Software Application Error Analysis.
In: Silhavy R. (eds) Intelligent Algorithms in Software Engineering. CSOC 2020.
Advances in Intelligent Systems and Computing, vol 1224. Springer, Cham.
https://doi.org/10.1007/978-3-030-51965-0_1
[22] Wong H.M., Amalathas S.S., Ray S.K. (2021) A Hybrid Approach and Implementation on Root Cause Analysis Logic Model for Enterprise Software Application. In: Silhavy R., Silhavy P., Prokopova Z. (eds) Data Science and Intelligent Systems. CoMeSySo 2021. Lecture Notes in Networks and Systems, vol 231. Springer, Cham.
https://doi.org/10.1007/978-3-030-90321-3_19