6.57 It can be said that a computer can be both ‘reliable’ (but not infallible) and yet perform functions without the authority or knowledge of the owner or software writer. This may be when the code happens to execute in a way, because of a strange or unforeseen conjunction of inputs, which neither the owner nor the writer had imagined. For instance, one Jonathan Moore designed and produced forged railway tickets that were accepted by ticket machines controlled by computers. It took a ticket inspector to notice subtle differences in the colour and material of the ticket, which led to his arrest and prosecution for forgery.1
1 Tom Pugh, ‘IT expert sentenced for rail ticket forgery’, The Independent (London, 2 October 2009).
6.58 It is important to understand that programmers are aware of the limitations, as famously articulated by Ken Thompson:
You can’t trust code that you did not totally create yourself. (Especially code from companies that employ people like me). No amount of source-level verification or scrutiny will protect you from using untrusted code.1
1 Ken Thompson, ‘Reflections on trusting trust’, Turing Award Lecture (1984) 27 Communications of the ACM 761; further references Donald MacKenzie, Mechanizing Proof Computing, Risk and Trust (MIT Press 2004) 299, fn 1.
6.59 These comments are decidedly relevant, given that Thompson demonstrated how to create a C program fragment that would introduce Trojan Horse code into another compiled C program by compromising the C compiler. Thomas Wadlow explained this process as follows:
For example, when compiling the program that accepts passwords for login, you could add code that would cause the [first] program to accept legitimate passwords or a special backdoor password known to the creator of the Trojan.
This is a common strategy even today and is often detectable through source- code analysis.
Thompson went one step further. Since the C compiler is written in the C programming language, he used a similar technique to apply a Trojan to the C compiler source itself. When the C compiler is compiled, the resulting binary
program could be used to compile other programs just as before; but when the program that accepts passwords for login is compiled with the new compiler from clean, uncompromised source code, the backdoor-password Trojan code is inserted into the binary, even though the original source code used was completely clean. Source-code analysis would not reveal the Trojan because it was lower in the tool chain than the login program.1
1 Thomas Wadlow, ‘Who must you trust?’ (2014) 12 acmqueue Security 2.
6.60 Just because a person is in physical control of a computer or shop cash till, it does not follow that he will be aware whether it is working ‘reliably’, ‘properly’,
‘consistently’, ‘correctly’ or ‘dependably’.2 As indicated above, even the writer of the software will not be in such a luxurious position. It follows that the following comment by Kerr LCJ was not correct:
In the modern world the presumption of equipment being properly constructed and operating correctly must be strong. It is a particularly strong presumption in the case of equipment within the control of the defendant who alone would know if there was evidence of incorrect operation or incorrect setting.3
1 Stephen Castell, ‘Letter to the Editor’ (1994) 10 Computer L & Secur Rep 158 pointed out that the observation by Lord Griffiths, at 387D, that a till was a ‘computer … of the simplest kind’ in R v Shephard [1993] AC 380 was, even at the time, an assumption that did not reflect the truth.
2 The use of the word ‘dependability’ is a global concept that subsumes attributes of reliability, availability, safety, integrity and maintainability, and ‘reliability’ provides for continuity of correct service: Algirdas Avižienis, Jean-Claude Laprie and others, ‘Basic concepts and taxonomy of dependable and secure computing’ (2004) 1 IEEE Transactions on Dependable & Secure Computing 11, 13.
3 Public Prosecution Service v McGowan [2008] NICA 13, [2009] N.I. 1, [20]; it is acknowledged that many standards in the safety critical community require some element of proof in the tools they use, such as evidence that the supplier tracks and corrects defects, for instance.
6.61 That software code is imperfect and remains so may be illustrated by the comments of an early pioneer in computing, the late Professor Sir Maurice V. Wilkes FRS FREng:1
By June 1949 people had begun to realize that it was not so easy to get a program right as had at one time appeared. I well remember when this realization first came on me with full force. The EDSAC was on the top floor of the building and the tape-punching and editing equipment one floor below on a gallery that ran round the room in which the differential analyzer was installed. I was trying to get working my first non-trivial program, which was one for the numerical integration of Airy’s differential equation. It was on one of my journeys between the EDSAC room and the punching equipment that ‘hesitating at the angles of the stairs’ the realization came over me with full force that a good part of the remainder of my life was going to be spent in finding errors in my own programs.
Turing had evidently realized this too, for he spoke at the conference on ‘checking a large routine’.
1 Maurice V Wilkes, Memories of a Computer Pioneer (MIT Press, 1985) 145.
6.62 This observation has been repeated many times since.1 Professor Lloyd has expressed the view that the received wisdom is ‘that all software contains defects’2 – although he does not explain whether ‘received wisdom’ is based on evidence from technicians. Programmer errors are caused by a mix of novelty (applying software to previously unsolved problems), and the difficulty of the tasks software is required to perform, magnitude and complexity.3 To address this problem, the approach of many of
the existing software safety standards is to define requirements for and put constraints on the software development and assurance processes.4 Theodore A. Linden observed in 1976 that:
It is more difficult to build a 50,000 line program than it is to write 1,000 programs that are each 50 lines long. This phenomenon leads to rapidly escalating costs for the development and maintenance of large software systems, and its leads to serious reliability problems due to the difficulty of adequately debugging and testing a large program.5
1 The reader might wish to begin with the following, which is only one of many articles by many eminent people: Les Hatton, ‘Characterising the diagnosis of software failure’ (2001) 18 IEEE Software 34.2 Ian J Lloyd, Information Technology Law (7th edn, Oxford University Press 2014) 482.
3 B Littlewood and L Strigini, ‘Software reliability and dependability: a roadmap’ in A Finkelstein (ed.), The Future of Software Engineering (New York: ACM Press 2000) 177–88.
4 John McDermid and Tim Kelly, ‘Software in safety critical systems: achievement and prediction’
(2006) 2 Nuclear Future 34.
5 Theodore A Linden, ‘Operating system structures to support security and reliable software’ (1976) 8 ACM Computing Surveys (CSUR) 418.
6.63 Using the taxonomy of the provision of services, Algirdas Avižienis and colleagues have defined a ‘correct service’ as one where the service implements the system function. Its failure is an event that occurs when the service does not do what the function provides. This deviation is described as an ‘error’. For instance, if the function when using an ATM is to dispense cash, and the ATM dispenses the correct amounts of cash, then there is a correct service, and the service is carried out in accordance with the function. If the amount of cash withdrawn from an ATM is greater or less than the amount keyed in, or no cash is provided, this is service failure that can be an error or fault. The authors go on to say:
Since a service is a sequence of the system’s external states, a service failure means that at least one (or more) external state of the system deviates from the correct service state. … In most cases, a fault first causes an error in the service state of a component that is a part of the internal state of the system and the external state is not immediately affected.
For this reason, the definition of an error is the part of the total state of the system that may lead to its subsequent service failure. It is important to note that many errors do not reach the system’s external state and cause a failure. A fault is active when it causes an error, otherwise it is dormant.1
1 Avižienis and others, ‘Basic concepts and taxonomy of dependable and secure computing’ 13; for additional discussions on this topic, see John Rushby, ‘Critical system properties: survey and taxonomy’
(1994) 43 Reliability Engineering and System Safety 189, and Donald MacKenzie, Mechanizing Proof Computing, Risk and Trust (MIT Press 2004) 337, fn. 16.
6.64 For instance, an ATM might provide a receipt that £100 has been withdrawn, but does not dispense the money. Given this set of facts, clearly a fault has occurred, because the sensors or the software code (or both) in the machine failed to detect the lack of movement of cash. The bank might provide a print-out of the machine’s internal functioning that shows the balance of cash held in the machine before the transaction, and again after it. This proves very little. In the New York case of Porter v Citibank, N.A.,1 a similar set of facts occurred. The customer used his card, but no money was dispensed. Employees of the bank testified that on average machines were
out of balance once or twice a week. From an evidence point of view, the information on the print-out is restricted to a single transaction. For the bank to prove that the machine actually dispensed £100 (and therefore the customer is lying), it is necessary for the bank to balance the ATM and report the results for the material time. The overall balance might indicate that it had gone down by £100. But the report might be inaccurate. This is because of a number of associated variables, such as (this is not an exhaustive list): the multiple layers of outsourcing, the fact that people cover up mistakes, and the fact that people rely on other people to be diligent in dual-control tasks. Equally, if the machine happens to overpay someone else by £100, the error will cancel out the previous error and the end result could not have been detected by human intervention either. Human cross checks may suggest that everything appears correct, but the system is failing repeatedly. A further reason for the machine to be in error is that a third party may have successfully inserted code to bypass the software in the machine, leaving the thief to recover the cash after the customer left the scene.2 1 123 Misc.2d 28, 472 N.Y.S.2d 582 (N.Y.City Civ.Ct. 1984).
2 Stephen Mason, ‘Debit cards, ATMs and negligence of the bank and customer’ (2012) 27 Butterworths Journal of International Banking and Financial Law 163; Maryke Silalahi Nuth,
‘Unauthorized use of bank cards with or without the PIN: a lost case for the customer?’ (2012) 9 Digital Evidence and Electronic Signature Law Review 95; Stephen Mason, ‘Electronic banking and how courts approach the evidence’ (2013) 29 Computer Law and Security Review 144.
6.65 For all these reasons, it is very hard to show that a computer is working
‘properly’, even for highly skilled professionals.1 Part of the problem is that computers fail in discontinuous ways, which is a characteristic of discrete complexity, unlike most mechanical devices.
1 There is a technique called code verification, where code functionalities are verified as mathematical properties. But this process is time-consuming and limited. I owe this observation to Professor Seng.