Software Systems Safety Ebook free download pdf pdf


Teks penuh


This Series presents the results of scientific meetings supported under the NATO Programme: Science for Peace and Security (SPS).

The NATO SPS Programme supports meetings in the following Key Priority areas: (1) Defence Against Terrorism; (2) Countering other Threats to Security and (3) NATO, Partner and Mediterranean Dialogue Country Priorities. The types of meeting supported are generally “Advanced Study Institutes” and “Advanced Research Workshops”. The NATO SPS Series collects together the results of these meetings. The meetings are co-organized by scientists from NATO countries and scientists from NATO’s “Partner” or “Mediterranean Dialogue” countries. The observations and recommendations made at the meetings, as well as the contents of the volumes in the Series, reflect those of participants and contributors only; they should not necessarily be regarded as reflecting NATO views or policy.

Advanced Study Institutes (ASI) are high-level tutorial courses to convey the latest developments in a subject to an advanced-level audience.

Advanced Research Workshops (ARW) are expert meetings where an intense but informal exchange of views at the frontiers of a subject aims at identifying directions for future action. Following a transformation of the programme in 2006 the Series has been named and re-organised. Recent volumes on topics not related to security, which result from meetings supported under the programme earlier, may be found in the NATO Science Series.

The Series is published by IOS Press, Amsterdam, and Springer Science and Business Media, Dordrecht, in conjunction with the NATO Emerging Security Challenges Division.


A. Chemistry and Biology Springer Science and Business Media B. Physics and Biophysics Springer Science and Business Media C. Environmental Security Springer Science and Business Media D. Information and Communication Security IOS Press

E. Human and Societal Dynamics IOS Press

Sub-Series D: Information and Communication Security – Vol. 36 ISSN 1874-6268 (print)


Software Systems Safety

Edited by

Orna Grumberg

Computer Science Department, Technion, Haifa, Israel

Helmut Seidl

Technische Universität München, Germany


Maximilian Irlbeck

Technische Universität München, Germany


Marktoberdorf, Germany 30 July - 11 August 2013

© 2014 The authors and IOS Press.

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without prior written permission from the publisher.

ISBN 978-1-61499-384-1 (print) ISBN 978-1-61499-385-8 (online)

Library of Congress Control Number: 2014 936 390

Publisher IOS Press BV Nieuwe Hemweg 6B 1013 BG Amsterdam Netherlands

fax: +31 20 687 0019 e-mail:

Distributor in the USA and Canada IOS Press, Inc.

4502 Rachael Manor Drive Fairfax, VA 22032 USA

fax: +1 703 323 3668





Only 20 years ago the correctness and security of software systems was a largely theoretical problem, relevant only for a small group of computer science specialists. To-day it has become a fundamental problem of society at large: software bugs in programs used by hundred million people, security breaches in banking software, and malware attacks compromising millions of computers make it to the headlines almost daily. The computer science community is attacking the problem by developing verification and synthesis tools that mechanize more and more tasks in the design of correct and secure programs. The School provided courses about verification and synthesis tools, methodo-logies (testing, model-checking, synthesis and information-flow analysis), and adaptati-on and tailoring to applicatiadaptati-on domains.

The lecturers of the NATO Advanced Study Institute “Summer School Marktober-dorf 2013 – Software Systems Safety“ represented research groups from industry as well as universities and showed a detailed and excellent overview on current research results with special emphasis to solve such problems in software systems security.

Software Model Checking via Systematic Testingwas the title of PATRICE GO-DEFROID’s lecture. Over the last two decades, significant progress has been made on how to broaden the scope of model checking from finite-state abstractions to actual soft-ware implementations. Adapting model checking into a form of systematic testing that is applicable to industrial-size software might be a solution. His lectures presented a com-prehensive overview of this strand of software model checking, by describing the main ideas, techniques and results obtained in this area, including combinations with static program analysis.

ORNA GRUMBERG lectured on SAT-Based Model Checking: Interpolation, IC3 and Beyond. Model checking is an automatic approach to formally verifying that a given system satisfies a given specification. In spite of its great success in verifying hardware and software systems, the applicability of model checking is impeded by its high space and time requirements. We surveyed several approaches to enhancing SAT-based model checking. All these approaches result in efficient and complete SAT-SAT-based verification algorithms.

SUMIT GULWANI explained thatProgram Synthesisis the task of searching for a program in some underlying domain-specific language that matches the user’s intent. His lectures described three key dimensions in program synthesis. The first one was un-derlying domain-specific language in which programs are synthesized. Second he talked about user intent specification mechanism and interaction model to resolve ambiguities in under-specification. The third topic were search techniques. These dimensions were il-lustrated via a variety of applications in algorithm discovery (e.g., Bitvector/SIMD/graph algorithms, program inverses) and end-user programming (e.g., Spreadsheet programs, and smartphone scripts). Some surprising applications in the area of computer-aided edu-cation were presented (including problem synthesis, solution synthesis, feedback syn-thesis, and content authoring for a variety of subject domains including math, logic, and programming).

IOS Press, 2014


Interactive Proof: Applications to Data Flow Analysis and Securitywas GER-WIN KLEIN’s lecture which was built on the previous introduction to interactive proof. He demonstrated a specific application of interactive proof assistants: the semantics of programming languages. In particular, he taught how to formalize a small imperative programming language in the theorem prover Isabelle/HOL and how to define its seman-tics. We covered a basic type system with type safety proof, a more complex security sys-tem also with soundness proof, and different kinds of data flow analysis, such as liveness analysis.

RALF KÜSTERS’ lecture was onE-Voting Systems. Systems for electronic vo-ting (e-vovo-ting), including systems for vovo-ting in a vovo-ting booth, are employed in many countries and numerous problems with e-voting systems have been reported. Most of the systems used in practice today do not provide a sufficient level of security. Programming errors and malicious behaviour easily go undetected. Ralf covered central security requi-rements of e-voting systems, and how they can be formally defined and analysed. While analysis is mostly done based on cryptographic models or even more abstract so-called Dolev-Yao models, he also discussed approaches to perform (cryptographic) analysis directly on the implementation-/language-level of a system.

The lecture of MARTA KWIATKOWSKA held was titled Probabilistic Model Checking for Biology. Probabilistic model checking is an automated method for verify-ing the correctness and performance of probabilistic models. Property specifications are expressed in probabilistic extensions of temporal logic, and the probabilistic model che-cker can compute the probability of a given event, of its occurrence within a given time interval, or expected cost reward. Discrete- and continuous-time Markov chains were in-troduced, as well as the logics PCTL and CSL, and the corresponding model checking algorithms. An application of probabilistic model checking to two biological examples was shown: molecular signalling pathways and DNA computation. Marta demonstrated how design errors can be automatically detected in DNA gate designs using probabilistic model checking, analogously to model checking for digital circuits.

In the set of lectures onInfinite-state Model Checkingby RUPAK MAJUMDAR looked at the problem of automated analysis of infinite-state systems. Infinite-state sys-tems arise in many program verification problems, due to many different features in the models. For example, systems may contain unbounded data values (integers), data struc-tures (lists, arrays, etch.), real-valued clocks or other physical variables, unbounded exe-cution stacks, or unboundedly many threads of exeexe-cution. The purpose of these lectures was to understand how techniques from finite-state model checking can be extended to the infinite-state case. He showed both heuristic techniques (for special cases that are de-cidable). The first part dealt with abstraction of systems and counterexample-guided refi-nement. He showed how states can be represented as logical constraints and how decisi-on procedures for logical theories can be used to compute finite-state abstractidecisi-ons of sys-tems. He also presented a model checking algorithm based on abstraction and refinement. The second part focused on decidability results for infinite-state systems. He showed a general decidability result for well-structured transition systems using Petri nets. Also an algorithm to verify recursive programs with finite data domains was taught. Finally, he considered asynchronous programs, a programming model including both recursion and unbounded task creation for which safety and liveness properties remain decidable.


formula can ever evaluate to true. Several relatively recent research developments have enabled us to tackle formulas with millions of variables and constraints enabling SAT solvers to be effectively deployed in practical applications including in the analysis and verification of software systems. In the first part of this series of lectures, he covered techniques used in modern SAT solvers. In the second part, he considered extension of these solvers that have proved to be useful in analysis and verification. He showed how these concepts are related, present algorithms to derive them and show their application in design debugging.

The courseInteractive Proof: Hands-on Introductionby TOBIAS NIPKOW in-troduced interactive theorem proving with the Isabelle/HOL system in the following 3 steps. The first one was verified functional programming. The logic HOL contains an ML-style functional programming language. It was shown how to verify functional pro-grams in this language by induction and simplification. The second topic was predicate logic. Formulas of predicate logic and set theory were introduced together with induc-tively defined predicates. In a third step we talked about structured proofs. The proof language Isar was introduced and it was shown how to write structured proofs that are readable by both the machine and the human. The course assumed basic familiarity with some functional programming language of the ML or Haskell family, in particular with recursive data types and pattern matching. No specific background in logics is necessary beyond the ability to read predicate logic formulas.

DAVID SANDS held a lecture onSoftware Security by Information Flow Con-trol. A long-term goal is to construct innovative design methods for the construction of secure systems that put security requirements at the heart of the construction process, namely security by design. But to do this we had to understand how to unambiguously formulate the policy aims for secure systems and to develop a technology to integrate these goals into design mechanisms and technologies that enables an efficient construc-tion or verificaconstruc-tion of systems with the respect to those policies. He managed to better understand the semantics of information flow, explore dynamic and state-dependent po-licies, and to see how these ideas can be incorporated into a programming language with the support for static verification of a rich variety of information flow policies.

HELMUT SEIDL talked about A General Infrastructure for Interprocedural Analysis of Concurrent C. His tutorial was about infrastructures for general-purpose interprocedural analyses. It consisted of two parts. At first he argued that side-effecting constraint systems might serve as kind of a Swiss army knife for specifying analyses, while secondly he provided an overview on solving techniques for such systems. He re-ported on techniques for solving side-effecting constraint systems. One major issue he-re was that non-trivial analysis problems he-requihe-re complete lattices with infinite ascen-ding and descenascen-ding chains. In order to compute reasonably precise post-fixpoints of the resulting systems of equations, Cousot and Cousot have suggested accelerated fixpoint iteration by means of widening and narrowing.

Practical Program Analysis and Synthesiswas the lecture by ERAN YAHAV. For


abili-ty to hide some aspects of concurrency in the library, even application-level program-mers might still need to reason about atomicity. Eran taught a framework for synthesi-zing efficient synchronization in concurrent programs, a task known to be difficult and error-prone when done manually. Given a program, a specification, and an abstraction, we inferred synchronization that avoids all (abstract) interleaving that may violate the specification, but permits as many valid interleaving as possible.

Over the years, the Marktoberdorf Summer School has facilitated its scientifically profitable and well-known tradition and will follow future challenges in formal compu-ter science to ensure cyber security. The Summer School provides two weeks of chal-lenging and fruitful learning, discussion and development of new ideas, and should be a productive and beneficial event, at both the professional and the social level.

From the very first, the ASI "Marktoberdorf"was planned as a periodical forum with leading researchers to teach, learn and discuss the state-of-the-art research with the target to build a special international community. Out series of ASIs has been proved as a high-level scientific nucleus for international scientific networking and is settled as one of the most famous InternationalComputer ScienceSummer Schools – shown in an impressive way by the high number of applications every year.

We thank all lecturers, the staff, and hosts in Marktoberdorf. Special thanks must be addressed to Dr. Katharina Spies and Silke Müller for their great support.

The Marktoberdorf Summer School was arranged as an Advanced Study Institute of the NATO Science for Peace and Security Programmewith support fromDeutscher Akademischer Austausch Dienst (DAAD). We thank all authorities involved.



Preface v

May/Must Abstraction-Based Software Model Checking for Sound

Verification and Falsification 1

Patrice Godefroid

SAT-based Model Checking: Interpolation, IC3, and Beyond 17

Orna Grumberg, Sharon Shoham and Yakir Vizel

Program Synthesis 43

Sumit Gulwani

Applications of Interactive Proof to Data Flow Analysis and Security 77

Gerwin Klein and Tobias Nipkow

E-Voting Systems 135

Ralf Küsters

Probabilistic Model Checking for Biology 165

Marta Kwiatkowska and Chris Thachuk

Foundations of Infinite-State Verification 191

Rupak Majumdar

Boolean Satisfiability: Solvers and Extensions 223

Georg Weissenbacher, Pramod Subramanyan and Sharad Malik

Programming in Paragon 279

Bart van Delft, Niklas Broberg and David Sands

Frameworks for Interprocedural Analysis of Concurrent Programs 309

Helmut Seidl, Kalmer Apinis and Vesal Vojdani

Practical Software Synthesis 349

Eran Yahav

Subject Index 379


May/Must Abstraction-Based

Software Model Checking

for Sound Verification and Falsification


Microsoft Research

Abstract.Three-valued models, in which properties of a system are either true, false or unknown, have recently been advocated as a better representation for reac-tive program abstractions generated by automatic techniques such as predicate ab-straction. Indeed, for the same cost, model checking three-valued abstractions, also calledmay/must abstractions, can be used to both prove and disprove any temporal-logic property, whereas traditional conservative abstractions can only prove uni-versal properties. Also, verification results can be more precise withgeneralized model checking, which checks whether there exists a concretization of an abstrac-tion satisfying a temporal-logic formula. Generalized model checking generalizes both model checking (when the model is complete) and satisfiability (when every-thing in the model is unknown), probably the two most studied problems related to temporal logic and verification.

This paper presents an introduction to the main ideas behind this framework, namely models for three-valued abstractions, completeness preorders to measure the level of completeness of such models, three-valued temporal logics and gen-eralized model checking. It also discusses algorithms and complexity bounds for three-valued model checking and generalized model-checking for various tempo-ral logics. Finally, it discusses applications to program verification via automatic abstraction.

Keywords.Software Model Checking, Automatic Abstraction, Verification, Bug Finding

1. Introduction

How to broaden the scope of model checking to software is currently one of the most challenging problems related to computer-aided verification. Essentially two approaches have been proposed and are still actively being investigated. The first approach consists of adapting model checking into a form of systematic testing that simulates the effect of model checking while being applicable to operating-system processes executing arbitrary code [17,23]; although counter-examples reported with this approach are sound, it is inherently incomplete for large systems. The second approach consists of automatically extracting a model out of a software application by statically analyzing its code, and then of analyzing this model using traditional model-checking algorithms (e.g., [4,8,49,41, 29]); although automatic abstraction may be able to prove correctness, counter-examples are generally unsound since abstraction usually introduces unrealistic behaviors that may yield to spurious errors being reported when analyzing the model.

IOS Press, 2014


In this paper, we present an overview of a series of articles [5,6,20,21,22,18,14,19, 25] discussing how automatic abstraction can be performed to verify arbitrary formu-las of the propositionalµ-calculus [35] in such a way that both correctness proofs and counter-examples are guaranteed to be sound.

The key to make this possible is to represent abstract systems using richer models that distinguish properties that aretrue,falseandunknownof the concrete system. Ex-amples of such richer modeling formalisms are partial Kripke structures [5] and Modal Transition Systems [36,20]. Reasoning about such systems requires 3-valued temporal logics [5], i.e., temporal logics whose formulas may evaluate totrue,falseor⊥ (“un-known”) on a given model. Then, by using an automatic abstraction process that gener-ates by construction an abstract model which is less complete than the concrete system with respect to a completeness preorder logically characterized by 3-valued temporal logic, every temporal property that evaluates totrue(resp.false) on the abstract model automatically holds (resp. does not hold) of the concrete system, hence guaranteeing soundness of both proofs and counter-examples. In case a property evaluates to⊥on the model, a more complete (i.e., less abstract) model is then necessary to provide a defi-nite answer concerning this property of the concrete system. This approach is applica-ble to check arbitrary formulas of the propositionalµ-calculus (thus including negation and arbitrarily nested path quantifiers), not just universal properties as with a traditional “conservative” abstraction that merely simulates the concrete system.

2. Three-Valued Modeling Formalisms

Examples of 3-valued modeling formalisms for representing partially defined systems are partial Kripke structures (PKS) [5], Modal Transition Systems (MTS) [36,20] or

Kripke Modal Transition Systems(KMTS) [30].

Definition 1 AKMTSM is a tuple(S,P,−→must,−→may,L), where S is a nonempty finite set of states, P is a finite set of atomic propositions,−→⊆may S×S and−→⊆must S×S are transition relations such that−→⊆must −→may, and L:S×P→ {true,⊥,false}is aninterpretationthat associates a truth value in{true,⊥,false} with each atomic proposition in P for each state in S. An MTS is a KMTS where P=/0. A PKS is a KMTS where−→must=−→may.

The third value⊥(read “unknown”) andmay-transitions that are notmust-transitions are used to model explicitly a loss of information due to abstraction concerning, respectively, state or transition properties of the concrete system being modeled. A standard,complete

Kripke structure is a special case of KMTS where−→must=−→may andL:S×P→ {true,false}, i.e., no proposition takes value⊥in any state.

It can be shown [22] that PKSs, MTSs, KMTSs and variants of KMTSs where tran-sitions are labeled and/or two interpretation functionsLmayandLmust are used [30], are


3. Three-Valued Temporal Logics

When evaluating a temporal-logic formula on a 3-valued model, there are three possible outcomes: the formula can evaluate totrue,falseor⊥(unknown). Formally, we define 3-valued (temporal) logics as follows.

In interpreting propositional operators on KMTSs, we use Kleene’s strong 3-valued propositional logic [34], which generalizes the standard 2-valued semantics. Conjunction ∧ in this logic is defined as the function that returnstrueif both of its arguments are

true,falseif either argument isfalse, and⊥otherwise. We define negation¬using the function ‘comp’ that mapstruetofalse,falsetotrue, and⊥to⊥. Disjunction∨is defined as usual using De Morgan’s laws:pq=¬(¬p∧ ¬q). Note that these functions give the usual meaning of the propositional operators when applied to valuestrueandfalse.

Propositional modal logic (PML) is propositional logic extended with the modal operatorAX (which is read “for all immediate successors”). Formulas of PML have the following abstract syntax: φ ::=p| ¬φ |φ1∧φ2|AXφ, where p ranges overP. The following 3-valued semantics generalizes the traditional 2-valued semantics for PML.

Definition 2 The value of a formulaφ of 3-valued PML in a state s of a KMTS M=

This 3-valued logic can be used to define a preorder on KMTSs that reflects their degree of completeness. Let≤be the information orderingon truth values, in which ⊥ ≤true,⊥ ≤false,xx(for allx∈ {true,⊥,false}), andx6≤yotherwise. Note that the operators comp, min and max are monotonic with respect to the information order-ing≤: ifxx′andyy′, we have comp(x)≤comp(x′), min(x,y)≤min(x′,y′), and KMTSs. The completeness preorderis the greatest relationB SA×SC such that (sa,sc)∈Bimplies the following:

This definition allows to abstractMC byMAby letting truth values of propositions


com-plete, thanMC. The inverse of the completeness preorder is also calledrefinement pre-orderin [36,30,20]. Note that relationBreduces to a simulation relation when applied to MTSs with may-transitions only. Also note that relationBreduces to bisimulation when applied to MTSs withmust-transitions only and where all atomic propositions in

Pare eithertrueorfalse.

It can be shown that 3-valued PML logically characterizes the completeness pre-order [5,30,20].

Theorem 4 [5] Let MA= (SA,P, must

−→A, may

−→A,LA)and MC= (SC,P, must

−→C, may

−→C,LC)be KMTSs such that saSAand scSC, and letΦbe the set of all formulas of 3-valued PML. Then,


In other words, KMTSs that are “more complete” with respect tohave more def-inite properties with respect to≤, i.e., have more properties that are eithertrueorfalse. Moreover, any formulaφ of 3-valued PML that evaluates totrueorfalseon a KMTS has the same truth value when evaluated on any more complete structure. This result also holds for PML extended with fixpoint operators, i.e., the propositionalµ-calculus [5].

The following theorem states that 3-valued propositional modal logic logically char-acterizes the equivalence relation induced by the completeness preorder.

Theorem 5 [5] Let M1= (S1,P,


−→1,−→1may ,L1) and M2= (S2,P,


−→2,−→2may ,L2) be

KMTSs such that s1∈S1and s2∈S2, and letΦdenote the set of all formulas of 3-valued

propositional modal logic. Then

(∀φ∈Φ:[(M1,s1)|=φ] = [(M2,s2)|=φ])iff(s1s2and s2s1).

Note that if two statess1ands2arebisimilar, denoteds1∼s2, this implies boths1

s2ands2s1. This means that 3-valued propositional modal logic cannot distinguish between bisimilar states.

However, the converse is not true:s1s2ands2s1does not implys1∼s2. This is illustrated by the example below. The existence of such an example proves that, in contrast with 2-valued propositional modal logic, 3-valued propositional modal logic is

nota logical characterization of bisimulation.

Example 6 [5] Here is an example of two non-bisimilar states that cannot be distin-guished by any formula of 3-valued propositional modal logic.


s1 s2 s3

(true, )⊥ ( , ) ( ,true) ( , )

s’1 s’3



(true,true) (true,true)


These two partial Kripke structures have two atomic propositions p andq, whose truth value is defined in each state as indicated in the figure by a pair of the form(p,q). We have the following relations:





We have thats0s′0ands′0s0, buts06∼s′0sinces1is not bisimilar to any state in the second partial Kripke structure.

4. Three-Valued Model Checking

Given a state s of a 3-valued modelM and a formula φ, how to compute the value


This is the 3-valued model checkingproblem. In [6], it is shown that computing

[(MA,s)|=φ]can be reduced to two traditional (2-valued) model-checking problems on

complete systems (such as Kripke structures or Labeled Transition Systems).

Theorem 7 [6] The model-checking problem for a 3-valued temporal logic can be re-duced to two model-checking problems for the corresponding 2-valued logic.

The reduction can be performed in 3 steps as follows.

Step 1.CompleteMinto two “extreme” complete Kripke structures, called the op-timisticMoandpessimisticMpcompletions, defined as follows:

• ExtendPtoP′such that, for everypPthere exists a ¯pP′such thatL(s,p) =

comp(L(s,p))¯ for allsinS.

Mo= (S,Lo, must



true ifL(s,p) =


Mp= (S,Lp,−→may)with Lp(s,p)def=

f alse ifL(s,p) =


Step 2.Transform the formulaφ to its positive formT(φ)by pushing negation in-wards using De Morgan’s laws, and replacing remaining negations¬pat the proposi-tional level by ¯(p).

Step 3.EvaluateT(φ)on MoandMpusing traditional 2-valued model checking,

and combine the results as follows:

[(M,s)|=φ] =   

true if(Mp,s)|=T(φ) f alseif(Mo,s)6|=T(φ)

⊥ otherwise

This can be done using existing model-checking tools! The formula istrueatsif it is


It can be proved [6] that the above procedure computes the correct value for

[(M,s)|=φ]according to the 3-valued semantics defined in the previous section. An immediate corollary from this result is that 3-valued model checking has the same (time and space) complexity as traditional 2-valued model checking. Indeed, the transformations ofMintoMoandMp, and ofφintoT(φ)can be done in linear time and

logarithmic space in the size ofMandφ, respectively.

Example 8 [5] Consider the three following partial Kripke structures with a single atomic propositionp, whose truth value is defined in each state as indicated in the figure.

⊥ p= ⊥

p= ⊥



s1 s3



p=false p=false

p=true p=false

The formulaA(trueU p)of 3-valued CTL is read “for all paths, doespeventually hold?”. It has a different truth value in each of the top states of these partial Kripke structures:[s1|=A(trueU p)] =true,[s2|=A(trueU p)] =⊥, and[s3|=A(trueU p)] =


5. Generalized Model Checking

However, as argued in [6], the semantics of [(M,s)|=φ]returns⊥more often than it should. Consider a KMTSMconsisting of a single statessuch that the value of proposi-tionpatsis⊥and the value ofqatsistrue. The formulasp∨ ¬pandq∧(p∨ ¬p)are ⊥ats, although in all complete Kripke structures more complete than(M,s)both for-mulas evaluate totrue. This problem is not confined to formulas containing subformulas that are tautological or unsatisfiable. Consider a KMTSM′with two statess0ands1such thatp=q=trueins0andp=q=falseins1, and with amay-transition froms0tos1. The formulaAX p∧ ¬AX q(which is neither a tautology nor unsatisfiable) is⊥ats0, yet in all complete structures more complete than(M′,s

0)the formula isfalse.

This observation is used in [6] to define an alternative 3-valued semantics for modal logics called thethoroughsemantics since it does more than the other semantics to dis-cover whether enough information is present in a KMTS to give a definite answer. Let thecompletionsC(M,s)of a statesof a KMTSMbe the set of all statessof complete Kripke structuresM′such thatss′.

Definition 9 Letφbe a formula of any two-valued logic for which a satisfaction relation

|=is defined on complete Kripke structures. The truth value ofφin a state s of a KMTS M under thethoroughinterpretation, written[(M,s)|=φ]t, is defined as follows:

[(M,s)|=φ]t=   

true if(M′,s)|=φ for all(M,s)inC(M,s) false if(M′,s)6|=φ for all(M,s)inC(M,s)


It is easy to see that, by definition, we always have[(M,s)|=φ]≤[(M,s)|=φ]t. In gen-eral, interpreting a formula according to the thorough three-valued semantics is equiva-lent to solving two instances of the generalized model-checking problem [6].

Definition 10 (Generalized Model-Checking Problem) Given a state s of a KMTS M and a formula φ of a (two-valued) temporal logic L, does there exist a state sof a complete Kripke structure Msuch that ssand(M′,s)|=φ?

This problem is called generalized model-checking since it generalizes both model checking and satisfiability checking. At one extreme, whereM= ({s0},P,−→must=−→may=

{(s0,s0)},L)withL(s0,p) =⊥for all pP, all complete Kripke structures are more complete thanMand the problem reduces to the satisfiability problem. At the other ex-treme, whereMis complete, only a single structure needs to be checked and the problem reduces to model checking.

Therefore, the worst-case complexity for the generalized model-checking problem will never be better than the worst-case complexities for the model-checking and satisfi-ability problems for the corresponding logic. The following theorem formally states that the generalized model-checking problem is at least as hard as the satisfiability problem.

Theorem 11 [6] Let L denote the propositional µ-calculus or any of its fragments (propositional logic, PML, LTL, CTL, CTL, etc.). Then the satisfiability problem for L is reducible (in linear-time and logarithmic space) to the generalized model-checking problem for L.

Is generalized model checking harder than satisfiability? It depends.

For branching-time temporal logics, it can be shown [6] that generalized model checking has the same complexity as satisfiability.

Theorem 12 [6] Let L denote propositional logic, PML, CTL, or any branching-time logic including CTL (such as CTLor the propositional µ-calculus). The generalized model-checking problem for the logic L has the same complexity as the satisfiability problem for L.

In contrast, forlinear-timetemporal logic (LTL), generalized model checking can be harder than satisfiability [25]. We have the following.

Theorem 13 [25] Given a state s0 of partial Kripke structure M= (S,L,R)and an

LTL formulaφ, one can construct an alternating parity word automaton A(M,s

0),φover a 1-letter alphabet with at most O(|S| ·22|φ|log(|φ|))states and2O(|φ|)priorities such that


Theorem 14 [25] The generalized model-checking problem for linear-time temporal logic is 2EXPTIME-complete.



Propositional Logic Linear NP-complete NP-complete

PML Linear PSPACE-complete PSPACE-complete

CTL Linear EXPTIME-complete EXPTIME-complete

µ-calculus NP∩co-NP EXPTIME-complete EXPTIME-complete

LTL PSPACE-complete PSPACE-complete 2EXPTIME-complete

Figure 1. Known results on the complexity in the size of the formula for (2-valued and 3-valued) model checking (MC), satisfiability (SAT) and generalized model checking (GMC).

that flavor include therealizability [1] andsynthesis [42,43] problems for linear-time temporal logic specifications.

Figure 1 summarizes the previous complexity results. These results show that the complexity in the size of the formula of computing[(M,s)|=φ]t(GMC) is always higher than that of computing[(M,s)|=φ](MC).

Regarding the complexity in the size of the model|M|, it is shown in [6] that gener-alized model checking for CTL can be solved in time quadratic in|M|. For LTL, gener-alized model checking can be solved in time polynomial in|M|[25]. More precisely, the complexity in|M|is

linearfor safety (✷p) and weak (i.e., recognizable by Deterministic Weak Word

automata) properties;

quadraticfor response (✷(p→✸q), persistence (✸ ✷p) and generalized

reactiv-ity[1] properties [32].

Note that for CTL and LTL, generalized model checking is PTIME-hard in |M|while model checking is NLOGSPACE-complete in|M|[18].

6. How to Generate 3-Valued Abstractions

In [20], it is shown how to adapt the abstraction mappings of [9] to construct abstractions that are less complete than a given concrete program with respect to the completeness preorder.

Definition 15 Let MC= (SC,P, must

−→C, may

−→C,LC)be a (concrete) KMTS. Given a set SA of abstract states and a total1abstraction relation on statesρ ⊆SC×SA, we define the (abstract) KMTS MA= (SA,P,−→mustA,


−→A,LA)as follows:a−→mustAaifcSC:cρa⇒(∃c′∈SC:c′ρa′∧c


−→Cc′); • a−→mayAaifc,c′∈SC:cρac′ρa′∧c



LA(a,p) =   

true ifc:cρaLC(c,p) =true false ifc:cρaLC(c,p) =false


The previous definition can be used to build abstract KMTSs.


Theorem 16 Given a KMTS MC, any KMTS MAobtained by applying Definition 15 is such that MAMC.

Given a KMTSMC, any abstractionMAless complete thanMCwith respect to the

com-pleteness preordercan be constructed using Definition 15 by choosing the inverse ofρ asB[20]. When applied to MTSs withmay-transitions only, the above definition coin-cides with traditional “conservative” abstraction that is asimulationof the concrete sys-tem. Building a 3-valued abstraction can be done using existing abstraction techniques at thesame computational costas building a conservative abstraction [20].

7. Application to Software Model Checking

The usual procedure for performing program verification via predicate abstraction and iterative abstraction refinement is the following (e.g., see [3,12]).

1. Abstract: compute an abstractionMAthat simulates the concrete programMC.

2. Check: given a universal propertyφ, decide whetherMA|=φ. • ifMA|=φ: stop (the property is proved:MC|=φ).

• ifMA6|=φ: go to Step 3.

3. Refine: refineMA (possibly using a counter-example found in Step 2). Then go

to Step 1.

Using predicate abstraction [26,13,50], the abstraction computed in Step 1 is de-fined with respect to a set of predicatesΨ={ψ1, . . . ,ψn}, which are typically quantifier-free formulas of first-order logic (for instance,(x==y+1)∨(x<y−5)). An abstract state is defined as a vector ofn bits induced by n-ary conjunctions, with each predi-cateψicontributing eitherψior¬ψi, which identifies all concrete states that satisfy the same set of predicates inΨ. Thus, a concrete statecis abstracted by an abstract state

[c] = (b1, . . . ,bn)such that∀1≤in:bi=ψi(c). A transition is defined between

ab-stract states[c1]and[c2]if there exists a transition from some concrete state in[c1]to some concrete state in[c2]. The resulting abstract transition systemMAis guaranteed by

construction to simulate the concrete transition systemMC.

SinceMAsimulatesMC, one can only prove the correctness of universal properties

(i.e., properties over all paths) ofMC by analyzingMA in Step 2. In particular, the

vi-olation of a universal property (or equivalently, the satisfaction of an existential prop-erty) cannot be established by analyzing such abstractions in general. Step 3 typically in-volves the addition of new predicates to refine the current abstraction. Note that the three steps above can also be interleaved and performed in a strict demand-driven fashion as described in [28].


1. Abstract: compute an abstractionMAusing Def. 15 such thatMAMC.

2. Check: givenanypropertyφ,

(a) (3-valued model checking) compute[MA|=φ].

• if[MA|=φ] =trueorfalse: stop (the property is proved (resp. disproved)


• if[MA|=φ] =⊥, continue.

(b) (generalized model checking) compute[MA|=φ]t.

• if[MA|=φ]t=trueorfalse: stop (the property is proved (resp. disproved)


• if[MA|=φ] =⊥, go to Step 3.

3. Refine: refineMA(possibly using a counter-example found in Step 2). Then go

to Step 1.

This new procedure strictly generalizes the traditional one in several ways. First, any temporal logic formula can be checked (not just universal properties). Second, all cor-rectness proofs and counter-examples obtained by analyzing any abstractionMA such

thatMAMC are guaranteed to be sound (i.e., hold onMC) for any property (by

The-orem 4). Third, verification results can be more precise than with the traditional proce-dure: the new procedure will not only returntruewhenever the traditional one returns

true(trivially, since the former includes the latter), but it can also returntruemore of-ten thanks to a more thorough check using generalized model-checking, and it can also returnfalse. The new procedure can thus terminate sooner and more often than the tra-ditional procedure — the new procedure will never loop through its 3 steps more often than the traditional one.

Remarkably, each of the 3 steps of the new procedure can be performed at roughly the same cost as the corresponding step of the traditional procedure: as shown in [20], building a 3-valued abstraction using Definition 15 (Step 1 of new procedure) can be done at the same computational cost as building a conservative abstraction (Step 1 of traditional procedure); computing[MA|=φ]in Step 2.a can be done at the same cost at

traditional (2-valued) model checking [6]; following the results of Section 5, computing

[MA|=φ]t in Step 2.b can be more expensive than Step 2.a, but is still polynomial

(typ-ically linear or quadratic) in the size ofMA; Step 3 of the new procedure is similar to

Step 3 of the traditional one (in the case of LTL properties for instance, refinement can be guided by error traces found in Step 2 as in the traditional procedure). Finally note that the new procedure could also be adapted so that the different steps are performed in a demand-driven basis following the work of [28].

8. Examples

We now give examples of programs, models and properties, all taken from [21], where computing[(M,s)|=φ]t returns a more precise answer than[(M,s)|=φ].


program C1(){ program C2(){ program C3(){

x,y = 1,0; x,y = 1,0; x = 1;

x,y = f(x),f(y); x,y = 2*f(x),f(y); x = f(x);

x,y = 1,0; x,y = 1,0; }

} }



M1 M3


⊥ (p=T) s3 (p=T,q=F)



(p= ) (p= ,q= )


(p=T,q=F) (p=T,q=F)

(p=F,q= )

Figure 2. Examples of programs and models

p: “is x odd?” andq: “is y odd?”. Figure 2 shows an example of KMTS model for each of the three programs. These models can be computed automatically using Definition 15, predicate abstraction techniques and predicates p and q, so that by construction they satisfy Theorem 16. Each model is a KMTS withmust-transitions only and with atomic propositionspandqwhose truth value is defined in each state as indicated in the figure. Consider the LTL formulaφ1=✸q⇒✷(p∨q)(where✸is read “eventually” and

✷is read “always” [38]). While[(M1,s1)|=φ1] =⊥,[(M1,s1)|=φ1]t =true. In other words, using the thorough interpretation yields a more definite answer in this case. Note that the gain in precision obtained in this case is somewhat similar to the gain in precision that can be obtained using an optimization calledfocusing[3] aimed at recovering some of the imprecision introduced when usingcartesian abstraction(see [3,20]).

Consider now the formulaφ2=✸q∧✷(p∨ ¬q)evaluated on(M2,s2). In this case, we have [(M2,s2)|=φ2] =⊥, while [(M2,s2)|=φ2]t =false. Again, using the thor-ough interpretation yields a more definite answer, alththor-ough solving a generalized model-checking problem is necessary to return a negative answer. Indeed, one needs to prove in this case that there exists a computation of(M2,s2)(namelys2s′2s′′2ω – there is only one computation in this simple example) that does not have any completion satisfying

φ2, which itself requires using alternating automata and can thus be more expensive as discussed in Section 5. Another example of formula isφ2′=q∧✷(p∨ ¬q)(whereis

read “next” [38]). Again we have that[(M2,s2)|=φ2′] =⊥, while[(M2,s2)|=φ2′]t=false. Note that, althoughφ2′is an LTL safety formula and hence is within the scope of analysis of existing tools ([4], [8], etc.), none of these tools can prove thatφ2′does not hold: this result can only be obtained using generalized model checking.

Last, consider(M3,s3)and formulaφ3=✷p. In this case, we have both[(M3,s3)|=


9. Precision of GMC Vs. MC

How often is generalized model checking (GMC) more precise than model checking (MC)? This question is addressed in [19]. Specifically, [19] studies when it is possible to reduce GMC(M,φ) to MC(M,φ′). Such a transformed formulaφ′is called asemantic minimizationofφ. [19] shows that propositional logic, PML and the propositional µ -calculus areclosed under semantic minimization, i.e., that a reduction from GMC(M,φ) to MC(M,φ′) is always possible for φ andφ′ in propositional logic, PML or the µ -calculus. But in contrast, the temporal logics LTL, CTL and CTL∗arenotclosed under semantic minimization.

[19] also identifiesself-minimizingformulas, i.e., formulasφfor which GMC(M,φ) and MC(M,φ) are equivalent. By definition, GMC and MC have thus the same precision for any self-minimizing formula. Self-minimizing formulas can be defined both seman-tically using automata-theoretic techniques (for instance, this is EXPTIME-hard in|φ| for theµ-calculus) and syntactically by providing syntacticsufficientcriteria which are linear in|φ|. For instance, [19] shows thatany formula that does not contain any atomic proposition in mixed polarity (in its negation normal form) is self-minimizing.

Fortunately, in practice, many frequent formulas are self-minimizing, and MC is as precise as GMC for those.

10. Other Related Work

The framework presented in the previous sections has also been extended toopen sys-tems [18] (i.e., systems which interacts with their environment), and togamesin gen-eral [14]. For instance, [14] studiesabstractions of gameswhere moves of each player can be abstracted while preserving winning strategies of bothplayers. An abstraction of a game is now a game where each player has both may and must moves, yielding may/must strategies. In this context, the completeness preorder becomes analternating refinementrelation, logically characterized by 3-valued alternatingµ-calculus [2].

Another interesting topic issemantic completeness: given an infinite-state systemC

and propertyφ, ifCsatisfiesφ, does there exist a finite-state abstractionAofCsuch that


For arbitrary formulasφ of LTL, the existence of such finite abstractionsAcan be guaranteed provided that abstractionsAare extended to include the modeling of fair-nessconstraints [33], which are used to model termination in loops. For arbitrary for-mulasφ of the propositionalµ-calculus (hence including existential properties), the ex-istence of such finite abstractions can again be guaranteed but now provided that ab-stractionsAmay includenondeterministicmust transitions [37], also calledhyper-must


11. Concluding Remarks

This paper presents an introduction to 3-valued “may/must” abstraction-based software model checking for sound property verification and falsification. The results presented here previously appeared in a series of papers [5,6,20,21,22,18,14,19,25]. These results shed light on the techniques used in abstraction-based software model checking tools like SLAM [4], BLAST [28], YASM [27], TERMINATOR [7] and YOGI [24]. In partic-ular, YASM [27] uses 3-valued models as described in this paper, while YOGI [24] uses (compositional) may/must abstractions that share transitions instead of states.

The reader interested in the topic of this paper should consult the references listed above, as well as the related work discussed in those references. We mention below only a few other main pointers to related work.

The study of abstraction for model checking of both universal and existential pro-gram properties was pioneered in [9,10]. This work defines a general abstraction frame-work where abstractions aremixed transition systems. Intuitively, a mixed transition sys-tem is a modal transition syssys-tem without the constraint−→⊆must −→. Mixed transition sys-may tems are more expressive and, in full generality, allow for a 4-valued world where some mixed transition systems cannot be refined by any complete (2-valued) system [31]. Nev-ertheless, the goal and some of the results of this prior work are very similar to our own work with 3-valued models and logics. The use of “conservative” abstractions for proving properties of the fullµ-calculus is also discussed in [45].

Extended transition systems [39] are Labeled Transition Systems extended with a

divergence predicate, and can be viewed as a particular class of 3-valued models [5,30]. In [5], it is shown that Hennessy-Milner Logic with a 3-valued interpretation provides an alternative characterization of the divergence preorder in addition to the intuitionistic interpretation of Plotkin [48]. Further work on divergence preorders and logics to char-acterize them can be found in [48,51]. In all this work, logic formulas are interpreted normally in the 2-valued sense. The close correspondence between 3-valued logic and Plotkin’s intuitionistic modal logic inspired the reduction procedure from 3-valued model checking to 2-valued model checking of [6] (see Section 4).

Prior to the work reported here, most work on 3-valued modal logic focused on its proof theory (e.g., [46,15]). Our definition of partial Kripke structure is closest to [16], where two interpretations of modal logic are presented: a many-valued version and another version based on obtaining 2-valued interpretations from each of a set of experts. [16] shows that such a multi-expert interpretation corresponds in a precise way to a multi-valued interpretation, similarly to how we show that a 3-valued interpretation can be obtained by separate optimistic and pessimistic interpretations. However, [16] does not define a completeness preorder over models or characterization results.


(transi-tion) systems and the sound verification (and falsifica(transi-tion) oftemporalproperties of such systems.

Acknowledgements.This paper covers one of the lectures (Lecture 4) which I gave at the 2013 Marktoberdorf Summer School. I thank the organizers of the Summer School for encouraging me to write this paper. I also thank my co-authors (in chronological order) Glenn Bruns, Radha Jagadeesan, Michael Huth, Luca de Alfaro, and Nir Piterman for their insights and without whom this work would not exist.


[1] M. Abadi, L. Lamport, and P. Wolper. Realizable and unrealizable concurrent program specifications. InProc. 16th Int. Colloquium on Automata, Languages and Programming, volume 372 ofLecture Notes in Computer Science, pages 1–17. Springer-Verlag, July 1989.

[2] R. Alur, T.A. Henzinger, and O. Kupferman. Alternating time temporal logic. Journal of the ACM, 49:672–713, 2002.

[3] T. Ball, A. Podelski, and S. K. Rajamani. Boolean and Cartesian Abstraction for Model Checking C Programs. InProceedings of TACAS’2001 (Tools and Algorithms for the Construction and Analysis of Systems), volume 2031 ofLecture Notes in Computer Science. Springer-Verlag, April 2001.

[4] T. Ball and S. Rajamani. The SLAM Toolkit. InProceedings of CAV’2001 (13th Conference on Com-puter Aided Verification), volume 2102 ofLecture Notes in Computer Science, pages 260–264, Paris, July 2001. Springer-Verlag.

[5] G. Bruns and P. Godefroid. Model Checking Partial State Spaces with 3-Valued Temporal Logics. In

Proceedings of CAV’99 (11th Conference on Computer Aided Verification), volume 1633 ofLecture Notes in Computer Science, pages 274–287, Trento, July 1999. Springer-Verlag.

[6] G. Bruns and P. Godefroid. Generalized Model Checking: Reasoning about Partial State Spaces. In

Proceedings of CONCUR’2000 (11th International Conference on Concurrency Theory), volume 1877 ofLecture Notes in Computer Science, pages 168–182, University Park, August 2000. Springer-Verlag. [7] B. Cook, A. Podelski, and A. Rybalchenko. Termination Proofs for Systems Code. InProceedings of PLDI’2006 (ACM SIGPLAN 2006 Conference on Programming Language Design and Implementation), pages 415–426, Ottawa, June 2006.

[8] J. C. Corbett, M. B. Dwyer, J. Hatcliff, S. Laubach, C. S. Pasareanu, Robby, and H. Zheng. Bandera: Extracting Finite-State Models from Java Source Code. InProceedings of the 22nd International Con-ference on Software Engineering, 2000.

[9] D. Dams.Abstract interpretation and partition refinement for model checking. PhD thesis, Technische Universiteit Eindhoven, The Netherlands, 1996.

[10] D. Dams, R. Gerth, and O. Grumberg. Abstract interpretation of reactive systems. ACM Transactions on Programming Languages and Systems, 19(2):253–291, 1997.

[11] D. Dams and K. Namjoshi. The existence of finite abstractions for branching time model checking. In

Proceedings of LICS’2004 (19th IEEE conference on Logic in Computer Science), Turku, July 2004. [12] S. Das and D. L. Dill. Successive Approximation of Abstract Transition Relations. InProceedings of

LICS’2001 (16th IEEE Symposium on Logic in Computer Science), pages 51–58, Boston, June 2001. [13] S. Das, D. L. Dill, and S. Park. Experience with Predicate Astraction. InProc. of the 11th

Interna-tional Conference on Computer-Aided Verification, Lecture Notes in Computer Science, pages 160–172, Trento, July 1999. Springer Verlag.

[14] L. de Alfaro, P. Godefroid, and R. Jagadeesan. Three-Valued Abstractions of Games: Uncertainty, but with Precision. InProceedings of LICS’2004 (19th IEEE Symposium on Logic in Computer Science), pages 170–179, Turku, July 2004.

[15] M. Fitting. Many-Valued Modal Logics I.Fundamenta Informaticae, 15:235–254, 1992. [16] M. Fitting. Many-Valued Modal Logics II.Fundamenta Informaticae, 17:55–73, 1992.


[19] P. Godefroid and M. Huth. Model Checking Vs. Generalized Model Checking: Semantic Minimizations for Temporal Logics. InProceedings of LICS’2005 (20th IEEE Symposium on Logic in Computer Science), pages 158–167, Chicago, June 2005.

[20] P. Godefroid, M. Huth, and R. Jagadeesan. Abstraction-based Model Checking using Modal Transition Systems. InProceedings of CONCUR’2001 (12th International Conference on Concurrency Theory), volume 2154 ofLecture Notes in Computer Science, pages 426–440, Aalborg, August 2001. Springer-Verlag.

[21] P. Godefroid and R. Jagadeesan. Automatic Abstraction Using Generalized Model Checking. In Pro-ceedings of CAV’2002 (14th Conference on Computer Aided Verification), volume 2404 ofLecture Notes in Computer Science, pages 137–150, Copenhagen, July 2002. Springer-Verlag.

[22] P. Godefroid and R. Jagadeesan. On the Expressiveness of 3-Valued Models. InProceedings of VM-CAI’2003 (4th Conference on Verification, Model Checking and Abstract Interpretation), volume 2575 ofLecture Notes in Computer Science, pages 206–222, New York, January 2003. Springer-Verlag. [23] P. Godefroid, N. Klarlund, and K. Sen. DART: Directed Automated Random Testing. InProceedings of

PLDI’2005 (ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation), pages 213–223, Chicago, June 2005.

[24] P. Godefroid, A.V. Nori, S.K. Rajamani, and S.D. Tetali. Compositional May-Must Program Analy-sis: Unleashing The Power of Alternation. InProceedings of POPL’2010 (37th ACM Symposium on Principles of Programming Languages), pages 43–55, Madrid, January 2010.

[25] P. Godefroid and N. Piterman. LTL Generalized Model Checking Revisited. International Journal on Software Tools for Technology Transfer (STTT), 13(6):571–584, 2011.

[26] S. Graf and H. Saidi. Construction of Abstract State Graphs with PVS. InProceedings of the 9th International Conference on Computer Aided Verification, volume 1254 ofLecture Notes in Computer Science, pages 72–83, Haifa, June 1997. Springer-Verlag.

[27] A. Gurfinkel, O. Wei, and M. Chechik. Yasm: A Software Model Checker for Verification and Refuta-tion. InProceedings of CAV’2006 (18th Conference on Computer Aided Verification), volume 4144 of

Lecture Notes in Computer Science, pages 170–174, Seattle, August 2006. Springer-Verlag.

[28] T. Henzinger, R. Jhala, R. Majumdar, and G. Sutre. Lazy Abstraction. InProceedings of the 29th ACM Symposium on Principles of Programming Languages, pages 58–70, Portland, January 2002. [29] G. J. Holzmann and M. H. Smith. A Practical Method for Verifying Event-Driven Software. In

Pro-ceedings of the 21st International Conference on Software Engineering, pages 597–607, 1999. [30] M. Huth, R. Jagadeesan, and D. Schmidt. Modal Transition Systems: a Foundation for Three-Valued

Program Analysis. InProceedings of the European Symposium on Programming (ESOP’2001), volume 2028 ofLecture Notes in Computer Science. Springer-Verlag, April 2001.

[31] M. Huth, R. Jagadeesan, and D. Schmidt. A Domain Equation for Refinement of Partial Systems. Submitted to Mathematical Structures in Computer Science, 2002.

[32] Y. Kesten, N. Piterman, and A. Pnueli. Bridging the gap between fair simulation and trace containment. In15th Computer Aided Verification, volume 2725 ofLecture Notes in Computer Science, pages 381– 393. Springer-Verlag, 2003.

[33] Y. Kesten and A. Pnueli. Verification by Augmented Finitary Abstraction. Information and Computa-tion, 163(1), 2000.

[34] S. C. Kleene.Introduction to Metamathematics. North Holland, 1987.

[35] D. Kozen. Results on the Propositional Mu-Calculus.Theoretical Computer Science, 27:333–354, 1983. [36] K. G. Larsen and B. Thomsen. A Modal Process Logic. InProceedings of Third Annual Symposium on

Logic in Computer Science, pages 203–210. IEEE Computer Society Press, 1988.

[37] K. G. Larsen and Liu Xinxin. Equation solving using modal transition systems. InProceedings of the 5th IEEE conference on Logic in Computer Science, pages 108–117. IEEE, 1990.

[38] Z. Manna and A. Pnueli. The Temporal Logic of Reactive and Concurrent Systems: Specification. Springer-Verlag, 1992.

[39] R. Milner. A Modal Characterization of Observable Machine Behavior. InProc. CAAP’81, volume 112 ofLecture Notes in Computer Science, pages 25–34. Springer-Verlag, 1981.

[40] K. S. Namjoshi. Abstraction for branching time properties. InProceedings of CAV’2003 (15th Interna-tional Conference on Computer Aided Verification), volume 2725 ofLecture Notes in Computer Science, pages 288–300. Springer, 2003.

[41] K. S. Namjoshi and R. K. Kurshan. Syntactic Program Transformations for Automatic Abstraction. In


Computer Science, pages 435–449, Chicago, July 2000. Springer-Verlag.

[42] A. Pnueli and R. Rosner. On the synthesis of a reactive module. InProc. of the Sixteenth Symposim on Principles of Programming Languages, Austin, January 1989.

[43] A. Pnueli and R. Rosner. On the synthesis of an asynchronous reactive module. InProceedings of ICALP’89, Stresa, July 1989.

[44] M. Sagiv, T. Reps, and R. Wilhelm. Parametric Shape Analysis Via 3-Valued Logic. InProceedings of the 26th ACM Symposium on Principles of Programming Languages, January 1999.

[45] H. Saidi and N. Shankar. Abstract and model check while you prove. InProc. of the 11th Conference on Computer-Aided Verification, number 1633 in Lecture Notes in Computer Science, pages 443–454. Springer, 1999.

[46] K. Segerberg. Some Modal Logics Based on a Three-Valued Logic.Theoria, 33:53–71, 1967. [47] S. Shoham and O. Grumberg. Monotonic Abstraction-Refinement for CTL. InTools and Algorithms

for the Construction and Analysis of Systems: 10th International Conference, number 2988 in Lecture Notes in Computer Science, pages 546–560. Springer Verlag, 2004.

[48] C. Stirling. Modal Logics for Communicating Systems. Theoretical Computer Science, 49:331–347, 1987.

[49] W. Visser, K. Havelund, G. Brat, and S. Park. Model Checking Programs. InProceedings of ASE’2000 (15th International Conference on Automated Software Engineering), Grenoble, September 2000. [50] W. Visser, S. J. Park, and J. Penix. Using Predicate Abstraction to Reduce Object-oriented Programs

for Model Checking. InProceedings of FMSP’00 (Formal methods in Software Practice), pages 3–12, Portland, August 2000.


SAT-based Model Checking:

Interpolation, IC3, and Beyond

Orna GRUMBERGa, Sharon SHOHAMband Yakir VIZELa a

Computer Science Department, Technion, Haifa, Israel b

School of Computer Science, Academic College of Tel Aviv-Yaffo


SAT-based model checking is currently one of the most successful ap-proaches to checking very large systems. In its early days, SAT-based (bounded) model checking was mainly used for bug hunting. The intro-duction ofinterpolation and IC3\PDRenable efficient complete algo-rithms that can provide full verification as well.

In this paper, we survey several approaches to enhancing SAT-based model checking. They are all based on iteratively computing an over-approximationof the set of reachable system states. They use different mechanisms to achieve scalability and faster convergence.

The first one usesinterpolation sequencerather than interpolation in order to obtain a more precise over-approximation of the set of reachable states. The other approach integrates lazy abstraction with IC3 in order to achieve scalability.Lazy abstraction, originally developed for software model checking, is a specific type of abstraction that allows hiding dif-ferent model details at difdif-ferent steps of the verification. We find the IC3 algorithm most suitable for lazy abstraction since its state traversal is performed by means oflocal reachability checks, each involving only two consecutive sets. A different abstraction can therefore be applied in each of the local checks.

The survey focuses on hardware model checking, but the presented ideas can be extended to other systems as well.

Keywords.Model Checking, SAT-based Model Checking, Interpolation, Interpolation Sequence, Bounded Model Checking (BMC), IC3, Unbounded Model Checking, Abstraction, Lazy abstraction, Hardware model checking

1. Introduction

Computerized systems dominate almost every aspect of our lives and their cor-rect behavior is essential. Model checking [8, 10, 28] is an automated verification technique for checking whether a given system satisfies a desired property. The system is usually described as a finite-state model in a form of a state transition graph. The specification is given as a temporal logic formula. Unlike testing or simulation based verification, model checking tools are exhaustive in the sense that they traverseall behaviors of the system, and either confirm that the system behaves correctly or present a counterexample.

IOS Press, 2014


Model checking has been successfully applied to verifying hardware and soft-ware systems. Its main limitation, however, is the state explosion problem which arises due to the huge state space of real-life systems. The size of the model in-duces high memory and time requirements that may make model checking not applicable to large systems. Much of the research in this area is dedicated to increase model checking applicability and scalability.

The first significant step in this direction was the introduction of BDDs [4] into model checking. BDD-based Symbolic Model Checking (SMC) [5] enabled model checking of real-life hardware designs with a few hundreds of state ele-ments. However, current design blocks with well-defined functionality typically have thousands of state elements and more. To handle designs of that scale, SAT-based Bounded Model Checking (BMC) [2] has been developed. Its main draw-back, however, is its orientation towards ”bug-hunting” rather than full verifica-tion.

Several approaches have been suggested to remedy the problem. Induc-tion[29],interpolation[22],interpolation sequence[6,32],IC3/PDR[3,12], and L-IC3 [31] developed different techniques for SAT-basedUnbounded Model Checking (UMC), which provide full verification.

Of these SAT-based unbounded model checking techniques, L-IC3 and [6] also useAbstraction-refinement [9], which is another well known methodology for tackling the state-explosion problem. Abstraction hides model details that are not relevant for the checked property. The resulting abstract model is then smaller, and therefore easier to handle by model checking algorithms.Lazy abstraction[15, 23], developed for software model checking, is a specific type of abstraction that allows hiding different model details at different steps of the verification.

In this paper we discuss four of the above mentioned SAT-based approaches for full verification. These methods all compute an over-approximated set of the system’s reachable states while checking that the specification is not violated. The first two approaches we describe are taken from [22] and [32]. They combine BMC withinterpolation[11] orinterpolation-sequence [16,23] respectively. These methods use an unrolling of the model’s transition relation in order to traverse the system’s state space. The third algorithm we discuss is IC3/PDR [3, 12]. In contrast to the first two methods, IC3 avoids unrolling of the transition relation. Instead, the computation of reachable states useslocal reachability checksbetween consecutive time frames. Last, we present L-IC3 [31] which provides a SAT-based lazy abstraction-refinement algorithm based on IC3/PDR. L-IC3 uses the visible variables abstraction [18], which is particularly suitable for hardware systems. However, the abstraction is used in a lazy manner in the sense that different sets of visible variables are used in different iterations of the state-space traversal.


2. Preliminaries

Temporal logic model checking [10] is an automatic approach to formally verifying that a given system satisfies a given specification. The system is often modelled by a finite state transition system and the specification is written in a temporal logic. Determining whether a model satisfies a given specification is often based on an exploration of the model’s state space in a search for violations of the specification.

In this survey we focus on hardware. As such we consider finite state transition systems defined over Boolean variables, as follows.

Definition 2.1. A finite state transition system (a model) is a tuple M = (V, U,INIT,TR) where V is a set of Boolean variables, U ⊆V is a set of state variables,V\U is a set of input variables,INIT(V) is a propositional formula over V describing the initial states, andTR(V, V′) describes a total transition relation which is defined as a propositional formula over V and the next-state variables V′ ={v′|v∈V}.

The transition relation is described using next-state functions for each state variable. Namely, TR(V, V′) = V


(v′ =fv(V, V)) where fv(V, V) is a proposi-tional formula that assigns the next value to v ∈U based on current and next-state variables. Note that for an input variablev∈V \U,fv is not defined.

The set of Boolean variables ofM induces a set of statesS={0,1}|V|, where each state s ∈ S is given by a valuation of the variables in V. A formula over V (resp.V, V′) represents the set of states (resp. pairs of states) obtained by its satisfying assignments. With abuse of notation we will refer to a formula η over V as a set of states and therefore use the notions∈η for states represented by η. Similarly for a formulaη overV, V′, we will sometimes write (s, s′)∈η.

The formula η[V ← V′], or η′ in short, is identical to η except that each variablev ∈V is replaced with v′. In the general case Vi is used to denote the variables inV afteritime units (thus,V0V). Letη be a formula overVi, the formulaη[Vi Vj] is identical to η except that for each variable v V, vi is replaced withvj.

A path in M is a sequence of states π = s0, s1, . . . such that for all i ≥ 0, si∈S and (si, si+1)∈TR. The length of a path is denoted by|π|. Ifπis infinite then|π| =∞. Ifπ =s0, s1, . . . , sn then |π|=n. A path is aninitial path when s0∈INIT. We sometimes refer to a prefix of a path as a path as well.

A formula inLinear Temporal Logic(LTL) [10, 27] is of the formAf wheref is a path formula. A model M satisfies an LTL propertyAf if all infinite initial paths inM satisfyf. If there exists an infinite initial path not satisfyingf, this path is defined to be acounterexample.


Figure 1. Known results on the complexity in the size of the formula for (2-valued and 3-valued) modelchecking (MC), satisfiability (SAT) and generalized model checking (GMC).

Figure 1.

Known results on the complexity in the size of the formula for 2 valued and 3 valued modelchecking MC satis ability SAT and generalized model checking GMC . View in document p.21
Figure 2. Updating the reachability sequence Ω

Figure 2.

Updating the reachability sequence . View in document p.38
Figure 3. Checking if a fixpoint has been reached

Figure 3.

Checking if a xpoint has been reached. View in document p.39
Figure 5. Computing reachable states using interpolation and BMC with a specific bound k

Figure 5.

Computing reachable states using interpolation and BMC with a speci c bound k. View in document p.40
Figure 6. L-IC3

Figure 6.

L IC3. View in document p.46
Figure 7. A-IC3

Figure 7.

A IC3. View in document p.47
Figure 8. Iterative strengthening of A-IC3

Figure 8.

Iterative strengthening of A IC3. View in document p.49
Figure 9. BlockState procedure of A-IC3

Figure 9.

BlockState procedure of A IC3. View in document p.50
Table 2. Lazy abstraction. N stands for the name of the verified property.given time framenumber of state variables in the concrete model ♯Vars stands for the Mc

Table 2.

Lazy abstraction N stands for the name of the veri ed property given time framenumber of state variables in the concrete model Vars stands for the Mc. View in document p.52
Figure 4. Small sampling of different ways of generating parts of an output string from the input string.

Figure 4.

Small sampling of different ways of generating parts of an output string from the input string . View in document p.65
Figure 8. Problem Generation for algebraic proof problems involving identities over analytic functions [37]such as trigonometry and determinants

Figure 8.

Problem Generation for algebraic proof problems involving identities over analytic functions 37 such as trigonometry and determinants. View in document p.76
Figure 12. Solution Generation for geometry constructions [21].

Figure 12.

Solution Generation for geometry constructions 21 . View in document p.80
Figure 14. Automated Grading of introductory programming problems [38]. (a) depicts a reference imple-mentation for the problem of computing a derivative of a polynomial

Figure 14.

Automated Grading of introductory programming problems 38 a depicts a reference imple mentation for the problem of computing a derivative of a polynomial. View in document p.84
Figure 7. Inductive definition of

Figure 7.

Inductive de nition of. View in document p.110
Figure 9. Inductive definition of op ⊢ :: tyenv ⇒ com ⇒ bool

Figure 9.

Inductive de nition of op tyenv com bool. View in document p.111
Figure 10. Definition of sec type :: nat ⇒ com ⇒ bool

Figure 10.

De nition of sec type nat com bool. View in document p.118
Figure 12. Definition of the bottom-up security type system.

Figure 12.

De nition of the bottom up security type system . View in document p.122
Figure 11. Definition of sec type′ :: nat ⇒ com ⇒ bool

Figure 11.

De nition of sec type nat com bool. View in document p.122
Figure 13. Termination-sensitive security type system.

Figure 13.

Termination sensitive security type system . View in document p.124
Figure 14. Termination-sensitive security type system — standard formulation.

Figure 14.

Termination sensitive security type system standard formulation . View in document p.126
Figure 15. Definite initialisation D :: vname set ⇒ com ⇒ vname set ⇒ bool

Figure 15.

De nite initialisation D vname set com vname set bool. View in document p.131
Figure 16. Small-step semantics, initialisation sensitive

Figure 16.

Small step semantics initialisation sensitive. View in document p.134
Figure 17. Big-step semantics with error propagation

Figure 17.

Big step semantics with error propagation. View in document p.136
Figure 18. Definition of defs.

Figure 18.

De nition of defs . View in document p.140
Figure 19. Congruence rules for conditional semantic equivalence.

Figure 19.

Congruence rules for conditional semantic equivalence . View in document p.143
Figure 2. Level of privacy (the smallerThreeBallot with two candidates, δ the higher privacy) for the considered variants of p0 = 0.3, p1 = 0.35, p2 = 0.35

Figure 2.

Level of privacy the smallerThreeBallot with two candidates the higher privacy for the considered variants of p0 0 3 p1 0 35 p2 0 35. View in document p.161
Figure 3. Level of coercion (δ) for different protocols with two candidates, p0=0.3,p1 = p2 = 0.35

Figure 3.

Level of coercion for di erent protocols with two candidates p0 0 3 p1 p2 0 35. View in document p.163
Figure 4. The lower-bound of coercion-resistance (δ) for ThreeBallot in the variant by Marneffeet al

Figure 4.

The lower bound of coercion resistance for ThreeBallot in the variant by Marne eet al. View in document p.163
Figure 1. A DTMC and its transition probability matrix P for an approximate majority chemical reactionnetwork (CRN) that initially contains 2 molecules of X and 1 molecule of Y.

Figure 1.

A DTMC and its transition probability matrix P for an approximate majority chemical reactionnetwork CRN that initially contains 2 molecules of X and 1 molecule of Y . View in document p.180
Figure 2. Determining the probabilitiesis used to determine the probabilities for the remaining states (i.e.,labelled with the Prob(S, ¬aUx) for the DTMC from Figure 1

Figure 2.

Determining the probabilitiesis used to determine the probabilities for the remaining states i e labelled with the Prob S aUx for the DTMC from Figure 1. View in document p.183