This Series presents the results of scientific meetings supported under the NATO Programme: Science for Peace and Security (SPS).

The NATO SPS Programme supports meetings in the following Key Priority areas: (1) Defence Against Terrorism; (2) Countering other Threats to Security and (3) NATO, Partner and Mediterranean Dialogue Country Priorities. The types of meeting supported are generally “Advanced Study Institutes” and “Advanced Research Workshops”. The NATO SPS Series collects together the results of these meetings. The meetings are co-organized by scientists from NATO countries and scientists from NATO’s “Partner” or “Mediterranean Dialogue” countries. The observations and recommendations made at the meetings, as well as the contents of the volumes in the Series, reflect those of participants and contributors only; they should not necessarily be regarded as reflecting NATO views or policy.

**Advanced Study Institutes (ASI) are high-level tutorial courses to convey the latest **
developments in a subject to an advanced-level audience.

**Advanced Research Workshops (ARW) are expert meetings where an intense but informal **
exchange of views at the frontiers of a subject aims at identifying directions for future action.
Following a transformation of the programme in 2006 the Series has been named and
re-organised. Recent volumes on topics not related to security, which result from meetings
supported under the programme earlier, may be found in the NATO Science Series.

The Series is published by IOS Press, Amsterdam, and Springer Science and Business Media, Dordrecht, in conjunction with the NATO Emerging Security Challenges Division.

**Sub-Series **

A. Chemistry and Biology Springer Science and Business Media B. Physics and Biophysics Springer Science and Business Media C. Environmental Security Springer Science and Business Media D. Information and Communication Security IOS Press

E. Human and Societal Dynamics IOS Press

http://www.nato.int/science http://www.springer.com http://www.iospress.nl

Sub-Series D: Information and Communication Security – Vol. 36 ISSN 1874-6268 (print)

## Software Systems Safety

### Edited by

### Orna Grumberg

*Computer Science Department, Technion, Haifa, Israel *

### Helmut Seidl

*Technische Universität München, Germany *

### and

### Maximilian Irlbeck

*Technische Universität München, Germany *

Marktoberdorf, Germany 30 July - 11 August 2013

© 2014 The authors and IOS Press.

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without prior written permission from the publisher.

ISBN 978-1-61499-384-1 (print) ISBN 978-1-61499-385-8 (online)

Library of Congress Control Number: 2014 936 390

*Publisher *
IOS Press BV
Nieuwe Hemweg 6B
1013 BG Amsterdam
Netherlands

fax: +31 20 687 0019 e-mail: order@iospress.nl

*Distributor in the USA and Canada *
IOS Press, Inc.

4502 Rachael Manor Drive Fairfax, VA 22032 USA

fax: +1 703 323 3668

e-mail: iosbooks@iospress.com

LEGAL NOTICE

### Preface

Only 20 years ago the correctness and security of software systems was a largely theoretical problem, relevant only for a small group of computer science specialists. To-day it has become a fundamental problem of society at large: software bugs in programs used by hundred million people, security breaches in banking software, and malware attacks compromising millions of computers make it to the headlines almost daily. The computer science community is attacking the problem by developing verification and synthesis tools that mechanize more and more tasks in the design of correct and secure programs. The School provided courses about verification and synthesis tools, methodo-logies (testing, model-checking, synthesis and information-flow analysis), and adaptati-on and tailoring to applicatiadaptati-on domains.

The lecturers of the NATO Advanced Study Institute “Summer School Marktober-dorf 2013 – Software Systems Safety“ represented research groups from industry as well as universities and showed a detailed and excellent overview on current research results with special emphasis to solve such problems in software systems security.

Software Model Checking via Systematic Testingwas the title of PATRICE GO-DEFROID’s lecture. Over the last two decades, significant progress has been made on how to broaden the scope of model checking from finite-state abstractions to actual soft-ware implementations. Adapting model checking into a form of systematic testing that is applicable to industrial-size software might be a solution. His lectures presented a com-prehensive overview of this strand of software model checking, by describing the main ideas, techniques and results obtained in this area, including combinations with static program analysis.

ORNA GRUMBERG lectured on SAT-Based Model Checking: Interpolation, IC3 and Beyond. Model checking is an automatic approach to formally verifying that a given system satisfies a given specification. In spite of its great success in verifying hardware and software systems, the applicability of model checking is impeded by its high space and time requirements. We surveyed several approaches to enhancing SAT-based model checking. All these approaches result in efficient and complete SAT-SAT-based verification algorithms.

SUMIT GULWANI explained thatProgram Synthesisis the task of searching for a program in some underlying domain-specific language that matches the user’s intent. His lectures described three key dimensions in program synthesis. The first one was un-derlying domain-specific language in which programs are synthesized. Second he talked about user intent specification mechanism and interaction model to resolve ambiguities in under-specification. The third topic were search techniques. These dimensions were il-lustrated via a variety of applications in algorithm discovery (e.g., Bitvector/SIMD/graph algorithms, program inverses) and end-user programming (e.g., Spreadsheet programs, and smartphone scripts). Some surprising applications in the area of computer-aided edu-cation were presented (including problem synthesis, solution synthesis, feedback syn-thesis, and content authoring for a variety of subject domains including math, logic, and programming).

*IOS Press, 2014 *

Interactive Proof: Applications to Data Flow Analysis and Securitywas GER-WIN KLEIN’s lecture which was built on the previous introduction to interactive proof. He demonstrated a specific application of interactive proof assistants: the semantics of programming languages. In particular, he taught how to formalize a small imperative programming language in the theorem prover Isabelle/HOL and how to define its seman-tics. We covered a basic type system with type safety proof, a more complex security sys-tem also with soundness proof, and different kinds of data flow analysis, such as liveness analysis.

RALF KÜSTERS’ lecture was onE-Voting Systems. Systems for electronic vo-ting (e-vovo-ting), including systems for vovo-ting in a vovo-ting booth, are employed in many countries and numerous problems with e-voting systems have been reported. Most of the systems used in practice today do not provide a sufficient level of security. Programming errors and malicious behaviour easily go undetected. Ralf covered central security requi-rements of e-voting systems, and how they can be formally defined and analysed. While analysis is mostly done based on cryptographic models or even more abstract so-called Dolev-Yao models, he also discussed approaches to perform (cryptographic) analysis directly on the implementation-/language-level of a system.

The lecture of MARTA KWIATKOWSKA held was titled Probabilistic Model Checking for Biology. Probabilistic model checking is an automated method for verify-ing the correctness and performance of probabilistic models. Property specifications are expressed in probabilistic extensions of temporal logic, and the probabilistic model che-cker can compute the probability of a given event, of its occurrence within a given time interval, or expected cost reward. Discrete- and continuous-time Markov chains were in-troduced, as well as the logics PCTL and CSL, and the corresponding model checking algorithms. An application of probabilistic model checking to two biological examples was shown: molecular signalling pathways and DNA computation. Marta demonstrated how design errors can be automatically detected in DNA gate designs using probabilistic model checking, analogously to model checking for digital circuits.

In the set of lectures onInfinite-state Model Checkingby RUPAK MAJUMDAR looked at the problem of automated analysis of infinite-state systems. Infinite-state sys-tems arise in many program verification problems, due to many different features in the models. For example, systems may contain unbounded data values (integers), data struc-tures (lists, arrays, etch.), real-valued clocks or other physical variables, unbounded exe-cution stacks, or unboundedly many threads of exeexe-cution. The purpose of these lectures was to understand how techniques from finite-state model checking can be extended to the infinite-state case. He showed both heuristic techniques (for special cases that are de-cidable). The first part dealt with abstraction of systems and counterexample-guided refi-nement. He showed how states can be represented as logical constraints and how decisi-on procedures for logical theories can be used to compute finite-state abstractidecisi-ons of sys-tems. He also presented a model checking algorithm based on abstraction and refinement. The second part focused on decidability results for infinite-state systems. He showed a general decidability result for well-structured transition systems using Petri nets. Also an algorithm to verify recursive programs with finite data domains was taught. Finally, he considered asynchronous programs, a programming model including both recursion and unbounded task creation for which safety and liveness properties remain decidable.

formula can ever evaluate to true. Several relatively recent research developments have enabled us to tackle formulas with millions of variables and constraints enabling SAT solvers to be effectively deployed in practical applications including in the analysis and verification of software systems. In the first part of this series of lectures, he covered techniques used in modern SAT solvers. In the second part, he considered extension of these solvers that have proved to be useful in analysis and verification. He showed how these concepts are related, present algorithms to derive them and show their application in design debugging.

The courseInteractive Proof: Hands-on Introductionby TOBIAS NIPKOW in-troduced interactive theorem proving with the Isabelle/HOL system in the following 3 steps. The first one was verified functional programming. The logic HOL contains an ML-style functional programming language. It was shown how to verify functional pro-grams in this language by induction and simplification. The second topic was predicate logic. Formulas of predicate logic and set theory were introduced together with induc-tively defined predicates. In a third step we talked about structured proofs. The proof language Isar was introduced and it was shown how to write structured proofs that are readable by both the machine and the human. The course assumed basic familiarity with some functional programming language of the ML or Haskell family, in particular with recursive data types and pattern matching. No specific background in logics is necessary beyond the ability to read predicate logic formulas.

DAVID SANDS held a lecture onSoftware Security by Information Flow Con-trol. A long-term goal is to construct innovative design methods for the construction of secure systems that put security requirements at the heart of the construction process, namely security by design. But to do this we had to understand how to unambiguously formulate the policy aims for secure systems and to develop a technology to integrate these goals into design mechanisms and technologies that enables an efficient construc-tion or verificaconstruc-tion of systems with the respect to those policies. He managed to better understand the semantics of information flow, explore dynamic and state-dependent po-licies, and to see how these ideas can be incorporated into a programming language with the support for static verification of a rich variety of information flow policies.

HELMUT SEIDL talked about A General Infrastructure for Interprocedural Analysis of Concurrent C. His tutorial was about infrastructures for general-purpose interprocedural analyses. It consisted of two parts. At first he argued that side-effecting constraint systems might serve as kind of a Swiss army knife for specifying analyses, while secondly he provided an overview on solving techniques for such systems. He re-ported on techniques for solving side-effecting constraint systems. One major issue he-re was that non-trivial analysis problems he-requihe-re complete lattices with infinite ascen-ding and descenascen-ding chains. In order to compute reasonably precise post-fixpoints of the resulting systems of equations, Cousot and Cousot have suggested accelerated fixpoint iteration by means of widening and narrowing.

Practical Program Analysis and Synthesiswas the lecture by ERAN YAHAV. For

abili-ty to hide some aspects of concurrency in the library, even application-level program-mers might still need to reason about atomicity. Eran taught a framework for synthesi-zing efficient synchronization in concurrent programs, a task known to be difficult and error-prone when done manually. Given a program, a specification, and an abstraction, we inferred synchronization that avoids all (abstract) interleaving that may violate the specification, but permits as many valid interleaving as possible.

Over the years, the Marktoberdorf Summer School has facilitated its scientifically profitable and well-known tradition and will follow future challenges in formal compu-ter science to ensure cyber security. The Summer School provides two weeks of chal-lenging and fruitful learning, discussion and development of new ideas, and should be a productive and beneficial event, at both the professional and the social level.

From the very first, the ASI "Marktoberdorf"was planned as a periodical forum with leading researchers to teach, learn and discuss the state-of-the-art research with the target to build a special international community. Out series of ASIs has been proved as a high-level scientific nucleus for international scientific networking and is settled as one of the most famous InternationalComputer ScienceSummer Schools – shown in an impressive way by the high number of applications every year.

We thank all lecturers, the staff, and hosts in Marktoberdorf. Special thanks must be addressed to Dr. Katharina Spies and Silke Müller for their great support.

The Marktoberdorf Summer School was arranged as an Advanced Study Institute of the NATO Science for Peace and Security Programmewith support fromDeutscher Akademischer Austausch Dienst (DAAD). We thank all authorities involved.

### Contents

Preface v

May/Must Abstraction-Based Software Model Checking for Sound

Verification and Falsification 1

* Patrice Godefroid *

SAT-based Model Checking: Interpolation, IC3, and Beyond 17

* Orna Grumberg, Sharon Shoham and Yakir Vizel *

Program Synthesis 43

* Sumit Gulwani *

Applications of Interactive Proof to Data Flow Analysis and Security 77

* Gerwin Klein and Tobias Nipkow *

E-Voting Systems 135

* Ralf Küsters *

Probabilistic Model Checking for Biology 165

* Marta Kwiatkowska and Chris Thachuk *

Foundations of Infinite-State Verification 191

* Rupak Majumdar *

Boolean Satisfiability: Solvers and Extensions 223

* Georg Weissenbacher, Pramod Subramanyan and Sharad Malik *

Programming in Paragon 279

* Bart van Delft, Niklas Broberg and David Sands *

Frameworks for Interprocedural Analysis of Concurrent Programs 309

* Helmut Seidl, Kalmer Apinis and Vesal Vojdani *

Practical Software Synthesis 349

* Eran Yahav *

Subject Index 379

### May/Must Abstraction-Based

### Software Model Checking

### for Sound Verification and Falsification

Patrice GODEFROID

*Microsoft Research*

**Abstract.***Three-valued models*, in which properties of a system are either true,
false or unknown, have recently been advocated as a better representation for
reac-tive program abstractions generated by automatic techniques such as predicate
ab-straction. Indeed, for the same cost, model checking three-valued abstractions, also
called*may/must abstractions*, can be used to both prove and disprove any
temporal-logic property, whereas traditional conservative abstractions can only prove
uni-versal properties. Also, verification results can be more precise with*generalized*
*model checking*, which checks whether there exists a concretization of an
abstrac-tion satisfying a temporal-logic formula. Generalized model checking generalizes
both model checking (when the model is complete) and satisfiability (when
every-thing in the model is unknown), probably the two most studied problems related to
temporal logic and verification.

This paper presents an introduction to the main ideas behind this framework, namely models for three-valued abstractions, completeness preorders to measure the level of completeness of such models, three-valued temporal logics and gen-eralized model checking. It also discusses algorithms and complexity bounds for three-valued model checking and generalized model-checking for various tempo-ral logics. Finally, it discusses applications to program verification via automatic abstraction.

**Keywords.**Software Model Checking, Automatic Abstraction, Verification, Bug
Finding

**1. Introduction**

How to broaden the scope of model checking to software is currently one of the most challenging problems related to computer-aided verification. Essentially two approaches have been proposed and are still actively being investigated. The first approach consists of adapting model checking into a form of systematic testing that simulates the effect of model checking while being applicable to operating-system processes executing arbitrary code [17,23]; although counter-examples reported with this approach are sound, it is inherently incomplete for large systems. The second approach consists of automatically extracting a model out of a software application by statically analyzing its code, and then of analyzing this model using traditional model-checking algorithms (e.g., [4,8,49,41, 29]); although automatic abstraction may be able to prove correctness, counter-examples are generally unsound since abstraction usually introduces unrealistic behaviors that may yield to spurious errors being reported when analyzing the model.

*IOS Press, 2014 *

In this paper, we present an overview of a series of articles [5,6,20,21,22,18,14,19, 25] discussing how automatic abstraction can be performed to verify arbitrary formu-las of the propositionalµ-calculus [35] in such a way that both correctness proofs and counter-examples are guaranteed to be sound.

The key to make this possible is to represent abstract systems using richer models
that distinguish properties that are*true*,*false*and*unknown*of the concrete system.
Ex-amples of such richer modeling formalisms are partial Kripke structures [5] and Modal
Transition Systems [36,20]. Reasoning about such systems requires 3-valued temporal
logics [5], i.e., temporal logics whose formulas may evaluate to*true*,*false*or⊥
(“un-known”) on a given model. Then, by using an automatic abstraction process that
gener-ates by construction an abstract model which is less complete than the concrete system
with respect to a completeness preorder logically characterized by 3-valued temporal
logic, every temporal property that evaluates to*true*(resp.*false*) on the abstract model
automatically holds (resp. does not hold) of the concrete system, hence guaranteeing
soundness of both proofs and counter-examples. In case a property evaluates to⊥on the
model, a more complete (i.e., less abstract) model is then necessary to provide a
defi-nite answer concerning this property of the concrete system. This approach is
applica-ble to check arbitrary formulas of the propositionalµ-calculus (thus including negation
and arbitrarily nested path quantifiers), not just universal properties as with a traditional
“conservative” abstraction that merely simulates the concrete system.

**2. Three-Valued Modeling Formalisms**

Examples of 3-valued modeling formalisms for representing partially defined systems
are *partial Kripke structures* (PKS) [5], *Modal Transition Systems* (MTS) [36,20] or

*Kripke Modal Transition Systems*(KMTS) [30].

**Definition 1** *A*KMTS*M is a tuple*(S,*P*,−→*must*,−→*may*,*L), where S is a nonempty finite set of*
*states, P is a finite set of atomic propositions,*−→⊆*may* *S*×*S and*−→⊆*must* *S*×*S are transition*
*relations such that*−→⊆*must* −→*may, and L*:*S*×*P*→ {*true,*⊥,*false*}*is an*interpretation*that*
*associates a truth value in*{*true,*⊥,*false*} *with each atomic proposition in P for each*
*state in S. An MTS is a KMTS where P*=/0*. A PKS is a KMTS where*−→*must*=−→*may.*

The third value⊥(read “unknown”) and*may*-transitions that are not*must*-transitions are
used to model explicitly a loss of information due to abstraction concerning, respectively,
state or transition properties of the concrete system being modeled. A standard,*complete*

Kripke structure is a special case of KMTS where−→*must*=−→*may* and*L*:*S*×*P*→ {*true,false*},
i.e., no proposition takes value⊥in any state.

It can be shown [22] that PKSs, MTSs, KMTSs and variants of KMTSs where
tran-sitions are labeled and/or two interpretation functions*Lmay*_{and}_{L}must_{are used [30], are}

**3. Three-Valued Temporal Logics**

When evaluating a temporal-logic formula on a 3-valued model, there are three possible
outcomes: the formula can evaluate to*true*,*false*or⊥(unknown). Formally, we define
3-valued (temporal) logics as follows.

In interpreting propositional operators on KMTSs, we use Kleene’s strong 3-valued
propositional logic [34], which generalizes the standard 2-valued semantics. Conjunction
∧ in this logic is defined as the function that returns*true*if both of its arguments are

*true*,*false*if either argument is*false*, and⊥otherwise. We define negation¬using the
function ‘comp’ that maps*true*to*false*,*false*to*true*, and⊥to⊥. Disjunction∨is defined
as usual using De Morgan’s laws:*p*∨*q*=¬(¬*p*∧ ¬*q)*. Note that these functions give the
usual meaning of the propositional operators when applied to values*true*and*false*.

Propositional modal logic (PML) is propositional logic extended with the modal
operator*AX* (which is read “for all immediate successors”). Formulas of PML have the
following abstract syntax: φ ::=*p*| ¬φ |φ1∧φ2|*AX*φ, where *p* ranges over*P*. The
following 3-valued semantics generalizes the traditional 2-valued semantics for PML.

**Definition 2** *The value of a formula*φ *of 3-valued PML in a state s of a KMTS M*=

This 3-valued logic can be used to define a preorder on KMTSs that reflects their
degree of completeness. Let≤be the *information ordering*on truth values, in which
⊥ ≤*true*,⊥ ≤*false*,*x*≤*x*(for all*x*∈ {*true,*⊥,*false*}), and*x*6≤*y*otherwise. Note that
the operators comp, min and max are monotonic with respect to the information
order-ing≤: if*x*≤*x*′and*y*≤*y*′, we have comp(x)≤comp(x′), min(x,*y)*≤min(x′,*y*′), and
*KMTSs. The* completeness preorder*is the greatest relation*B _{⊆}*SA*×*SC* *such that*
(s*a,sc)*∈B*implies the following:*

This definition allows to abstract*MC* by*MA*by letting truth values of propositions

*com-plete*, than*MC*. The inverse of the completeness preorder is also called*refinement *
*pre-order*in [36,30,20]. Note that relationB_{reduces to a simulation relation when applied}
to MTSs with *may*-transitions only. Also note that relationB_{reduces to bisimulation}
when applied to MTSs with*must*-transitions only and where all atomic propositions in

*P*are either*true*or*false*.

It can be shown that 3-valued PML logically characterizes the completeness pre-order [5,30,20].

**Theorem 4** *[5] Let MA*= (S*A*,*P*,
*must*

−→*A*,
*may*

−→*A*,L*A)and MC*= (S*C*,*P*,
*must*

−→*C*,
*may*

−→*C*,*LC)be*
*KMTSs such that sa*∈*SAand sc*∈*SC, and let*Φ*be the set of all formulas of 3-valued*
*PML. Then,*

*sasciff*(∀φ∈Φ:[(M*A*,*sa)*|=φ]≤[(M*C*,s*c)*|=φ]).

In other words, KMTSs that are “more complete” with respect tohave more
def-inite properties with respect to≤, i.e., have more properties that are either*true*or*false*.
Moreover, any formulaφ of 3-valued PML that evaluates to*true*or*false*on a KMTS
has the same truth value when evaluated on any more complete structure. This result also
holds for PML extended with fixpoint operators, i.e., the propositionalµ-calculus [5].

The following theorem states that 3-valued propositional modal logic logically char-acterizes the equivalence relation induced by the completeness preorder.

**Theorem 5** *[5] Let M*1= (S1,*P*,

*must*

−→1,−→1*may* ,L1) *and M*2= (S2,*P*,

*must*

−→2,−→2*may* ,*L*2) *be*

*KMTSs such that s*1∈*S*1*and s*2∈*S*2*, and let*Φ*denote the set of all formulas of 3-valued*

*propositional modal logic. Then*

(∀φ∈Φ:[(M1,*s*1)|=φ] = [(M2,*s*2)|=φ])*iff*(s1*s*2*and s*2*s*1).

Note that if two states*s*1and*s*2are*bisimilar*, denoted*s*1∼*s*2, this implies both*s*1

*s*2and*s*2*s*1. This means that 3-valued propositional modal logic cannot distinguish
between bisimilar states.

However, the converse is not true:*s*1*s*2and*s*2*s*1does not imply*s*1∼*s*2. This
is illustrated by the example below. The existence of such an example proves that, in
contrast with 2-valued propositional modal logic, 3-valued propositional modal logic is

*not*a logical characterization of bisimulation.

**Example 6** [5] Here is an example of two non-bisimilar states that cannot be
distin-guished by any formula of 3-valued propositional modal logic.

s0

s1 s2 s3

(true, )⊥ _{( , )}_{⊥} _{⊥} _{( ,true)}_{⊥} _{( , )}_{⊥} _{⊥}

s’1 s’3

s’0

s’2

(true,true) (true,true)

These two partial Kripke structures have two atomic propositions *p* and*q*, whose
truth value is defined in each state as indicated in the figure by a pair of the form(p,q).
We have the following relations:

• *s*2*s*′2and*s*′2*s*2,

• *s*3*s*′3and*s*′3*s*3,

• *s*1*s*′2and*s*′3*s*1,*s*′1*s*2and*s*3*s*′1,

• *s*0*s*′_{0}and*s*′_{0}*s*0.

We have that*s*0*s*′0and*s*′0*s*0, but*s*06∼*s*′0since*s*1is not bisimilar to any state in the
second partial Kripke structure.

**4. Three-Valued Model Checking**

Given a state *s* of a 3-valued model*M* and a formula φ, how to compute the value

[(M,*s)*|=φ]?

This is the *3-valued model checking*problem. In [6], it is shown that computing

[(M*A*,*s)*|=φ]can be reduced to two traditional (2-valued) model-checking problems on

complete systems (such as Kripke structures or Labeled Transition Systems).

**Theorem 7** *[6] The model-checking problem for a 3-valued temporal logic can be *
*re-duced to two model-checking problems for the corresponding 2-valued logic.*

The reduction can be performed in 3 steps as follows.

**Step 1.**Complete*M*into two “extreme” complete Kripke structures, called the
**op-timistic***Mo*and**pessimistic***Mp*completions, defined as follows:

• Extend*P*to*P*′such that, for every*p*∈*P*there exists a ¯*p*∈*P*′such that*L(s,p) =*

comp(L(s,*p))*¯ for all*s*in*S*.

• *Mo*= (S,*Lo*,
*must*

−→)with

*Lo(s,p)*def=

*true* if*L(s,p) =*⊥

*L(s,p)*otherwise

• *Mp*= (S,*Lp,*−→*may*)with
*Lp(s,p)*def=

*f alse* if*L(s,p) =*⊥

*L(s,p)*otherwise

**Step 2.**Transform the formulaφ to its positive form*T*(φ)by pushing negation
in-wards using De Morgan’s laws, and replacing remaining negations¬*p*at the
proposi-tional level by ¯(p).

**Step 3.**Evaluate*T*(φ)on *Mo*and*Mp*using traditional 2-valued model checking,

and combine the results as follows:

[(M,*s)*|=φ] =

*true* if(M*p*,*s)*|=*T*(φ)
*f alse*if(M*o*,s)6|=*T*(φ)

⊥ otherwise

This can be done using existing model-checking tools! The formula is*true*at*s*if it is

It can be proved [6] that the above procedure computes the correct value for

[(M,*s)*|=φ]according to the 3-valued semantics defined in the previous section.
An immediate corollary from this result is that *3-valued model checking has the*
*same (time and space) complexity as traditional 2-valued model checking*. Indeed, the
transformations of*M*into*Mo*and*Mp*, and ofφinto*T*(φ)can be done in linear time and

logarithmic space in the size of*M*andφ, respectively.

**Example 8** [5] Consider the three following partial Kripke structures with a single
atomic proposition*p*, whose truth value is defined in each state as indicated in the figure.

⊥ p= ⊥

p= ⊥

p=

s2

s1 s3

p=true

p=true

p=false p=false

p=true p=false

The formula*A(true*U _{p)}_{of 3-valued CTL is read “for all paths, does}*p*eventually
hold?”. It has a different truth value in each of the top states of these partial Kripke
structures:[s1|=*A(true*U *p)] =true*,[s2|=*A(true*U *p)] =*⊥, and[s3|=*A(true*U *p)] =*

*false*.

**5. Generalized Model Checking**

However, as argued in [6], the semantics of [(M,*s)*|=φ]returns⊥more often than it
should. Consider a KMTS*M*consisting of a single state*s*such that the value of
proposi-tion*p*at*s*is⊥and the value of*q*at*s*is*true*. The formulas*p*∨ ¬*p*and*q*∧(p∨ ¬*p)*are
⊥at*s*, although in all complete Kripke structures more complete than(M,*s)*both
for-mulas evaluate to*true*. This problem is not confined to formulas containing subformulas
that are tautological or unsatisfiable. Consider a KMTS*M*′with two states*s*0and*s*1such
that*p*=*q*=*true*in*s*0and*p*=*q*=*false*in*s*1, and with a*may*-transition from*s*0to*s*1.
The formula*AX p*∧ ¬*AX q*(which is neither a tautology nor unsatisfiable) is⊥at*s*0, yet
in all complete structures more complete than(M′_{,}_{s}

0)the formula is*false*.

This observation is used in [6] to define an alternative 3-valued semantics for modal
logics called the*thorough*semantics since it does more than the other semantics to
dis-cover whether enough information is present in a KMTS to give a definite answer. Let
the*completions*C_{(M,}_{s)}_{of a state}_{s}_{of a KMTS}_{M}_{be the set of all states}* _{s}*′

_{of complete}Kripke structures

*M*′such that

*ss*′.

**Definition 9** *Let*φ*be a formula of any two-valued logic for which a satisfaction relation*

|=*is defined on complete Kripke structures. The truth value of*φ*in a state s of a KMTS*
*M under the*thorough*interpretation, written*[(M,*s)*|=φ]t*, is defined as follows:*

[(M,*s)*|=φ]t=

*true if*(M′_{,}* _{s}*′

_{)}

_{|}

_{=}

_{φ}

_{for all}_{(M}′

_{,}

*′*

_{s}_{)}

_{in}_{C}

_{(M,}

_{s)}*false if*(M′

_{,}

*′*

_{s}_{)}

_{6|}

_{=}

_{φ}

_{for all}_{(M}′

_{,}

*′*

_{s}_{)}

_{in}_{C}

_{(M,}

_{s)}It is easy to see that, by definition, we always have[(M,*s)*|=φ]≤[(M,*s)*|=φ]t. In
gen-eral, interpreting a formula according to the thorough three-valued semantics is
equiva-lent to solving two instances of the generalized model-checking problem [6].

**Definition 10 (Generalized Model-Checking Problem)** *Given a state s of a KMTS M*
*and a formula* φ *of a (two-valued) temporal logic L, does there exist a state s*′ *of a*
*complete Kripke structure M*′*such that ss*′*and*(M′_{,}* _{s}*′

_{)}

_{|}

_{=}

_{φ}

_{?}This problem is called *generalized model-checking* since it generalizes both model
checking and satisfiability checking. At one extreme, where*M*= ({*s*0},*P*,−→*must*=−→*may*=

{(s0,*s*0)},*L)*with*L(s*0,*p) =*⊥for all *p*∈*P*, all complete Kripke structures are more
complete than*M*and the problem reduces to the satisfiability problem. At the other
ex-treme, where*M*is complete, only a single structure needs to be checked and the problem
reduces to model checking.

Therefore, the worst-case complexity for the generalized model-checking problem will never be better than the worst-case complexities for the model-checking and satisfi-ability problems for the corresponding logic. The following theorem formally states that the generalized model-checking problem is at least as hard as the satisfiability problem.

**Theorem 11** *[6] Let L denote the propositional* µ-calculus or any of its fragments
*(propositional logic, PML, LTL, CTL, CTL*∗*, etc.). Then the satisfiability problem for L*
*is reducible (in linear-time and logarithmic space) to the generalized model-checking*
*problem for L.*

Is generalized model checking harder than satisfiability? It depends.

For *branching-time* temporal logics, it can be shown [6] that generalized model
checking has the same complexity as satisfiability.

**Theorem 12** *[6] Let L denote propositional logic, PML, CTL, or any branching-time*
*logic including CTL (such as CTL*∗*or the propositional* µ-calculus). The generalized
*model-checking problem for the logic L has the same complexity as the satisfiability*
*problem for L.*

In contrast, for*linear-time*temporal logic (LTL), generalized model checking can
be harder than satisfiability [25]. We have the following.

**Theorem 13** *[25] Given a state s*0 *of partial Kripke structure M*= (S,*L,*R)*and an*

*LTL formula*φ*, one can construct an alternating parity word automaton A*_{(}_{M,s}

0),φ*over a*
*1-letter alphabet with at most O(*|*S*| ·22|φ|*log*(|φ|))*states and*2*O*(|φ|)*priorities such that*

(∃(M′,s′_{0}):*s*0*s*′0*and*(M′,*s*′0)|=φ)*iff*L(A(*M,s*0),φ)6=/0.

**Theorem 14** *[25] The generalized model-checking problem for linear-time temporal*
*logic is 2EXPTIME-complete.*

**Logic** **MC** **SAT** **GMC**

Propositional Logic Linear NP-complete NP-complete

PML Linear PSPACE-complete PSPACE-complete

CTL Linear EXPTIME-complete EXPTIME-complete

µ-calculus NP∩co-NP EXPTIME-complete EXPTIME-complete

LTL PSPACE-complete PSPACE-complete 2EXPTIME-complete

**Figure 1.** Known results on the complexity in the size of the formula for (2-valued and 3-valued) model
checking (MC), satisfiability (SAT) and generalized model checking (GMC).

that flavor include the*realizability* [1] and*synthesis* [42,43] problems for linear-time
temporal logic specifications.

Figure 1 summarizes the previous complexity results. These results show that the
complexity in the size of the formula of computing[(M,*s)*|=φ]t(GMC) is always higher
than that of computing[(M,*s)*|=φ](MC).

Regarding the complexity in the size of the model|*M*|, it is shown in [6] that
gener-alized model checking for CTL can be solved in time quadratic in|*M*|. For LTL,
gener-alized model checking can be solved in time polynomial in|*M*|[25]. More precisely, the
complexity in|*M*|is

• *linear*for safety (✷*p*) and weak (i.e., recognizable by Deterministic Weak Word

automata) properties;

• *quadratic*for response (✷(p→✸*q*), persistence (✸ ✷*p*) and generalized

reactiv-ity[1] properties [32].

Note that for CTL and LTL, generalized model checking is PTIME-hard in |*M*|while
model checking is NLOGSPACE-complete in|*M*|[18].

**6. How to Generate 3-Valued Abstractions**

In [20], it is shown how to adapt the abstraction mappings of [9] to construct abstractions that are less complete than a given concrete program with respect to the completeness preorder.

**Definition 15** *Let MC*= (S*C*,P,
*must*

−→*C*,
*may*

−→*C*,L*C*)*be a (concrete) KMTS. Given a set SA*
*of abstract states and a total*1*abstraction relation on states*ρ ⊆*SC*×*SA, we define the*
*(abstract) KMTS MA*= (S*A*,P,−→*mustA*,

*may*

−→*A*,*LA)as follows:*
• *a*−→*mustAa*′*if*∀*c*∈*SC*:*c*ρ*a*⇒(∃*c*′∈*SC*:*c*′ρ*a*′∧*c*

*must*

−→*Cc*′);
• *a*−→*mayAa*′*if*∃*c,c*′∈*SC*:*cρa*∧*c*′ρ*a*′∧*c*

*may*

−→*Cc*′*;*

• *LA(a,p) =*

*true if*∀*c*:*c*ρ*a*⇒*LC(c,p) =true*
*false if*∀*c*:*c*ρ*a*⇒*LC(c,p) =false*

⊥ *otherwise*

The previous definition can be used to build abstract KMTSs.

**Theorem 16** *Given a KMTS MC, any KMTS MAobtained by applying Definition 15 is*
*such that MAMC.*

Given a KMTS*MC*, any abstraction*MA*less complete than*MC*with respect to the

com-pleteness preordercan be constructed using Definition 15 by choosing the inverse ofρ
asB_{[20]. When applied to MTSs with}*may*-transitions only, the above definition
coin-cides with traditional “conservative” abstraction that is a*simulation*of the concrete
sys-tem. Building a 3-valued abstraction can be done using existing abstraction techniques
at the*same computational cost*as building a conservative abstraction [20].

**7. Application to Software Model Checking**

The usual procedure for performing program verification via predicate abstraction and iterative abstraction refinement is the following (e.g., see [3,12]).

1. Abstract: compute an abstraction*MA*that simulates the concrete program*MC*.

2. Check: given a universal propertyφ, decide whether*MA*|=φ.
• if*MA*|=φ: stop (the property is proved:*MC*|=φ).

• if*MA*6|=φ: go to Step 3.

3. Refine: refine*MA* (possibly using a counter-example found in Step 2). Then go

to Step 1.

Using predicate abstraction [26,13,50], the abstraction computed in Step 1 is
de-fined with respect to a set of predicatesΨ={ψ1, . . . ,ψn}, which are typically
quantifier-free formulas of first-order logic (for instance,(x==*y*+1)∨(x<*y*−5)). An abstract
state is defined as a vector of*n* bits induced by *n*-ary conjunctions, with each
predi-cateψicontributing eitherψior¬ψi, which identifies all concrete states that satisfy the
same set of predicates inΨ. Thus, a concrete state*c*is abstracted by an abstract state

[c] = (b1, . . . ,*bn)*such that∀1≤*i*≤*n*:*bi*=ψi(c). A transition is defined between

ab-stract states[c1]and[c2]if there exists a transition from some concrete state in[c1]to
some concrete state in[c2]. The resulting abstract transition system*MA*is guaranteed by

construction to simulate the concrete transition system*MC*.

Since*MA*simulates*MC*, one can only prove the correctness of universal properties

(i.e., properties over all paths) of*MC* by analyzing*MA* in Step 2. In particular, the

vi-olation of a universal property (or equivalently, the satisfaction of an existential prop-erty) cannot be established by analyzing such abstractions in general. Step 3 typically in-volves the addition of new predicates to refine the current abstraction. Note that the three steps above can also be interleaved and performed in a strict demand-driven fashion as described in [28].

1. Abstract: compute an abstraction*MA*using Def. 15 such that*MAMC*.

2. Check: given*any*propertyφ,

(a) (3-valued model checking) compute[M*A*|=φ].

• if[M*A*|=φ] =*true*or*false*: stop (the property is proved (resp. disproved)

on*MC*).

• if[M*A*|=φ] =⊥, continue.

(b) (generalized model checking) compute[M*A*|=φ]t.

• if[M*A*|=φ]t=*true*or*false*: stop (the property is proved (resp. disproved)

on*MC*).

• if[M*A*|=φ] =⊥, go to Step 3.

3. Refine: refine*MA*(possibly using a counter-example found in Step 2). Then go

to Step 1.

This new procedure strictly generalizes the traditional one in several ways. First, any
temporal logic formula can be checked (not just universal properties). Second, all
cor-rectness proofs and counter-examples obtained by analyzing any abstraction*MA* such

that*MAMC* are guaranteed to be sound (i.e., hold on*MC*) for any property (by

The-orem 4). Third, verification results can be more precise than with the traditional
proce-dure: the new procedure will not only return*true*whenever the traditional one returns

*true*(trivially, since the former includes the latter), but it can also return*true*more
of-ten thanks to a more thorough check using generalized model-checking, and it can also
return*false*. The new procedure can thus terminate sooner and more often than the
tra-ditional procedure — the new procedure will never loop through its 3 steps more often
than the traditional one.

Remarkably, each of the 3 steps of the new procedure can be performed at roughly
the same cost as the corresponding step of the traditional procedure: as shown in [20],
building a 3-valued abstraction using Definition 15 (Step 1 of new procedure) can be
done at the same computational cost as building a conservative abstraction (Step 1 of
traditional procedure); computing[M*A*|=φ]in Step 2.a can be done at the same cost at

traditional (2-valued) model checking [6]; following the results of Section 5, computing

[M*A*|=φ]t in Step 2.b can be more expensive than Step 2.a, but is still polynomial

(typ-ically linear or quadratic) in the size of*MA*; Step 3 of the new procedure is similar to

Step 3 of the traditional one (in the case of LTL properties for instance, refinement can be guided by error traces found in Step 2 as in the traditional procedure). Finally note that the new procedure could also be adapted so that the different steps are performed in a demand-driven basis following the work of [28].

**8. Examples**

We now give examples of programs, models and properties, all taken from [21], where
computing[(M,*s)*|=φ]t returns a more precise answer than[(M,*s)*|=φ].

program C1(){ program C2(){ program C3(){

x,y = 1,0; x,y = 1,0; x = 1;

x,y = f(x),f(y); x,y = 2*f(x),f(y); x = f(x);

x,y = 1,0; x,y = 1,0; }

} }

(p=T,q=F)

M2

M1 M3

s2

⊥ (p=T) s3 (p=T,q=F)

s2’

s2’’

(p= )
(p= ,q= )_{⊥} _{⊥}

s1

(p=T,q=F) (p=T,q=F)

(p=F,q= )_{⊥}

**Figure 2.** Examples of programs and models

*p*: “is x odd?” and*q*: “is y odd?”. Figure 2 shows an example of KMTS model for each
of the three programs. These models can be computed automatically using Definition 15,
predicate abstraction techniques and predicates *p* and *q*, so that by construction they
satisfy Theorem 16. Each model is a KMTS with*must*-transitions only and with atomic
propositions*p*and*q*whose truth value is defined in each state as indicated in the figure.
Consider the LTL formulaφ1=✸*q*⇒✷(p∨*q)*(where✸is read “eventually” and

✷is read “always” [38]). While[(M1,s1)|=φ1] =⊥,[(M1,*s*1)|=φ1]t =*true*. In other
words, using the thorough interpretation yields a more definite answer in this case. Note
that the gain in precision obtained in this case is somewhat similar to the gain in precision
that can be obtained using an optimization called*focusing*[3] aimed at recovering some
of the imprecision introduced when using*cartesian abstraction*(see [3,20]).

Consider now the formulaφ2=✸*q*∧✷(p∨ ¬*q)*evaluated on(M2,*s*2). In this case,
we have [(M2,*s*2)|=φ2] =⊥, while [(M2,*s*2)|=φ2]t =*false*. Again, using the
thor-ough interpretation yields a more definite answer, alththor-ough solving a generalized
model-checking problem is necessary to return a negative answer. Indeed, one needs to prove
in this case that there exists a computation of(M2,*s*2)(namely*s*2*s*′2*s*′′2ω – there is only
one computation in this simple example) that does not have any completion satisfying

φ2, which itself requires using alternating automata and can thus be more expensive as
discussed in Section 5. Another example of formula isφ_{2}′=*q*∧✷(p∨ ¬*q)*(where_{is}

read “next” [38]). Again we have that[(M2,s2)|=φ2′] =⊥, while[(M2,*s*2)|=φ2′]t=*false*.
Note that, althoughφ_{2}′is an LTL safety formula and hence is within the scope of analysis
of existing tools ([4], [8], etc.), none of these tools can prove thatφ_{2}′does not hold: this
result can only be obtained using generalized model checking.

Last, consider(M3,*s*3)and formulaφ3=✷*p*. In this case, we have both[(M3,*s*3)|=

**9. Precision of GMC Vs. MC**

How often is generalized model checking (GMC) more precise than model checking
(MC)? This question is addressed in [19]. Specifically, [19] studies when it is possible
to reduce GMC(*M,*φ) to MC(*M,*φ′). Such a transformed formulaφ′is called a*semantic*
*minimization*ofφ. [19] shows that propositional logic, PML and the propositional µ
-calculus are*closed under semantic minimization*, i.e., that a reduction from GMC(*M,*φ)
to MC(*M,*φ′) is always possible for φ andφ′ in propositional logic, PML or the µ
-calculus. But in contrast, the temporal logics LTL, CTL and CTL∗are*not*closed under
semantic minimization.

[19] also identifies*self-minimizing*formulas, i.e., formulasφfor which GMC(*M,*φ)
and MC(*M,*φ) are equivalent. By definition, GMC and MC have thus the same precision
for any self-minimizing formula. Self-minimizing formulas can be defined both
seman-tically using automata-theoretic techniques (for instance, this is EXPTIME-hard in|φ|
for theµ-calculus) and syntactically by providing syntactic*sufficient*criteria which are
linear in|φ|. For instance, [19] shows that*any formula that does not contain any atomic*
*proposition in mixed polarity (in its negation normal form) is self-minimizing*.

Fortunately, in practice, many frequent formulas are self-minimizing, and MC is as precise as GMC for those.

**10. Other Related Work**

The framework presented in the previous sections has also been extended to*open *
*sys-tems* [18] (i.e., systems which interacts with their environment), and to*games*in
gen-eral [14]. For instance, [14] studies*abstractions of games*where moves of each player
can be abstracted while preserving winning strategies of *both*players. An abstraction
of a game is now a game where each player has both may and must moves, yielding
may/must strategies. In this context, the completeness preorder becomes an*alternating*
*refinement*relation, logically characterized by 3-valued alternatingµ-calculus [2].

Another interesting topic is*semantic completeness*: given an infinite-state system*C*

and propertyφ, if*C*satisfiesφ, does there exist a finite-state abstraction*A*of*C*such that

*A*satisfiesφ?

For arbitrary formulasφ of LTL, the existence of such finite abstractions*A*can be
guaranteed provided that abstractions*A*are extended to include the modeling of
*fair-ness*constraints [33], which are used to model termination in loops. For arbitrary
for-mulasφ of the propositionalµ-calculus (hence including existential properties), the
ex-istence of such finite abstractions can again be guaranteed but now provided that
ab-stractions*A*may include*nondeterministic*must transitions [37], also called*hyper-must*

**11. Concluding Remarks**

This paper presents an introduction to 3-valued “may/must” abstraction-based software model checking for sound property verification and falsification. The results presented here previously appeared in a series of papers [5,6,20,21,22,18,14,19,25]. These results shed light on the techniques used in abstraction-based software model checking tools like SLAM [4], BLAST [28], YASM [27], TERMINATOR [7] and YOGI [24]. In partic-ular, YASM [27] uses 3-valued models as described in this paper, while YOGI [24] uses (compositional) may/must abstractions that share transitions instead of states.

The reader interested in the topic of this paper should consult the references listed above, as well as the related work discussed in those references. We mention below only a few other main pointers to related work.

The study of abstraction for model checking of both universal and existential
pro-gram properties was pioneered in [9,10]. This work defines a general abstraction
frame-work where abstractions are*mixed transition systems*. Intuitively, a mixed transition
sys-tem is a modal transition syssys-tem without the constraint−→⊆*must* −→. Mixed transition sys-*may*
tems are more expressive and, in full generality, allow for a 4-valued world where some
mixed transition systems cannot be refined by any complete (2-valued) system [31].
Nev-ertheless, the goal and some of the results of this prior work are very similar to our
own work with 3-valued models and logics. The use of “conservative” abstractions for
proving properties of the fullµ-calculus is also discussed in [45].

Extended transition systems [39] are Labeled Transition Systems extended with a

*divergence predicate*, and can be viewed as a particular class of 3-valued models [5,30].
In [5], it is shown that Hennessy-Milner Logic with a 3-valued interpretation provides
an alternative characterization of the divergence preorder in addition to the intuitionistic
interpretation of Plotkin [48]. Further work on divergence preorders and logics to
char-acterize them can be found in [48,51]. In all this work, logic formulas are interpreted
normally in the 2-valued sense. The close correspondence between 3-valued logic and
Plotkin’s intuitionistic modal logic inspired the reduction procedure from 3-valued model
checking to 2-valued model checking of [6] (see Section 4).

Prior to the work reported here, most work on 3-valued modal logic focused on its proof theory (e.g., [46,15]). Our definition of partial Kripke structure is closest to [16], where two interpretations of modal logic are presented: a many-valued version and another version based on obtaining 2-valued interpretations from each of a set of experts. [16] shows that such a multi-expert interpretation corresponds in a precise way to a multi-valued interpretation, similarly to how we show that a 3-valued interpretation can be obtained by separate optimistic and pessimistic interpretations. However, [16] does not define a completeness preorder over models or characterization results.

(transi-tion) systems and the sound verification (and falsifica(transi-tion) of*temporal*properties of such
systems.

**Acknowledgements.**This paper covers one of the lectures (Lecture 4) which I gave
at the 2013 Marktoberdorf Summer School. I thank the organizers of the Summer School
for encouraging me to write this paper. I also thank my co-authors (in chronological
order) Glenn Bruns, Radha Jagadeesan, Michael Huth, Luca de Alfaro, and Nir Piterman
for their insights and without whom this work would not exist.

**References**

[1] M. Abadi, L. Lamport, and P. Wolper. Realizable and unrealizable concurrent program specifications.
In*Proc. 16th Int. Colloquium on Automata, Languages and Programming*, volume 372 of*Lecture Notes*
*in Computer Science*, pages 1–17. Springer-Verlag, July 1989.

[2] R. Alur, T.A. Henzinger, and O. Kupferman. Alternating time temporal logic. *Journal of the ACM*,
49:672–713, 2002.

[3] T. Ball, A. Podelski, and S. K. Rajamani. Boolean and Cartesian Abstraction for Model Checking C
Programs. In*Proceedings of TACAS’2001 (Tools and Algorithms for the Construction and Analysis of*
*Systems)*, volume 2031 of*Lecture Notes in Computer Science*. Springer-Verlag, April 2001.

[4] T. Ball and S. Rajamani. The SLAM Toolkit. In*Proceedings of CAV’2001 (13th Conference on *
*Com-puter Aided Verification)*, volume 2102 of*Lecture Notes in Computer Science*, pages 260–264, Paris,
July 2001. Springer-Verlag.

[5] G. Bruns and P. Godefroid. Model Checking Partial State Spaces with 3-Valued Temporal Logics. In

*Proceedings of CAV’99 (11th Conference on Computer Aided Verification)*, volume 1633 of*Lecture*
*Notes in Computer Science*, pages 274–287, Trento, July 1999. Springer-Verlag.

[6] G. Bruns and P. Godefroid. Generalized Model Checking: Reasoning about Partial State Spaces. In

*Proceedings of CONCUR’2000 (11th International Conference on Concurrency Theory)*, volume 1877
of*Lecture Notes in Computer Science*, pages 168–182, University Park, August 2000. Springer-Verlag.
[7] B. Cook, A. Podelski, and A. Rybalchenko. Termination Proofs for Systems Code. In*Proceedings of*
*PLDI’2006 (ACM SIGPLAN 2006 Conference on Programming Language Design and Implementation)*,
pages 415–426, Ottawa, June 2006.

[8] J. C. Corbett, M. B. Dwyer, J. Hatcliff, S. Laubach, C. S. Pasareanu, Robby, and H. Zheng. Bandera:
Extracting Finite-State Models from Java Source Code. In*Proceedings of the 22nd International *
*Con-ference on Software Engineering*, 2000.

[9] D. Dams.*Abstract interpretation and partition refinement for model checking*. PhD thesis, Technische
Universiteit Eindhoven, The Netherlands, 1996.

[10] D. Dams, R. Gerth, and O. Grumberg. Abstract interpretation of reactive systems. *ACM Transactions*
*on Programming Languages and Systems*, 19(2):253–291, 1997.

[11] D. Dams and K. Namjoshi. The existence of finite abstractions for branching time model checking. In

*Proceedings of LICS’2004 (19th IEEE conference on Logic in Computer Science)*, Turku, July 2004.
[12] S. Das and D. L. Dill. Successive Approximation of Abstract Transition Relations. In*Proceedings of*

*LICS’2001 (16th IEEE Symposium on Logic in Computer Science)*, pages 51–58, Boston, June 2001.
[13] S. Das, D. L. Dill, and S. Park. Experience with Predicate Astraction. In*Proc. of the 11th *

*Interna-tional Conference on Computer-Aided Verification*, Lecture Notes in Computer Science, pages 160–172,
Trento, July 1999. Springer Verlag.

[14] L. de Alfaro, P. Godefroid, and R. Jagadeesan. Three-Valued Abstractions of Games: Uncertainty, but
with Precision. In*Proceedings of LICS’2004 (19th IEEE Symposium on Logic in Computer Science)*,
pages 170–179, Turku, July 2004.

[15] M. Fitting. Many-Valued Modal Logics I.*Fundamenta Informaticae*, 15:235–254, 1992.
[16] M. Fitting. Many-Valued Modal Logics II.*Fundamenta Informaticae*, 17:55–73, 1992.

[19] P. Godefroid and M. Huth. Model Checking Vs. Generalized Model Checking: Semantic Minimizations
for Temporal Logics. In*Proceedings of LICS’2005 (20th IEEE Symposium on Logic in Computer*
*Science)*, pages 158–167, Chicago, June 2005.

[20] P. Godefroid, M. Huth, and R. Jagadeesan. Abstraction-based Model Checking using Modal Transition
Systems. In*Proceedings of CONCUR’2001 (12th International Conference on Concurrency Theory)*,
volume 2154 of*Lecture Notes in Computer Science*, pages 426–440, Aalborg, August 2001.
Springer-Verlag.

[21] P. Godefroid and R. Jagadeesan. Automatic Abstraction Using Generalized Model Checking. In
*Pro-ceedings of CAV’2002 (14th Conference on Computer Aided Verification)*, volume 2404 of*Lecture Notes*
*in Computer Science*, pages 137–150, Copenhagen, July 2002. Springer-Verlag.

[22] P. Godefroid and R. Jagadeesan. On the Expressiveness of 3-Valued Models. In*Proceedings of *
*VM-CAI’2003 (4th Conference on Verification, Model Checking and Abstract Interpretation)*, volume 2575
of*Lecture Notes in Computer Science*, pages 206–222, New York, January 2003. Springer-Verlag.
[23] P. Godefroid, N. Klarlund, and K. Sen. DART: Directed Automated Random Testing. In*Proceedings of*

*PLDI’2005 (ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation)*,
pages 213–223, Chicago, June 2005.

[24] P. Godefroid, A.V. Nori, S.K. Rajamani, and S.D. Tetali. Compositional May-Must Program
Analy-sis: Unleashing The Power of Alternation. In*Proceedings of POPL’2010 (37th ACM Symposium on*
*Principles of Programming Languages)*, pages 43–55, Madrid, January 2010.

[25] P. Godefroid and N. Piterman. LTL Generalized Model Checking Revisited. *International Journal on*
*Software Tools for Technology Transfer (STTT)*, 13(6):571–584, 2011.

[26] S. Graf and H. Saidi. Construction of Abstract State Graphs with PVS. In*Proceedings of the 9th*
*International Conference on Computer Aided Verification*, volume 1254 of*Lecture Notes in Computer*
*Science*, pages 72–83, Haifa, June 1997. Springer-Verlag.

[27] A. Gurfinkel, O. Wei, and M. Chechik. Yasm: A Software Model Checker for Verification and
Refuta-tion. In*Proceedings of CAV’2006 (18th Conference on Computer Aided Verification)*, volume 4144 of

*Lecture Notes in Computer Science*, pages 170–174, Seattle, August 2006. Springer-Verlag.

[28] T. Henzinger, R. Jhala, R. Majumdar, and G. Sutre. Lazy Abstraction. In*Proceedings of the 29th ACM*
*Symposium on Principles of Programming Languages*, pages 58–70, Portland, January 2002.
[29] G. J. Holzmann and M. H. Smith. A Practical Method for Verifying Event-Driven Software. In

*Pro-ceedings of the 21st International Conference on Software Engineering*, pages 597–607, 1999.
[30] M. Huth, R. Jagadeesan, and D. Schmidt. Modal Transition Systems: a Foundation for Three-Valued

Program Analysis. In*Proceedings of the European Symposium on Programming (ESOP’2001)*, volume
2028 of*Lecture Notes in Computer Science*. Springer-Verlag, April 2001.

[31] M. Huth, R. Jagadeesan, and D. Schmidt. A Domain Equation for Refinement of Partial Systems. Submitted to Mathematical Structures in Computer Science, 2002.

[32] Y. Kesten, N. Piterman, and A. Pnueli. Bridging the gap between fair simulation and trace containment.
In*15th Computer Aided Verification*, volume 2725 of*Lecture Notes in Computer Science*, pages 381–
393. Springer-Verlag, 2003.

[33] Y. Kesten and A. Pnueli. Verification by Augmented Finitary Abstraction. *Information and *
*Computa-tion*, 163(1), 2000.

[34] S. C. Kleene.*Introduction to Metamathematics*. North Holland, 1987.

[35] D. Kozen. Results on the Propositional Mu-Calculus.*Theoretical Computer Science*, 27:333–354, 1983.
[36] K. G. Larsen and B. Thomsen. A Modal Process Logic. In*Proceedings of Third Annual Symposium on*

*Logic in Computer Science*, pages 203–210. IEEE Computer Society Press, 1988.

[37] K. G. Larsen and Liu Xinxin. Equation solving using modal transition systems. In*Proceedings of the*
*5th IEEE conference on Logic in Computer Science*, pages 108–117. IEEE, 1990.

[38] Z. Manna and A. Pnueli. *The Temporal Logic of Reactive and Concurrent Systems: Specification*.
Springer-Verlag, 1992.

[39] R. Milner. A Modal Characterization of Observable Machine Behavior. In*Proc. CAAP’81*, volume 112
of*Lecture Notes in Computer Science*, pages 25–34. Springer-Verlag, 1981.

[40] K. S. Namjoshi. Abstraction for branching time properties. In*Proceedings of CAV’2003 (15th *
*Interna-tional Conference on Computer Aided Verification)*, volume 2725 of*Lecture Notes in Computer Science*,
pages 288–300. Springer, 2003.

[41] K. S. Namjoshi and R. K. Kurshan. Syntactic Program Transformations for Automatic Abstraction. In

*Computer Science*, pages 435–449, Chicago, July 2000. Springer-Verlag.

[42] A. Pnueli and R. Rosner. On the synthesis of a reactive module. In*Proc. of the Sixteenth Symposim on*
*Principles of Programming Languages*, Austin, January 1989.

[43] A. Pnueli and R. Rosner. On the synthesis of an asynchronous reactive module. In*Proceedings of*
*ICALP’89*, Stresa, July 1989.

[44] M. Sagiv, T. Reps, and R. Wilhelm. Parametric Shape Analysis Via 3-Valued Logic. In*Proceedings of*
*the 26th ACM Symposium on Principles of Programming Languages*, January 1999.

[45] H. Saidi and N. Shankar. Abstract and model check while you prove. In*Proc. of the 11th Conference*
*on Computer-Aided Verification*, number 1633 in Lecture Notes in Computer Science, pages 443–454.
Springer, 1999.

[46] K. Segerberg. Some Modal Logics Based on a Three-Valued Logic.*Theoria*, 33:53–71, 1967.
[47] S. Shoham and O. Grumberg. Monotonic Abstraction-Refinement for CTL. In*Tools and Algorithms*

*for the Construction and Analysis of Systems: 10th International Conference*, number 2988 in Lecture
Notes in Computer Science, pages 546–560. Springer Verlag, 2004.

[48] C. Stirling. Modal Logics for Communicating Systems. *Theoretical Computer Science*, 49:331–347,
1987.

[49] W. Visser, K. Havelund, G. Brat, and S. Park. Model Checking Programs. In*Proceedings of ASE’2000*
*(15th International Conference on Automated Software Engineering)*, Grenoble, September 2000.
[50] W. Visser, S. J. Park, and J. Penix. Using Predicate Abstraction to Reduce Object-oriented Programs

for Model Checking. In*Proceedings of FMSP’00 (Formal methods in Software Practice)*, pages 3–12,
Portland, August 2000.

### SAT-based Model Checking:

### Interpolation, IC3, and Beyond

Orna GRUMBERGa_{, Sharon SHOHAM}b_{and Yakir VIZEL}a
a

Computer Science Department, Technion, Haifa, Israel b

School of Computer Science, Academic College of Tel Aviv-Yaffo

Abstract.

SAT-based model checking is currently one of the most successful ap-proaches to checking very large systems. In its early days, SAT-based (bounded) model checking was mainly used for bug hunting. The intro-duction ofinterpolation and IC3\PDRenable efficient complete algo-rithms that can provide full verification as well.

In this paper, we survey several approaches to enhancing SAT-based model checking. They are all based on iteratively computing an over-approximationof the set of reachable system states. They use different mechanisms to achieve scalability and faster convergence.

The first one usesinterpolation sequencerather than interpolation in order to obtain a more precise over-approximation of the set of reachable states. The other approach integrates lazy abstraction with IC3 in order to achieve scalability.Lazy abstraction, originally developed for software model checking, is a specific type of abstraction that allows hiding dif-ferent model details at difdif-ferent steps of the verification. We find the IC3 algorithm most suitable for lazy abstraction since its state traversal is performed by means oflocal reachability checks, each involving only two consecutive sets. A different abstraction can therefore be applied in each of the local checks.

The survey focuses on hardware model checking, but the presented ideas can be extended to other systems as well.

Keywords.Model Checking, SAT-based Model Checking, Interpolation, Interpolation Sequence, Bounded Model Checking (BMC), IC3, Unbounded Model Checking, Abstraction, Lazy abstraction, Hardware model checking

1. Introduction

Computerized systems dominate almost every aspect of our lives and their cor-rect behavior is essential. Model checking [8, 10, 28] is an automated verification technique for checking whether a given system satisfies a desired property. The system is usually described as a finite-state model in a form of a state transition graph. The specification is given as a temporal logic formula. Unlike testing or simulation based verification, model checking tools are exhaustive in the sense that they traverseall behaviors of the system, and either confirm that the system behaves correctly or present a counterexample.

*IOS Press, 2014 *

Model checking has been successfully applied to verifying hardware and soft-ware systems. Its main limitation, however, is the state explosion problem which arises due to the huge state space of real-life systems. The size of the model in-duces high memory and time requirements that may make model checking not applicable to large systems. Much of the research in this area is dedicated to increase model checking applicability and scalability.

The first significant step in this direction was the introduction of BDDs [4] into model checking. BDD-based Symbolic Model Checking (SMC) [5] enabled model checking of real-life hardware designs with a few hundreds of state ele-ments. However, current design blocks with well-defined functionality typically have thousands of state elements and more. To handle designs of that scale, SAT-based Bounded Model Checking (BMC) [2] has been developed. Its main draw-back, however, is its orientation towards ”bug-hunting” rather than full verifica-tion.

Several approaches have been suggested to remedy the problem. Induc-tion[29],interpolation[22],interpolation sequence[6,32],IC3/PDR[3,12], and L-IC3 [31] developed different techniques for SAT-basedUnbounded Model Checking (UMC), which provide full verification.

Of these SAT-based unbounded model checking techniques, L-IC3 and [6] also useAbstraction-refinement [9], which is another well known methodology for tackling the state-explosion problem. Abstraction hides model details that are not relevant for the checked property. The resulting abstract model is then smaller, and therefore easier to handle by model checking algorithms.Lazy abstraction[15, 23], developed for software model checking, is a specific type of abstraction that allows hiding different model details at different steps of the verification.

In this paper we discuss four of the above mentioned SAT-based approaches for full verification. These methods all compute an over-approximated set of the system’s reachable states while checking that the specification is not violated. The first two approaches we describe are taken from [22] and [32]. They combine BMC withinterpolation[11] orinterpolation-sequence [16,23] respectively. These methods use an unrolling of the model’s transition relation in order to traverse the system’s state space. The third algorithm we discuss is IC3/PDR [3, 12]. In contrast to the first two methods, IC3 avoids unrolling of the transition relation. Instead, the computation of reachable states useslocal reachability checksbetween consecutive time frames. Last, we present L-IC3 [31] which provides a SAT-based lazy abstraction-refinement algorithm based on IC3/PDR. L-IC3 uses the visible variables abstraction [18], which is particularly suitable for hardware systems. However, the abstraction is used in a lazy manner in the sense that different sets of visible variables are used in different iterations of the state-space traversal.

2. Preliminaries

Temporal logic model checking [10] is an automatic approach to formally verifying that a given system satisfies a given specification. The system is often modelled by a finite state transition system and the specification is written in a temporal logic. Determining whether a model satisfies a given specification is often based on an exploration of the model’s state space in a search for violations of the specification.

In this survey we focus on hardware. As such we consider finite state transition systems defined over Boolean variables, as follows.

Definition 2.1. A finite state transition system (a model) is a tuple M = (V, U,INIT,TR) where V is a set of Boolean variables, U ⊆V is a set of state variables,V\U is a set of input variables,INIT(V) is a propositional formula over V describing the initial states, andTR(V, V′) describes a total transition relation which is defined as a propositional formula over V and the next-state variables V′ ={v′|v∈V}.

The transition relation is described using next-state functions for each state
variable. Namely, TR(V, V′_{) =} V

v∈U

(v′ _{=}_{f}_{v(}_{V, V}′_{)) where} _{f}_{v(}_{V, V}′_{) is a }
proposi-tional formula that assigns the next value to v ∈U based on current and
next-state variables. Note that for an input variablev∈V \U,fv is not defined.

The set of Boolean variables ofM induces a set of statesS={0,1}|V|_{, where}
each state s ∈ S is given by a valuation of the variables in V. A formula over
V (resp.V, V′_{) represents the set of states (resp. pairs of states) obtained by its}
satisfying assignments. With abuse of notation we will refer to a formula η over
V as a set of states and therefore use the notions∈η for states represented by
η. Similarly for a formulaη overV, V′, we will sometimes write (s, s′)∈η.

The formula η[V ← V′], or η′ in short, is identical to η except that each
variablev ∈V is replaced with v′. In the general case Vi _{is used to denote the}
variables inV afteritime units (thus,V0_{≡}_{V}_{). Let}_{η} _{be a formula over}_{V}i_{, the}
formulaη[Vi _{←} _{V}j_{] is identical to} _{η} _{except that for each variable} _{v} _{∈} _{V}_{,} _{v}i _{is}
replaced withvj_{.}

A path in M is a sequence of states π = s0, s1, . . . such that for all i ≥ 0, si∈S and (si, si+1)∈TR. The length of a path is denoted by|π|. Ifπis infinite then|π| =∞. Ifπ =s0, s1, . . . , sn then |π|=n. A path is aninitial path when s0∈INIT. We sometimes refer to a prefix of a path as a path as well.

A formula inLinear Temporal Logic(LTL) [10, 27] is of the formAf wheref is a path formula. A model M satisfies an LTL propertyAf if all infinite initial paths inM satisfyf. If there exists an infinite initial path not satisfyingf, this path is defined to be acounterexample.