Staged Evaluation of Partial-Instances in Alloy

Staged Evaluation of Partial-Instances in Alloy

4th International ABZ 2014 Conference June 2 - 6 2014

Derek Rayside [email protected]

Edward Zulkoski [email protected]

Vajih Montaghami [email protected]

Authors: Presented By:

[email protected] Staged Evaluation of Partial-Instances in Alloy

Alloy Analyzer is Sound ‣ No spurious counter-examples

‣ Reported counter-examples are logically correct


Surprises still happen ‣ Users do not always understand Alloy correctly

‣ Some counter-examples are perceived as spurious (even though they are not spurious)

‣ ∀∃ queries are one source of these surprises

- We propose the uniq keyword to help users in this case

- uniq ⇒ generator axiom + uniqueness predicate

-  Staged evaluation makes it run faster


∀∃ Queries ‣ Not commonly used in the past

‣ More uses proposed in recent work, such as:

-  Bounded model checking of temporal formulas with Alloy [Cunha @ ABZ’14]

-  Temporal logic model-checking in Alloy [Vakili & Day @ ABZ’12]

-  Modeling the supervisory control theory with Alloy [Fraikin, Frappier, St-Denis @ ABZ’12]

-  Feature-Oriented Modeling Language (FORML) [Shaker, Dietrich, Gorzny, Atlee]


Analyzing FOL in finite universe

‣ Small scope Analysis: Errors mostly occur in a small domain.

‣ Finite Universe: Exhaustively Searching for an instance in a bound universe

‣ NO Spurious counter-examples.

-  if a problem is reported, it’s real

- might miss some problems due to limited scope

5


Lack of well-declared universe

‣ How a universe has to be determined.

‣ What happens if a universe is not well declared.

‣ Unbound quantifier!

-  formulas of the form ∀∃ over composite structure produce spurious counter-examples unless accompanied by a generator axiom.

-  Small scope software model checking.

6


Why spurious counter-example?

‣ The universe does not includes all atoms

-  Infinite objects without solution

-  If we finitize the objects, the problem can be handled.

‣ Insufficient Bound

- Default scope is 3, there are 51 non-isomorphic RBT for 4 distinct nodes.

- Upper-bound VS. Exact-bound

‣ Also, there might be duplicate sorted-sets

7


Transition System Example

‣ States are subsets of {p,q}

‣ Intended transition system:

‣ Property to be checked: EF({p,q})

8


An Unpleasant Surprise ‣ Intended transition system:

‣ Property to be checked: EF({p,q})

9

‣ Counter-example:


Generate All States


User Challenges 1.  Write Generator Axiom (and Uniqueness fact)

2.  Guess proper bounds

3.  Solve efficiently

‣ We just saw how the uniq keyword addresses the first challenge. Now we’ll look at the next two.


Finding Bounds ‣ uniq keyword may only be used on sigs with fields that

range over other uniq/enum/one sigs, e.g.:

‣ uniq must be used in a partial order (no cycles)

‣ So the maximum potential number of atoms for a sig is easy to compute


Spurious counter-example?!

13

enum Label {P,Q}!sig S{next: set S, lbl:set Label}"one sig Init in S {}{no lbl}!fact sigma {" all s,s':S | " #(s'.lbl) = plus[1,#(s.lbl)] <=> s' in s.next}"fact gen {""all l: Label | some s: S | l = s.lbl""all s: S | some s': S-s | s.lbl!=Label => s.lbl in s'.lbl}"

fact unique {" all s,s': S | s.lbl = s'.lbl => s=s'}"-- EF [P + Q]!check { all i : Init | some s : S | " i in ^next.s and s.lbl = Label } for exactly 4 S"



14

enum Label {P,Q}!sig S{next: set S, lbl:set Label}"one sig Init in S {}{no lbl}!fact sigma {" all s,s':S | " #(s'.lbl) = plus[1,#(s.lbl)] <=> s' in s.next}"-- EF [P + Q]!check { all i : Init | some s : S | " i in ^next.s and s.lbl = Label } for 4 S"

✗Similar States like:



15

enum Label {P,Q}!sig S{next: set S, lbl:set Label}"one sig Init in S {}{no lbl}!fact sigma {" all s,s':S | " #(s'.lbl) = plus[1,#(s.lbl)] <=> s' in s.next}"fact unique {" all s,s': S | s.lbl = s'.lbl => s=s'}"-- EF [P + Q]!check { all i : Init | some s : S | " i in ^next.s and s.lbl = Label } for 4 S"

✗Incomplete Transitions:


enum Label {P,Q}!sig S{next: set S, lbl:set Label}"one sig Init in S {}{no lbl}!fact sigma {" all s,s':S | " #(s'.lbl) = plus[1,#(s.lbl)] <=> s' in s.next}"fact gen {""all l: Label | some s: S | l = s.lbl""all s: S | some s': S-s | s.lbl!=Label => s.lbl in s'.lbl}"

fact unique {" all s,s': S | s.lbl = s'.lbl => s=s'}"-- EF [P + Q]!check { all i : Init | some s : S | " i in ^next.s and s.lbl = Label } for exactly 4 S"


16

✓No counter-example

Either


Making universe ‣ Generator axioms: a formula that forces certain atoms

to exist in the universe of discourse

‣ Unique Predicate: User has to write the predicate to define the object identity for a given class.

- Atom identity VS Object identity

- Object’s identity is defined by the fields’ value

17


What users do now ‣ Write generator axiom by hand

‣ Write unique predicate by hand

‣ Guess the necessary bounds

‣ Solve generator & formula simultaneously

‣  ... difficult, error prone, computationally inefficient, hard to maintain

18


New approach

19

uniq sig S {r1: Ti, …}{ϕ} !

check { ∀ s:S | ∃ sʼ:S | π(s,sʼ)} "

‣ Introduce new uniq keyword

-  Finds tight bound

- Ensure all valid objects are

•  in the universe

• unique

‣ Mechanical and Localized generator axioms


Solving w/Gen. Axioms

Generator Axiom +

Property

Generator Axiom

+

Property

Non-staged Staged vs

Save the generated atoms as a partial instance for input to property check


Two Techniques ‣ SAT Solver Enumeration

‣ Feature of MiniSAT etc.

‣ Exposed by Kodkod (and Alloy)

‣ Find another solution, re-using learning from current solution

‣ Incremental Growth

‣ Guess a small bound

‣ Solve, save result

‣ Increase bound slowly

‣ Solve again, grow result

‣ Repeat until unsat

SAT Enumeration Incremental Growth


Staged Evaluation ‣ Two or more stages evaluation

- Generating universe of discourse

- Checking property

‣ Making the universe of an stage by finding the instance from previous stage

- Any intermediate instance is consistent

- No backtracking

22


Two techniques ❖ Staged-solving using Kodkod

-  Stage 1: Incrementally making the universe of discourse

• Systematically changing the bounds

• Partially encoding a part of universe

-  Stage 2: Check the property

23

❖ Staged-solving using a SAT solver

-  Stage 1: SAT solver finds the next consistent instance

• SAT-Solver Keeps the learned clauses

-  Stage 2: Check the property


Preliminary Evaluation ‣ Comparing three methods:

1.  Non-staged solving

2.  Staged-solving using Kodkod

3.  Staged-solving using a SAT solver

‣ Checking properties in the form of ∀∃ over the following specifications

- Address Book Example[Jackson2012]

-  Sorted Linked-List, Binary Search Tree, Red-Black Tree, with 2..7 Nodes

-  FORML[Dietrich2012]

24


Benchmarks ‣ Data structures:

- Check that insert + delete operations work properly

-  Sorted linked-list, Binary Search Tree, Red-Black tree

-  Sizes ranging from 2—7 nodes

‣ Address Book [modified version of standard example]

‣ FORML (Feature Oriented Requirements Language) [Dietrich, Shaker, Atlee, Gorzny, et alia]


Preliminary Evaluation

26

1. Non-staged solving 2. Staged-solving using Kodkod 3. Staged-solving using a SAT solver


Summary of Results

27

0 5 10 15 20

Non-staged

Incremental Growth

SAT Enumeration

Completed Translation TimeOut Solving TimeOut Generation TimeOut Complete data in paper


Conclusion ‣ uniq keyword is an easier way for the user to write a

generator axiom, and it facilitates staging

‣ Splitting atom generation and property checking into separate stages improves performance and scalability

‣ SAT Enumeration is the best technique for the atom generation stage

‣ uniq + staging support emerging uses of ∀∃ queries


Challenges

‣ Complexity: Harder than NP-complete problems

‣ Enforcing to generate unreasonably high instance

‣ Randomly breaking symmetry

29


Improvement

‣ Statically infer what statement needs generator axioms

‣ Symmetry breaking in the context of generator axioms

30

Software

Staged Evaluation of Partial-Instances in Alloy