# Artifact for Diaframe: Automated Verification of Fine-Grained Concurrent Programs in Iris # Getting Started Guide This is the artifact accompanying Diaframe. It contains the source-code of Diaframe (`diaframe-artifact-source.tgz`) as well as relevant dependencies. It also contains a virtual machine (`diaframe-artifact-vm.ova`) where this source-code and dependencies have been pre-installed. Pick a version that suits you, then follow the corresponding instructions below. ## Use the virtual machine (`diaframe-artifact-vm.ova`) To start the virtual machine, you need virtualization software. We recommend [Virtual Machine Manager](https://virt-manager.org/)(tested with version 2.2.1) or [VirtualBox](https://www.virtualbox.org/wiki/Downloads). VirtualBox can work with `.ova` files directly, to use Virtual Machine Manager one needs to convert this to a different format - see the end of this document. For the VM, we recommend 4 GB of RAM, although 2GB might also work. If you want to verify files in parallel we recommend 8GB of RAM. This VM contains all dependencies of Diaframe, as well as CoqIDE. To check that you have a functional installation, first open CoqIDE (fourth sidebar icon from the top, or press Start and type 'coq'). Now open (ctrl+O), for example, `spin_lock.v`, located in `~/Documents/paper-auto-iris-artifact/build/diaframe/theories/examples/comparison/`. Press Ctrl+End or the 'Go to end' arrow to run the verification. All (non-empty) lines should become green. To run this verification from the command line, open the terminal and navigate to `~/Documents/paper-auto-iris-artifact/build/diaframe/`. First run ``` eval $(opam env) ``` to make opam functional. Then run ``` touch theories/examples/comparison/spin_lock.v && make theories/examples/comparison/spin_lock.vo ``` to force recompilation/reverification of the spin lock. This should at least print ``` COQC theories/examples/comparison/spin_lock.v ``` and terminate succesfully without printing errors. The user of the VM image is 'diaframe' with password 'diaframe'. To change the keyboard layout, press the cog/settings logo, go to 'Region & Language' and add your desired keyboard layout with the '+' button. ## Build locally from source (`diaframe-artifact-source.tgz`) (Tested on 64-bits Ubuntu 20.04, and on a not very recent macbook) Building Diaframe from source requires a functioning (recent, i.e. >= 2.0.0) version of `opam`. To check whether this is the case, you can run `opam --version`. If you do not have a recent version of opam, follow [the instructions here](https://opam.ocaml.org/doc/Install.html). For macOS, we recommend installing the `gtime` command with `brew install gnu-time`. You can build the artifact by running the following commands: ``` tar xf diaframe-artifact-source.tgz cd diaframe-artifact-source ./build_artifact.sh ``` This will install all dependencies locally, compile Diaframe, and verify all but 3 of the examples listed in the paper (these 3 excluded examples take a while to verify, see the Step-by-Step guide if you wish to reverify all examples). Depending on your system, `opam` might ask you to install additional system dependencies like `libgmp-dev`. Among other things, `./build_artifact.sh` creates a local opam switch in the current directory. To make opam able to use this local switch, run ``` eval $(opam env) ``` After this, you can check that Diaframe is functional by running ``` touch theories/examples/comparison/spin_lock.v && make theories/examples/comparison/spin_lock.vo ``` to force recompilation/reverification of the spin lock. This should at least print ``` COQC theories/examples/comparison/spin_lock.v ``` and terminate succesfully without printing errors. ## Overview of the artifact The source code of the artifact can be found in the `diaframe` folder of the source code artifact, at `~/Documents/paper-auto-iris-artifact/build/diaframe` in the virtual machine artifact. This directory contains the following relevant subdirectories: - `theories/examples/comparison` contains all examples mentioned in the evaluation section of the paper. - `theories/examples/paper` contains the two examples shown in Section 2 of the paper. - `theories/steps`, `theories/hint_search`, `theories/symb_exec` contain the formalization of Diaframe's proof search strategy, the hint search procedure and rules for symbolic execution. - `theories/heap_lang` contains hints and specifications for Iris's default language `heap_lang`. - `theories/lib` contains various libraries of hints used by some of the examples. - `utils` contains various scripts and other utilities, independent of the Coq formalization. - `tutorial` contains two example verifications with some exercises. This should help you use Diaframe on your own programs. For more information on the source code of Diaframe, see `diaframe/README.md`. To get started easily, we have included all Coq dependencies of Diaframe. These can be found in the `coq-stdpp`, `coq-iris` and `coq-iris-heap-lang` folders, and are **not** part of the artifact. `appendix.pdf` contains the appendix for the paper, `table1_updated.png` contains a screenshot of the updated numbers of Table 1. ## Using Diaframe If you want to take a shot at using Diaframe to verify some program, these are the steps you need to take. They are discussed in more detail below. 1. Create a new file in `diaframe/theories/examples` 2. Import heap_lang and Diaframe (+ hint libraries) 3. Define your program in Iris's default language, heap_lang 4. Write down invariants and specifications 5. Use the Diaframe tactics to (help) prove the specifications __We recommend checking out the tutorial-style files first__. These can be found in `diaframe/tutorial` and contain lots of comments to help you get started. It explains the general process of verifying examples, and also explains how to define your own hints. You can also look at the examples in `diaframe/theories/examples/comparison`, although these are not commented on. #### 1. Create a new file We recommend using any Coq IDE (such as CoqIDE) to interact with Diaframe. Create a new file in a location where Coq can find Diaframe, such as `diaframe/theories/examples`. #### 2. Import heap_lang and Diaframe Get access to heap_lang by adding `From iris.heap_lang Require Import proofmode.` This makes it possible to write down programs in the heap_lang language, and gives you access to some tactics that may be helpful when verifying heap_lang programs. Get access to Diaframe for heap_lang by adding `From diaframe.heap_lang Require Import stepping_tacs.`. This gives you access to the `iStepS`, `iStepsS` and `iSmash` tactics. For more explanation on these tactics, see `diaframe/README.md`. #### 3. Write your program In this file, write down your program in the heap_lang language. heap_lang is a lambda-calculus with a heap, deeply embedded in Coq. An example function is: ``` Definition add_locs : val := λ: "l" "r", ! "l" + ! "r". ``` This defines a Coq-term `add_locs`, which has type `val`: a value in the heap_lang language. This value _is_ a function, and when applied to two locations, it returns the sum of their stored values. `add_locs` uses the `!` operation to load from a heap-location. Note that the binders in the function are strings (`"l"` and `"r"`), not regular Coq binders! Please also see the tutorial files in `diaframe/tutorial` for more examples and heap operations. You may also wish to consult [the heap_lang documentation](https://gitlab.mpi-sws.org/iris/iris/-/blob/master/docs/heap_lang.md). #### 4. Write down invariants and specifications Once you have written down your programs, please start a new Coq section with `Section .`, and add the following command directly after: ```Context `{!heapGS Σ}.```. This is necessary, since when verifying a heap_lang program, all specifications are parametrized over a 'ghost-state register' `Σ : gFunctors`. This `Σ` should at least give us access to the mapsto-connective `↦` from heap_lang, which is what the `heapGS` predicate tells us. At this point, you can start writing down invariants and specifications. Invariants are necessary whenever multiple threads may read or write from a shared location. It is customary to give a separate definition of the body of the invariant, and additionally create an `is_...` predicate to hide away the details of the invariant for clients. This becomes something like: ``` Definition program_inv (l : loc) : iProp Σ := ∃ v, l ↦ v ∗ ... . Let N := nroot.@"myprogram". Definition is_program_val (v : val) : iProp Σ := ∃ (l : loc), ⌜v = #l⌝ ∗ inv N (program_inv l). ``` This is not strictly required, but makes for clean specifications. Diaframe supports Iris's Texan Triples as goals, written as follows: `Lemma verification v : {{{ is_program_val v }}} my_func v {{{w, RET w; POST }}}.` These should be interpreted as regular Hoare triples, but are easier to use in Coq. If you want to use Diaframe to verify clients calling `my_func`, it is easier to directly write down a Diaframe SPEC-triple, as follows: `Global Instance verification v : SPEC {{ is_program_val }} my_func v {{(w : val), RET w; POST }}.` #### 5. Use Diaframe to prove specifications. After writing the specification, enter the Coq proof mode with `Proof.` After at least a single whitespace, you can start using Diaframe tactics to construct a proof. Make Diaframe run a chunk of steps with the `iStepS` tactic. Run the automation until it finishes or gets stuck with the `iStepsS` tactic. If your goal/invariants contain disjunctions and you wish to use backtracking to choose a side, use the `iSmash` tactic. If the function you are verifying is not recursive, proofs usually only consist of `Proof. iStepsS. Qed.`, where one possibly needs to replace `iStepsS` with `iSmash`. For recursive functions, one usually needs to Löb induction for the recursive call. Proofs then generally look like: `Proof. iStepsS. iLöb as "IH". wp_lam. iStepsS. Qed.`, where the last `iStepsS` may again need to be replaced with `iSmash`. It may be the case that Diaframe gets stuck. In this case you can try to manually prove the problematic part of the goal, then resume Diaframe's automation with `iStepsS`. Another option is to add appropriate hints, thereby teaching Diaframe to handle the problematic parts of the goal. See the tutorial files in `diaframe/tutorial` for examples. ## Extending or changing Diaframe Please see `diaframe/README.md` for some pointers on extending or changing Diaframe. This file also contains information on the directory structure and the function of each of the Coq source code files. # Step-by-Step Instructions To evaluate the artifact, we encourage you to check that the artifact supports the following list of claims made in the paper: - The verifications shown in Section 2 work - The update rules for token hold (Figure 4) - `sym-ex-fupd-exist` holds (Section 3.2) - `biabd-hint-apply` holds (Section 4.1) - The hint examples in Section 4.2 hold and are hints - The recursive hint rules in Section 4.3 hold. - The proof search strategy is similar to the one described in Section 5 - Diaframe can verify all examples in Table 1 (Section 6) - The numbers in Table 1 are accurate - Diaframe is foundational Each claim above comes with a subsection below, describing our suggestions to verify the claims. ## The verifications shown in Section 2 work Section 2 verifies two examples: a spin lock and an Atomic Reference Count. These examples can be found in `diaframe/theories/examples/paper/spin_lock_paper.v` and `diaframe/theories/examples/paper/arc_paper.v` respectively. To verify these files, navigate to the `diaframe` folder (reminder: make sure to activate the correct opam switch with `eval $(opam env)`) then run: ``` make theories/examples/paper/arc_paper.vo theories/examples/paper/spin_lock_paper.vo ``` Alternatively, use the CoqIDE to open and verify these files interactively. Note that these files differ slightly from those in `diaframe/theories/examples/comparison`. For the spinlock, the `newlock` spec we actually use is stronger, but also slightly more complicated. The ARC we verify frees the location storing the counter, which requires a slightly more complicated invariant. ## The update rules for token hold (Figure 4) See also the description of token-counting objects in the appendix. The fractional token object is defined in `theories/lib/frac_token.v` and is itself an instance of the more general `theories/lib/tokenizable.v`. The update rules are proven (with the token-object definitions unfolded) in line 39-81. The hints (`BiAbd` instances in the source code) corresponding to these rules can be found in the same file, section 'automation'. We have not yet built nice notation for hints. ## `sym-ex-fupd-exist` holds (Section 3.2) This statement and its proof can be found in `theories/symb_exec/weakestpre.v`, line 251. This rule is not used directly in Diaframe's implementation, since the implementation can actually handle more proof goals than just `WP`. See also the 'The proof search strategy is similar to the one described in Section 5' claim below. ## `biabd-hint-apply` holds (Section 4.1) This statement and its proof can be found in `theories/steps/solve_sep.v`, lines 68-87, named `solve_sep_biabd`. This statement is more general than `biabd-hint-apply`, since it does not require a fancy update modality. Instead, it requires that the goal modality `M` can be split (using `SplitModality3`) into the modality of the new goal `M1` and the modality of the hint `M2`. The version in the paper holds because we have `SplitModality3 (|={E1, E2}=>) (|={E1, E3}=>) (|={E3, E2}=>)`. Note that `solve_sep_biabd` does not mention an environment `Δ `. A version with environment `Δ` can be found at line 139, `tac_solve_sep_ext_biabd`. Instead of targetting a particular hypothesis `H` like `biabd-hint-apply` in the paper, this lemma takes any environment `Δ`, looks to find an applicable hint for some hypothesis `i` using the `FindInExtendedContext Δ (λ .., BiAbd) .. i ..` condition, then removes this hypothes `i` from the environment `Δ` using `envs_option_delete`. ## The hint examples in Section 4.2 hold and are hints For Example 1, see the discussion on the update rules for token above. For Example 2, see `theories/lib/iris_hints.v`, line 40, `bi_abduct_inv`. Example 3 is not a direct hint. It is an instantiation of `biabd_alloc_coalloc` in `theories/lib/own_hints.v`, line 2083. Since `locked γ` is defined as `Excl ()`, and Iris provides us with an instance of the `Exclusive` typeclass for this element, `exclusive_none_coallocate` at line 430 kicks in. Example 4 contains two heap-lang specific hints, which can be found in `theories/heap_lang/biabd_instances_heaplang.v`, lines 78-97. `mapsto_val_have_enough` is the first one, slightly more general than mentioned. `mapsto_val_may_need_more` is the second, more interesting one. Note the new quantification on `v'` in the sidecondition `⌜v1 = v2⌝ ∗ l ↦{#q} v'`. ## The recursive hint rules in Section 4.3 hold. These rules can be found in `theories/hint_search/lemmas_biabd.v`. The first one is `biabd_wand` at line 141. The second one (as mentioned in the footnote) is a consequence of multiple recursive rules: first `biabd_wand` (which is applicable by the `IntoWand2` instance in `theories/lib/iris_hints.v`), then `biabd_mod_intro_l`, then `biabd_sepl`. This might strip of the `▷` present in the paper by using the `LaterToExcept0` typeclass in `theories/lib/iris_hints.v`, making the recursive hint stronger than the one shown in the paper. ## The proof search strategy is similar to the one described in Section 5 The case analysis on G in 5.2 can be found in `theories/steps/small_steps.v`, line 62. This includes the extra case `MergeMod`, which is responsible for dealing with the bookkeeping of `▷`-modalities. `SolveSep` is case 5, `SolveOne` covers cases 3 and 4, `IntroduceHyp` is case 2 and `IntroduceVars` is case 1. The substeps of each case can be found at the end of the appropriately named files in `theories/steps`, i.e. `theories/steps/solve_sep.v`. These may be slightly more complicated, to improve the performance of Diaframe. Diaframe's implementation differs from the paper in cases 3 and 4. Instead of having a specific rule for `WP`, we have a more generic way of dealing with single atomic goals. This is the responsibility of `theories/steps/solve_one.v`, and of abduction hints. This means 3a) and 3b) and 4c) are actually 'just' abduction hints, found in `theories/symb_exec/weakestpre.v`, lines 510-524 (although 3b/4c is buried under some abstraction layers). See also the appendix for more details. ## Diaframe can verify all examples in Table 1 (Section 6) To (re)verify all these, navigate to the `diaframe` folder and run `make clean` and `rm time-of-build*`. Make sure your opam switch is functional with `eval $(opam env)`. Then run the following command: ``` make pretty-timed TGTS="comparison-all" TIMING_INCLUDE_MEM=0 ``` to verify all of the examples listed in Table 1. On a single core, this should take about 40 minutes, depending on your hardware. Enable parallel verification by additionally supplying make with the `-j NUMBEROFJOBS` argument. Once this is finished, `make` should print a table of verification times for each file. Note that the `barrier` and `peterson` examples are (much) slower than the other files, which is why `./built_artifact.sh` does not build them by default. You may also wish to check that the contents of each file in `theories/examples/comparison` correspond to real concurrent programs. ## The numbers in Table 1 are accurate The artifact comes with a python script that can generate Table 1. This script is located at `diaframe/utils/stats_gen.py`, and requires the `tabulate` python package to run. ### Generate table on the virtual machine On the virtual machine artifact, navigate on the command line to `~/Documents/paper-auto-iris-artifact/build/diaframe`. Now run ``` python utils/stats_gen.py ``` which will print the table. It is probably too wide to display nicely in the terminal. Try ``` python utils/stats_gen.py | cut -c 1-80 ``` to only display Diaframe specific numbers. ### Generate table from locally built Diaframe Navigate to your `diaframe` folder. To get access to `python` with the `tabulate` package, we recommend you install a virtual environment using ``` python3 -m venv utils/statenv ``` Now activate the virtual environment using ``` source utils/statenv/bin/activate ``` install tabulate using ``` pip install tabulate ``` then generate the table with the command ``` python utils/stats_gen.py ``` ### Differences between generated table and Table 1 The table generated with the artifact and Table 1 in the submitted version of the paper differ in various places. We mention and comment on large differences below. - `bag_stack`: larger annotation and proof burden. We added an additional `is_list_skel` predicate and corresponding hints, making it longer, but this makes the proof (in our opinion) more elegant and faster to verify. This increased the total line count with 21 lines, and the proof burden with 10 lines. The old version of `bag_stack` still works, but takes about twice as long to verify. - `barrier_client`: smaller annotation and proof burden, due to fixes and improvements in Diaframe's implementation - `lclist`: smaller annotation and proof burden, due to fixes and improvements in Diaframe's implementation - `msc_queue`: smaller annotation and proof burden, due to fixes and improvements in Diaframe's implementation, and a small change to the used ghost state - `queue`: smaller annotation and proof burden, due to fixes and improvements in Diaframe's implementation, and a small change to the used ghost state - `rwlock_duolock`: smaller annotation and proof burden, due to fixes and improvements in Diaframe's implementation - `total`: The total numbers have changed, not just because of the changed numbers. The calculation of totals also turned out to be wrong in the submitted version (you can check this by manually adding, for example, all implementation line counts - the actual total does not match the listed total) An updated screenshot of Table 1 is included in this artifact as `table1_updated.png`. The takeaway from this updated table is that the actual numbers are slightly better than originally presented in the paper: averaged over all examples, Diaframe actually requires about 0.4 lines of proof work per line of implementation (321 lines of proof work for 823 lines of implementation), whereas in the paper we spoke of 0.5 lines of proof work per line of implementation. ### Checking how lines are counted To see what lines of a file count as implementation, annotation, proof burden and customization, you can run (from the command-line, inside the `diaframe` folder) ``` python utils/stats_gen.py theories/examples/comparison/ ``` This will print the file in the terminal, where each line has been prefixed with how it has been counted. Some files have lines which are counted as 'reuse'. These lines are not necessary for the verification of the file, but necessary when clients want to *reuse* the verified specification. See for example the spin lock. Reuse lines are included in the total count, but not in the annotation count. Proof burden for reuse *is* included in the total proof burden. ### Verifying linenumbers from different tools The line-number data for other tools is stored in a `.csv` file it `diaframe/utils`. These have been manually extracted from examples verified in the other tools. If you wish to check whether these numbers are correct, you can find the examples for other tools here: - [Starling](https://github.com/septract/starling-tool/tree/master/Examples) - [Caper](https://github.com/caper-tool/caper/tree/master/examples/recursive) - [Voila](https://github.com/viperproject/voila/tree/master/src/test/resources/voila_evaluation_examples/weak_spec/correct) ## Diaframe is foundational As Diaframe claims to be a foundational tool, it is relevant to check that Diaframe can indeed produce closed proofs. This is only possible for closed programs. We have included a file which does this for `cas_counter_client`, it can be found inside the `diaframe` folder, at `theories/examples/comparison/cas_counter_client_closed.v`. For more information on generating a closed proof, see the tutorial file at `diaframe/tutorial/ex1_oneshot.v`, or the [Iris documentation](https://gitlab.mpi-sws.org/iris/iris/-/blob/master/docs/resource_algebras.md#obtaining-a-closed-proof). Uncomment the `Print Assumptions` line and compile the file, or run it interactively with CoqIDE. The `Print Assumptions` command should print `Closed under the global context`, which indicates that no Axioms were used to prove the `client_adequate` lemma. Note that the `client_adequate` lemma statement no longer contains anything Diaframe (or Iris!) specific: it is a pure statement about all possible evaluations of the `parallel_increment` program. By running `Print adequate.` you can see that `adequate` precisely states that the program is safe (does not get stuck) and if it terminates, returns the value 2. This corresponds to partial correctness. # Differences between the paper and the implementation of Diaframe To make the paper more accessible, it discusses Diaframe on a high-level and does not discuss all the implementation details. At various places, the implementation differs from the description in the paper, usually allowing it to handle more goals. The most important differences are given below. - In the implementation, we always use generic modalities, not fancy updates, in lemmas. These should be `ModalityEC` modalities: monotone over `⊢`, and satisfy `M P ∗ Q ⊢ M (P ∗ Q)`. Essentially, they are strong functors. Examples of such modalities are the basic update `|==>`, and the identity 'modality' `id`. See also claim '`biabd-hint-apply` holds (Section 4.1)' above. - The grammar described in Section 5 is *not enforced*. Essentially, the grammar is an informal agreement between Diaframe and its users. - The Coq development contains additional machinery to deal with `⊳` and other modalities, like `⋄`. - The proof search strategy described in section 5 is a simplified version. See also the appendix, and claim 'The proof search strategy is similar to the one described in Section 5' # Troubleshooting ### Starting the virtual machine with KVM's Virtual Machine Manager If you want to use `virt-manager` (`virsh` and friends), you have to convert the `diaframe-artifact-vm.ova` file to a `.qcow2` file. You can do this as follows. An `.ova` file is just a `tar` archive, extract it: ``` tar xvf diaframe-artifact-vm.ova ``` This will print a list of extracted files (in the current directory). The `.vmdk` file can be converted to a `.qcow2` file, which is the format `virt-manager` prefers. Now create a new virtual machine, choose 'import existing disk image' and select the `.qcow2` file. You should now be able to run the artifacts virtual machine. ### make complains: `make: *** no rule to make target `. Ensure that you have run `eval $(opam env)`. If you haven't, `opam switch list` will warn you that your environment is not in sync with the current switch. ### make says: `Nothing to be done for ...`, while I have just `touch`ed the source See above.