Yes or No: Trace Satisfaction

Welcome

This file contains anonymized, hand-labeled responses to the third and final segment of our Spring 2021 LTL/Electrum Survey.

Participants received the following instructions:

  Recall that Electrum formulas are statements about traces (sequences of
  states). A non-trivial formula is satisfied by some traces, but not all.

  The following questions present a formula and ask whether it is satisfied by a
  specific trace. For each, choose Yes or No and briefly explain which part(s) of
  the trace motivated your answer.

  Notation for states and traces: 
  - {} is the empty state, where all lights are off
  - {RGB} is the state where all lights are on
  - in {G} only the Green light is lit
  - ... and so on, for other combinations of R G and B
  - {} {} {} {R} {} is a trace in which the Red light is lit in the fourth state, and all lights are off at other times
  - all traces below show exactly five states; you can assume the final state repeats forever

  Template:
  HIDDEN FOR DOUBLE BLIND REVIEW

  Example Q/A:

  Q. Is the formula
    eventually { Blue in Panel.lit } 
   satisfied by trace {} {RG} {RG} {RG} {G} ?
  A. No
  Why? Because the Blue light never turns on --- neither in the first four
  states nor in the final lasso state.



Q6.3

formula:Red in Panel.lit
trace:{GB} {RGB} {RGB} {RGB} {RGB}
answerNo

49 / 57 correct (86%)

8 students (14%) fell for implicit F, most students understand question format

correct, N = 47
correct, implicit G, N = 1
correct, guessed, N = 1
incorrect, implicit F + misunderstood question format?, N = 8



Q6.5

formula:after { after { after { Red in Panel.lit } } }
trace:{R} {} {} {R} {}
answerYes

54 / 57 correct (95%)

no student thinks that X spans

correct, N = 46
correct, implicit F, 4
correct, miscount X, N = 4
incorrect, implicit G, N = 1
incorrect, mis-count X + confused off / on?, N = 2



Q6.7

formula:always { Red in Panel.lit implies after { after { after { Red in Panel.lit } } } }
trace:{} {RGB} {RGB} {} {RGB}
answerYes

50 / 57 correct (88%)

12 students (21%) do not understand that the final trace state is the lasso

correct, N = 43
correct, misunderstood question?, N = 2
correct, wrong lasso, N = 5
incorrect, wrong lasso, N = 7



Q6.9

formula:after { Red in Panel.lit } until after { Green in Panel.lit }
trace:{RB} {RB} {RB} {RGB} {B}
answerYes

44 / 57 correct (77%)

9 think until negates rhs

correct, N = 39
correct, N = 5
incorrect, U forces lhs to become false, N = 9
incorrect, missing U, N = 1
incorrect, implicit G, N = 2
incorrect, participant guessed, N = 1



Q6.11

formula:eventually { Red in Panel.lit} and eventually { Green in Panel.lit }
trace:{} {G} {} {} {R}
answerYes

57 / 57 correct (100%)

no students confused by scope here

correct, N = 56
correct, N = 1



Q6.13

formula:after { after { eventually { Red in Panel.lit } } }
trace:{RGB} {RGB} {RGB} {RGB} {RGB}
answerYes

56 / 57 correct (98%)

only one student thinks: "X negates current state"

correct, N = 48
correct, N = 8
incorrect, X requires negation in current state, N = 1



Q6.15

formula:Red in Panel.lit until Blue in Panel.lit
trace:{R} {R} {R} {R} {R}
answerNo

28 / 57 correct (49%)

51% think until is weak

correct, N = 28
incorrect, weak U semantics, N = 29



Q6.17

formula:eventually { always { Red in Panel.lit } }
trace:{} {RGB} {} {RGB} {}
answerNo

57 / 57 correct (100%)

3 think F disallows flickers

correct, N = 53
correct, F disallows flickers, N = 3 students
correct, wrong lasso, N = 1



Q6.19

formula:always { Red in Panel.lit implies Green in Panel.lit }
trace:{} {} {} {} {}
answerYes

57 / 57 correct (100%)

no evidence of "=> jumps to satisfying state" misconception

57, N = 57