Wrong Answers for Wrong Reasons: The Risks of Ad Hoc Instruments

This folder contains anonymized data from the experiment described in the paper "Wrong Answers for Wrong Reasons: The Risks of Ad Hoc Instruments".

• questions.md contains the 14 questions used as pre/post-test
• Participants are identified using numeric identifiers (from 1 to 40).
• conditions.csv contains the mapping between participants identifiers and their condition

• {test}_{assessment_type}_anon.csv contains data for a given test (pretest or posttest) using a certain assessment type. Two assessment types are used:

  • mc (multiple choice as assessed by Moodle: 0 means wrong, 1 means correct, - means missing)
  • revised (human revision after reading the explanation: the judgment is either a 0 or a 1 following what presented in the paper)

• Statistics are computed with Python and SciPy. You can execute the script by running:

  python3 main.py
  

  and you should get this output:

  === Analyzing answers using mc as assessment type ===
  Results on pretest - Condition Text: N: 21, mean ± std: 9.81 ± 3.03
  Results on pretest - Condition Graphic: N: 19, mean ± std: 9.68 ± 2.11
  Mann Whitney U on pretest scores: U=184.0, p=0.3406
  
  Results on posttest - Condition Text: N: 21, mean ± std: 11.19 ± 2.75
  Results on posttest - Condition Graphic: N: 19, mean ± std: 10.74 ± 2.23
  Mann Whitney U on posttest scores: U=164.0, p=0.1681
  
  === Analyzing answers using revised as assessment type ===
  Results on pretest - Condition Text: N: 20, mean ± std: 9.15 ± 3.10
  Results on pretest - Condition Graphic: N: 19, mean ± std: 7.89 ± 3.16
  Mann Whitney U on pretest scores: U=150.5, p=0.1344
  
  Results on posttest - Condition Text: N: 20, mean ± std: 11.15 ± 2.39
  Results on posttest - Condition Graphic: N: 19, mean ± std: 9.42 ± 3.11
  Mann Whitney U on posttest scores: U=123.0, p=0.0299
  

  You might need to install scipy (e.g., depending on your setup, pip install scipy).