A limited technical background is sufficient for attack-defense tree acceptability: Dataset

Abstract (English)

Study description

Attack-defense trees (ADTs) are a prominent graphical threat modeling method that is highly recommended for analyzing and communicating security-related information. Despite this, existing empirical studies of attack trees have established their acceptability only for users with highly technical (computer science) backgrounds while raising questions about their suitability for threat modeling stakeholders with a limited technical background. Our research addresses this gap by investigating the impact of the users' technical background on ADT acceptability in an empirical study.

Our Method Evaluation Model-based study consisted of n=102 participants (53 with a strong computer science background and 49 with a limited computer science background) who were asked to complete a series of ADT-related tasks. By analyzing their responses and comparing the results, we reveal that a very limited technical background is sufficient for ADT acceptability. This finding underscores attack trees' viability as a threat modeling method.

Artifact description

This artifact includes the models (attack-defense trees) created by the participants in the study tasks, the study responses to perception questions (both Likert and short answer), the code used to statistically evaluate those survey responses, the study question text and images, the qualitative evaluation rubric for self-drawn ADTs, and the lecture plan and slides. With these artifacts, it should be possible to verify our results, organize training on attack-defense trees, and develop future studies of attack-defense tree acceptability.

Table of contents (English)

The following is a description of each of the items in the repository

• ADT1.zip
  • Folder containing .png and .xml files representing ADTs created from the ADT1 task (drawing ADTs from leaf nodes)
• ADT2.zip
  • Folder containing .png and .xml files representing ADTs created from the ADT2 task (drawing ADTs from a written scenario)
• ADT3.zip
  • Folder containing .png and .xml files representing ADTs created from the ADT3 task (drawing ADTs of a self selected scenario)
• Survey Data.csv
  • .csv file containing the collected participant responses to both the Large Study (LS) and Small Study (SS). This includes processed data, such as the results of qualitative analysis and the quantitative data pulled from the ADTs.
• Statistics.ipynb
  • Jupyter notebook containing code to run our statistical tests on Survey Data.csv. This requires that Survey Data.csv is placed in the same folder or that the PATH variable is set.
• Small Study (SS).pdf 
  • The question text and images from the small study.
• Large Study (LS).pdf 
  • The question text and images from the large study.
• ADT3 Qualitative Evaluation Rubric.pdf
  • The rubric used to qualitatively evaluate the ADT3 (self drawn ADTs) according to four categories: cohesiveness, conciseness, clarity, and completeness.
• Lecture Plan.pdf
  • A detailed list of the topics covered in the training lecture, as well as advice on how to convert this into a training suitable in a professional/industry context
• Lecture Slides.pptx
  • The powerpoint presentation of the slides used during the training lecture.

Technical info (English)

The intructions to run the Statistics.ipynb jupyter notebook are as follows:

Notebook to run statistical testing for paper titled A limited technical background is sufficient for attack-defense tree acceptability accepted to 34th USENIX Security Symposium. The file necessary to run this notebook is Survey Data.csv. If this file is located in a different location to the jupyter notebook, you must change the PATH variable.

The cells in this notebook are designed to be run in order. Once the PATH variable is set, the notebook can be run by Running All Cells.