Artifact for "How hard can it be? Quantifying MITRE attack campaigns with attack trees and cATM logic", by Stefano M. Nicoletti, Milan Lopuhaä-Zwakenberg, Mariëlle Stoelinga, Fabio Massacci, and Carlos E. Budde

Minimal instructions

Setup a TACAS VM, copy artifact.tar.xz into the VM, uncompress it and execute the main script:

# in a terminal inside the VM:
tar xvf artifact.tar.xz
cd artifact/
./run.sh

NOTE: If working with Virtual Machines and shared folders, copy artifact.tar.xz in a directory inside the guest VM and extract it there—otherwise, e.g. if working directly inside of a shared directory, symbolic links contained in the archive may not work correctly. To manipulate shared folders in Oracle VirtualBox VMs see "Shared folders" at the end of this README.

The artifact

├── README.{md,html,pdf}
├── LICENSE       # (MIT) license of this artifact
├── custom_ATs/   # custom AT models for the Wocao and Dream Job campaigns:
│   ├── C0014_Wocao_custom_AT.ipynb     # Jupyter notebook for Wocao
│   ├── C0022_DreamJob_custom_AT.ipynb  # Jupyter notebook for Dream Job
│   ├── custom_C0014.pdf  # Figure of the custom AT for Wocao
│   ├── custom_C0022.pdf  # Figure of the custom AT for Dream Job
│   └── ...
├── MITRE_AT_templates/ # Scripts to create MITRE attack tree templates:
│   ├── MITRE_AT_template_creator.py  # Generator of MITRE attack tree templates
│   ├── create_MITRE_AT_templates.py  # Script to generate ATs for all MITRE campaigns
│   ├── create_MITRE_AT_plots.py      # Plotter of security indices as confidence bands
│   └── ...
├── src/   # Other scripts for experimental execution:
│   ├── AttackTree.py             # Python ADT for attack trees
│   ├── AttackTree_algorithms.py  # cATM (and AT metrics) implementation
│   ├── techniques_frequency_per_tactic.json   # probability estimates of MITRE attack techniques
│   ├── techniques_frequency_per_tactic.ipynb  # Jupyter notebook to generate the above
│   ├── enterprise-attack-v14.1-techniques.xmlx  # MITRE ATT&CK® data for attack techniques
│   ├── MITRE_campaigns/*.json  # MITRE ATT&CK® data for attack campaigns
│   └── ...
├── results_article/ # results reported in the article:
│   ├── MATT_<d>_<c>*.py    # script for template AT of difficulty <d> for campaign <c> 
│   ├── MATT_<d>_<c>*.out   # shell output of executing the above script
│   ├── MATT_<d>_<c>*.pdf   # corresponding (template) AT figure
│   ├── MATT_templates.log  # log of executing all templates, with attack probabilities
│   └── ...
├── dependencies/ # .deb files needed to execute the artifact
├── .venv_3_10/   # Python 3.10 virtual environment
└── run.sh        # <-- main execution script (artifact entry point)

Licenses and copying

All files and scripts generated by the authors for experimental reproduction—including the bundled results of the accompanying article—are distributed under the MIT license.
Some scripts and files in src/, and dependencies in the .venv_3_10/ and dependencies/ directories, are distributed with other (open-source compatible) licenses, e.g. GPLv3 and BSD—the corresponding licenses can be found in the scripts, source files, and directories themselves.
All data and files downloaded from the MITRE ATT&CK® knowledge base are under the terms and conditions provided by MITRE ATT&CK®: all credit for them goes to the MITRE Corporation.

Environment and requirements

The artifact is prepared to run in Debian-based Linux distributions—it has been tested in Ubuntu 22.04. You can download a VirtualBox VM image that satisfies all these requirements from: https://zenodo.org/record/7113223. We refer to this virtual machine in its standard configuration with 1 processor, 8 GB RAM, and no Internet connection, as the TACAS VM.

Files for inspection & connection to article results

More files relevant for inspection:

MITRE_AT_templates/techniques_frequency_per_tactic.ipynb is a Jupyter notebook that implements the estimation of probabilities from MITRE data, as described in Section 2.2 of the article, e.g. eq. (1);
custom_ATs/C0014_Wocao_custom_AT.ipynb is a Jupyter notebook that creates the custom AT model for the Wocao campaign, as presented in figures 5 and 11, and Table 1 of the article;
custom_ATs/C0022_DreamJob_custom_AT.ipynb is as above for the Dream Job campaign, as presented in figures 6 and 12, and Table 4 of the article;
src/AttackTree_algorithms.py contains a prototypical implementation of the cATM logic, as per Section 5 of the article, for the Pythonic attack tree ADT of this artifact;
results_article/MATT_default_C0014.pdf is the AT model figure for the default template of the Wocao campaign, presented (slightly rotated) as Figure 9 in the article;
results_article/MATT_template.log contains the (probability estimates whose) security indices are presented in Table 3 of the article;
results_article/MATT_template.pdf is the plotting of security indices for the AT templates of all MITRE campaigns, as presented in Figure 10 of the article;

Experimental reproduction

Instructions

To reproduce the experimental results of the article, start by setting up a Debian system as indicated above, e.g. the one provided by the TACAS VM. Copy the artifact.tar.xz archive into the home folder of that system, uncompress it, move into the artifact/ directory and execute the run.sh script:

cd "$HOME"
cp <path>/artifact.tar.xz .
tar xvf artifact.tar.xz
cd artifact/
./run.sh

Extract artifact

Running this artifact in a TACAS VM on a standard laptop has taken from 12 to 20 minutes.

Execution steps of the artifact

Running the main script as indicated above performs the following steps:

Install the required dependencies:
- .deb Debian packages are bundled in the artifact to complete the needs of the TACAS VM;
- installation requires root privileges: please enter the password (tacas23) when prompted;
- if a working binary of jupyter-notebook is found in $PATH, the script assumes that this is a recurrent invokation and this step is omitted.
Activate a Python 3.10 virtual environment:
- A hidden directory .venv_3_10 contains a Python virtual environment that includes all necessary dependencies, e.g. for plotting and executing the Jupyter notebooks;
- This environment is activated prior to experimental replication, and automatically deactivated when the script finishes successfully.
Perform the experimental replication:
1. execute MITRE_AT_templates/create_MITRE_AT_templates.py to instantiate the attack tree templates in all levels of difficulty, for all MITRE ATT&CK® campaigns,
2. save all results in a date-tagged subdirectory of MITRE_AT_templates/,
3. execute MITRE_AT_templates/create_MITRE_AT_plots.py to generate the security index comparison plot, using the results generated in the previous step,
4. display this plot in a window, using the system-default PDF reader.
Print the location of the Jupyter notebooks that implement the custom attack trees, presented in the accompanying article, for the Wocao and Dream Job campaigns.

NOTE: After executing run.sh at least once—to install all necessary dependencies—the Jupyter notebooks can be run to reproduce the experimental results of the Wocao and Dream Job campaigns. Reproduction of the Jupyter notebooks is done as indicated next.

In a file browser, navigate to $HOME/artifact/custom_ATs/ and double click any .ipynb file:
- Ignore the initial error message of the web browser (if any);
- Open, in a web browser, the tokenised URL printed in the terminal that opens upon clicking the Jupyter notebook (e.g. Ctrl+click the link in the shell).
Choose a notebook (.ipynb files) and open it by clicking on it in the web browser.

Open notebook

Change the Python kernel to be the one of the virtual environment installed during step 1. For that, on the tab menu choose Kernel → Change kernel → .venv_3_10.

Change the Python kernel to be .venv_3_10

Execute the notebook, e.g. by running all cells via Kernel → Restart & Run All → Restart and Run All Cells
- Results will be displayed directly in the notebook

Notebook results

Run the artifact natively in your own machine

Creating a Python virtual environment

We recommend using Python 3.11 or later to create a virtual environment:
```
python3.11 -m venv venv
```
Activate the virtual environment:
```
source venv/bin/activate
```

Installing Python dependencies

In an activated Python virtual environment, install the following packages using pip:

pip install openpyxl odfpy numpy pygraphviz notebook networkx matplotlib pandas dd

NOTE: Depending on your native setup, some system dependencies may need to be installed to satisfy the above Python dependencies. For instance, in the TACAS VM, the Debian/Ubuntu packages for graphviz and python3-graphviz must be installed for pip install pygraphviz to succeed.

Versions with known compatibility

Python lib	version
`openpyxl`	`3.1.5`
`odfpy`	`1.4.1`
`numpy`	`2.2.0`
`pygraphviz`	`1.14`
`notebook`	`7.3.1`
`networkx`	`3.4.2`
`matplotlib`	`3.10.0`
`pandas`	`2.2.3`
`dd`	`0.6.0`

Running the Python script

Change directory into MITRE_AT_templates/:
```
cd MITRE_AT_templates/
```

Run the script:

python3 create_MITRE_AT_templates.py

# >>  Creating templates  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# Creating "hard" MATT for campaign C0001
#     - Written file MATT_hard_C0001.py
# Creating "easy" MATT for campaign C0001
#         ...
# <more script output>
#         ...
# Executing "default" MATT for campaign C0028
#     - Probability of attack: 2.17e-30
#     - Written files MATT_default_C0028.{out,err}
# <<  DONE  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
# >>  Storing results  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
#  All results files are in results_Nov11_123456/
#  Activities log is results_Nov11_123456/MATT_templates.log
# <<  DONE  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

python3 create_MITRE_AT_plots.py results_Nov11_123456/MATT_templates.log

# Plot saved as results_Nov11_123456/MATT_templates.pdf

Running the Jupyter Notebooks

Launch Jupyter Notebook:
```
jupyter notebook
```
In your web browser, navigate to the custom_ATs/ directory and open each .ipynb file.
Run the notebooks by selecting:
- Run > Run All Cells

Run the artifact in a virtual machine: the TACAS VM

Shared folders

When using the TACAS VM, or any other Oracle Virtualbox VM, shared folders are a common way to move files between the host and guest systems. To create a shared folder shared/ in the home of the TACAS VM, you can do as follows:

Before turning on the VM, click on its "Settings" and select the tab "Shared Folders"
On the right menu, click the icon with a plus sign
- on mouse hover it reads "Adds new shared folder." or similar
Click "Folder Path:" and select "Other ..." from the dropdown menu: then choose the directory of your host system that you want to share with the guest system of the VM
- For example, let's say that you choose /home/user/artifact/ that contains the files of this artifact
In the field "Folder Name:" write shared
Check the "Auto-mount" checkbox
Click "Ok" to save that folder, and again "Ok" to close the "Settings"
Turn on the VM

In the guest system, open a shell and type in the following commands:

cd $HOME
mkdir -p shared
sudo mount -t vboxsf -o uid=1000,gid=1000 shared shared
ls shared/

Those steps will create a shared folder called shared in the $HOME directory of the guest VM. There you can read and write files from the directory /home/user/artifact/ (in this example) of your host system.

You can copy files or directories from /home/user/artifact/ into the $HOME directory of your guest system (e.g. cp -r shared/models/ $HOME): that is the recommended way to execute the artifacts inside the VM.

Troubleshooting artifact setup in the TACAS VM

If you are running this artifact in the TACAS VM, a daemon for unattended upgrades may lock the package manager on startup. This interferes with the installation of .deb packages that are needed as dependencies for the Python virtual environment.

Thus, if during installation you see the following error message: E: Could not get lock /var/lib/apt/lists ... then you will have to manually kill the "unattended-upgr" process prior to executing the artifact, e.g. running sudo killall unattended-upgr. Note that this requires root privileges, and you may need to do it twice in a row.