Analysing Technique Classification Performance of Cyber Threat Intelligence Extractors

This thesis undertakes a comprehensive assessment of two state-of-the-art CTI
extraction programs in terms of technique classification. Inspired by the development
of open-source Large Language Models (LLMs), we also evaluated four of these
models in terms of classification performance, to test the potential for cross-validation
with CTI extractors. Our analysis points out that despite recent progress in technique
classification, the performance of CTI extractors remains low. With the high number
of false positives, open-source LLMs are not useful for this task either.
In addition, we introduce a novel pipeline for extracting attack techniques from CTI
reports. This pipeline involves report summarization using ChatGPT, followed by
loading the output into a specially retrained SciBERT model tailored to the
cybersecurity domain. Our approach yields promising results: when applied to 10 CTI
reports, we observe a substantial 7-percentage-point increase in the F1-score compared
to the original SciBERT model for processing entire reports. Furthermore, this
research outlines potential avenues for future exploration within the field, aimed at
further enhancing the precision of technique extractions.