Published June 17, 2023 | Version 1.0.0

DarkDiff: Explainable web page similarity of TOR onion sites

  • 1. TU Delft
  • 2. cflw.com

Description

In large-scale data analysis, near-duplicates are often a problem. For example, with two near-duplicate phishing emails, a difference in the salutation (Mr vs Ms) is not essential, but whether it is bank A or B is important. The state-of-the-art in near-duplicate detection is a black box approach (MinHash), so one only knows that emails are near-duplicates, but not why. We present DarkDiff, which can efficiently detect near-duplicates while providing the reason why there is a near-duplicate. We have developed DarkDiff to detect near-duplicates of homepages on the Darkweb. DarkDiff works well on those pages because they resemble the clear web of the past.

Files

Files (520.8 kB)

Name Size
md5:58d33784dc6225bd1da24ea9add77c49
520.8 kB Download

Additional details

Funding

European Commission
STARLIGHT - Sustainable Autonomy and Resilience for LEAs using AI against High priority Threats 101021797