A Dataset of Information (DNS, IP, WHOIS/RDAP, TLS, GeoIP) for a Large Corpus of Benign, Phishing, and Malware Domain Names 2024
Description
The dataset contains DNS records, IP-related features, WHOIS/RDAP information, information from TLS handshakes and certificates, and GeoIP information for 368,956 benign domains from Cisco Umbrella, 461,338 benign domains from the actual CESNET network traffic, 164,425 phishing domains from PhishTank and OpenPhish services, and 100,809 malware domains from various sources like ThreatFox, The Firebog, MISP threat intelligence platform, and other sources. The ground truth for the phishing dataset was double-check with the VirusTotal (VT) service. Domain names not considered malicious by VT have been removed from phishing and malware datasets. Similarly, benign domain names that were considered risky by VT have been removed from the benign datasets. The data was collected between March 2023 and July 2024. The final assessment of the data was conducted in August 2024.
The dataset is useful for cybersecurity research, e.g. statistical analysis of domain data or feature extraction for training machine learning-based classifiers, e.g. for phishing and malware website detection.
The dataset was created using software available in the associated GitHub repository nesfit/domainradar-dib.
Data Files
-
The data is located in the following individual files:
- benign_umbrella.json - data for 368,956 benign domains from Cisco Umbrella,
- benign_cesnet.json - data for 461,338 benign domains from the CESNET network,
- phishing.json - data for 164,425 phishing domains, and
- malware.json - data for 100,809 malware domains.
- The schema.json file contains a JSON Schema with detailed description of the data entries.
Data Structure
Both files contain a JSON array of records generated using mongoexport (in the MongoDB Extended JSON (v2) format in Relaxed Mode). The following table documents the structure of a record. Please note that:
- some fields may be missing (they should be interpreted as nulls),
- extra fields may be present (they should be ignored).
|
Field name |
Field type |
Nullable |
Description |
|
domain_name |
String |
No |
The evaluated domain name |
|
url |
String |
No |
The source URL for the domain name |
|
evaluated_on |
Date |
No |
Date of last collection attempt |
|
source |
String |
No |
An identifier of the source |
|
sourced_on |
Date |
No |
Date of ingestion of the domain name |
|
dns |
Object |
Yes |
Data from DNS scan |
|
rdap |
Object |
Yes |
Data from RDAP or WHOIS |
|
tls |
Object |
Yes |
Data from TLS handshake |
|
ip_data |
Array of Objects |
Yes |
Array of data objects capturing the IP addresses related to the domain name |
|
malware_type |
String |
No |
The malware type/family or “unknown” (only present in malware.json) |
|
DNS data (dns field) |
|||
|
A |
Array of Strings |
No |
Array of IPv4 addresses |
|
AAAA |
Array of Strings |
No |
Array of IPv6 addresses |
|
TXT |
Array of Strings |
No |
Array of raw TXT values |
|
CNAME |
Object |
No |
The CNAME target and related IPs |
|
MX |
Array of Objects |
No |
Array of objects with the MX target hostname, priority and related IPs |
|
NS |
Array of Objects |
No |
Array of objects with the NS target hostname and related IPs |
|
SOA |
Object |
No |
All the SOA fields, present if found at the target domain name |
|
zone_SOA |
Object |
No |
The SOA fields of the target’s zone (closest point of delegation), present if found and not a record in the target domain directly |
|
dnssec |
Object |
No |
Flags describing the DNSSEC validation result for each record type |
|
ttls |
Object |
No |
The TTL values for each record type |
|
remarks |
Object |
No |
The zone domain name and DNSSEC flags |
|
RDAP data (rdap field) |
|||
|
copyright_notice |
String |
No |
RDAP/WHOIS data usage copyright notice |
|
dnssec |
Bool |
No |
DNSSEC presence flag |
|
entitites |
Object |
No |
An object with various arrays representing the found related entity types (e.g. abuse, admin, registrant). The arrays contain objects describing the individual entities. |
|
expiration_date |
Date |
Yes |
The current date of expiration |
|
handle |
String |
No |
RDAP handle |
|
last_changed_date |
Date |
Yes |
The date when the domain was last changed |
|
name |
String |
No |
The target domain name for which the data in this object are stored |
|
nameservers |
Array of Strings |
No |
Nameserver hostnames provided by RDAP or WHOIS |
|
registration_date |
Date |
Yes |
First registration date |
|
status |
Array of Strings |
No |
The state of the registered object (see RFC 7483, section 10.2.2) |
|
terms_of_service_url |
String |
No |
URL of the RDAP usage ToS |
|
url |
String |
No |
URL of the RDAP entity |
|
whois_server |
String |
No |
WHOIS server address |
|
TLS data (tls field) |
|||
|
cipher |
String |
No |
TLS cipher suite description according to IANA |
|
protocol |
String |
No |
One of “TLS”, ”TLSv1.2”, ”TLSv1.3” |
|
certificates |
Array of Objects |
No |
Array of objects representing the certificate chain, the first element is the root certificate |
|
IP data (elements in the ip_data array) |
|||
|
ip |
String |
No |
The IP address |
|
from_record |
String |
No |
The type of the DNS record the address was captured from |
|
remarks |
Object |
No |
Ping round-trip time, “is alive” flag and rdap/geo/asn evaluation dates |
|
rdap |
Object |
Yes |
RDAP data, similar to DNS RDAP, see the JSON Schema for details |
|
geo |
Object |
Yes |
Geolocation data from the GeoLite2 City database (e.g. latitude, longitude, city, country, etc.) |
|
asn |
Object |
Yes |
Autonomous system data from the GeoLite2 ASN database (ASN, organization, network) |
Acknowledgements
We would like to thank the OpenPhish Team for grating permission to use and publish their dataset. We also thank VirusTotal for providing us access to the API for research purposes.
This dataset includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.
The research has been supported by the Flow-based Encrypted Traffic Analysis project, no. VJ02010024, granted by the Ministry of the Interior of the Czech Republic and the Smart Information Technology for a Resilient Society project, no. FIT-S-23-8209, granted by Brno University of Technology.
Files
benign_cesnet.json
Files
(15.8 GB)
| Name | Size | Download all |
|---|---|---|
|
md5:cea8cbe33ec308f76c5ecf3ef6a6219e
|
6.4 GB | Preview Download |
|
md5:41e340c0714d7455494c0f893193cb5e
|
6.1 GB | Preview Download |
|
md5:725ff97c470f11d094a9ab81a582dceb
|
41.2 kB | Preview Download |
|
md5:ed9b14f24feb2e66513f9339492b071d
|
1.1 GB | Preview Download |
|
md5:77a4be494ae696371190a1f7724dc3d9
|
2.2 GB | Preview Download |
Additional details
Dates
- Available
-
2024-08-16
Software
- Repository URL
- https://github.com/nesfit/domainradar-dib
- Programming language
- Python, Shell
- Development Status
- Wip