Shadowserver reports automated tool

Project Specification

Every day, CERN receives mail notifications from Shadowserver, which include results of network scans for specific vulnerabilities of various types1 for autonomous system number (ASN) 513, which is under the control of CERN.

Checking these e-mail reports manually is time-consuming and not scalable. Instead, the CERN Computer Security Team prefers some kind of a tool for:

 extracting data from e-mails (csv.zip attachments or embedded links to CSV files);

 confirming reports by running additional scans from inside the network;

 handling repeated reports for the same device;

 dealing with known false positives / whitelisting;

 filtering out non-CERN hosts;

 sending Security Event Management System (SEMS) notifications;

 etc.

Abstract

The Shadowserver Foundation is offering a completely free-of-charge alerting and reporting service designed for ISPs, enterprises, hosting providers and other organizations that own or control a particular network space. The variety of reports provided to organizations serve as intelligence and assist in the process of locating and mitigating the security issues which occur inside their network. Being subscribed to this scanning and reporting service, CERN receives daily summaries of the security issues that happened during the past day.

Analysing and handling all the reported issues manually is a time-consuming, tedious and repetitive job, because it would require a particular person from the Computer Security Team to go through a series of steps every day. In addition, the manual approach is not scalable and tends to be error-prone, which might lead to important things being missed.

The main goal of this project is to create an automated tool that would be capable of extracting the relevant data from the received reports. However, it should not simply store the information in a database, but somehow notify the device owners that their devices were involved in a particular security issue. Also, it should be able to keep track of who was notified about what and when, in order to avoid sending multiple messages to a person about the same problem in a short period of time.

The output of the tool is a detailed report which provides an overview of the security vulnerabilities that occurred inside CERN's network during the last 24 hours, as well as a command line tool for whitelisting and managing already whitelisted devices.