Tech Notes: A Review of Streaming Systems and Architectures

Sometimes science happens slowly, sometimes it comes at us like a firehose. Experiments that produce huge volumes of data, e.g. scientific instruments producing streaming data, are becoming more common with time. Capturing, processing, storing, and sharing high-volume, high-speed scientific data is a challenging problem and one that often requires a mix of off-the-shelf tools and custom, bespoke solutions.

In this technical note, we will take a short look at the challenges in more detail, highlight some existing tools that are useful to address these challenges and discuss some general strategies and design principles for those times that you have to take data matters into your own hands.