Real Time Stream Processing in Astronomy

One of problems with “Big Data” is when large quantities of data have to be analyzed fast - (almost) in real time. So far in astronomy those situations were not very common. In the next few years a real time processing will become more relevant and the ﬁeld should prepare for the advent of new instruments and techniques.


Introduction
A nature of most astronomical objects is that they change slow or very slow and are hard to detect on timescales available to astronomers. Of course there are some objects/phenomena which are changing very rapidly. Timescales of changes of physical properties of astronomical objects range from milliseconds to millions of years.
Historically there were long term observational programs where specific, more interesting objects, which showed unusual changes, were followed for many months or years. Also, there were short term observational campaigns which were focused on specific objects, trying to understand their nature.
The Zwicky Transient Facility (ZTF) is a new time-domain survey that started operations at Palomar Observatory in the beginning of 2018. ZTF searches and catalogs rare and exotic transients and its data will drive novel studies of supernovae, variable stars, binaries, AGN and asteroids ZTF started to disseminate part of their detected events in mid-2018 (?). This is precursor of Large Synoptic Survey (LSST, which will come online 2022/23, ?) and the cadence of observations is similar (though depth is significantly shallower). Number of alerts about detected events per ZTF visit is in range of few hundreds to thousand while with LSST we can expect few tens of thousands. This paper deals with technologies and services which will possibly allow astronomers / observatories to easily handle vast quantities of data coming from these new facilities.

Event processing concepts
Event processing is an umbrella term for technologies that are conceptually centered around ingestion, manipulation and dissemination of events. An event could be any discrete occurrence, depending on the system ontology and granularity of its concepts. Event processing enables real-time pattern application, suitable for large input streams where automatized reaction is crucial.
Main difference between event processing applications and common transactional systems is their query paradigm. While traditional applications make queries and serve previ-ously stored data per request, event processing applications run continuous queries through streams of new data and deliver discoveries if and when they happen.
Some of the use cases for event processing are: • Real time analytics • Monitoring, fault detection • Diagnostics made in conjunction with historical data • Data is incoming continuously, organized in streams • Huge loads of data which are a challenge to process with traditional applications Concept of data windows is as autochthonous and iherent Complex Event Processing mechanism. Windows are basis for aggregation and most of the CEP patterns. Pattern matching is another key characteristic of CEP systems. Patterns shape multiple simple events into a single complex event. Pattern formulation is commonly given through definition of presence or absense of events, in a certain temporal interdependence.

Application of Event Processing to Astronomy
Esper EPL is a specialized language for event processing based on SQL query language. We chose to present CEP concepts with examples coded in EPL.
Given here is the mathematical model of a "mag per hr" pattern, given in pseudo code. It detects change in brightness of an object and triggers an alert if it increases at a rate of 1 magnitude per hour or quicker (with a constraint that object was previously undetected). The pattern may be used for GRB detection, or with modified constraints for detection of some other phenomena.

Conclusion
In the new data rich environment which will start at early 2020's one can expect that real time stream analytics need tools both scalable and expressive. We described one possible direction in achieving this using CEP engines.