The ALMA Data Science Initiative: Building a Data-driven Organization to Improve Operations

Building fully functional and successful observatories like ALMA requires the use of cutting-edge technologies and state-of-the-art engineering development. However, just 15 years ago most of the ground-based observatories worked like laboratories, and our field is still learning how to apply other disciplines, like operations research, system engineering and optimization, to find the way to operate these "big data factories" in the most efficient and effective way possible. One important aspect is the ability to make decisions based on data. We need to answer questions like 'how are we doing', 'why are things going well or bad', 'what is the reason for the success or for the failure', 'how can we improve the success rates or prevent failures', etc. to make informed decisions. Since ALMA stores a huge amount of data every day, and not just science data, it was easy to realize that we had a gold-mine waiting to be exploited.

Four years ago, ALMA started a collaboration with the industry (Dataiku) to provisionalize a platform and infrastructure that would allow us to use this data and our skills to answer these questions and make better decisions. This allowed us to understand what were our weaknesses, which are our strengths and what challenges we must still solve. The purpose of this talk is to share with the audience what we have learned, what have we achieved and what is our road ahead, in the process of applying data science to improve operations.