D4.4 Containerized workflows

One important part of the LUMI AI Factory Service Center's mission is to contribute to empowering Europe's AI ecosystem by developing and making available ready-to-use AI software solutions. Central to this effort is the provision of Containerized Workflows, which simplify the deployment of AI capabilities across various applications. The Containerized Workflows consists of pre-built container images with tailored software, example data sets and pre-tested models to improve the accessibility of AI usage and development. The development of these workflows is linked with the broader LUMI AI Factory software ecosystem, which includes High-Performance Computing API (HPC-API) and Machine Learning Operations (MLOps) tools. Together, these elements will form an integrated environment where the development of different components supports each other. For instance, the container base images for various parts of the ecosystem, and the Containerized Workflows will share the same foundations. The key objectives of developing the Containerized Workflows concept include delivering the first complete workflows early on. Additionally, new workflows will be continuously added throughout the lifetime of the LUMI AI Factory Service Center based on emerging opportunities and user needs. Initially, the focus is on developing workflows for Large Language Models (LLMs), including text processing workflows for tasks such as dataset curation, LLM fine-tuning, and LLM evaluation. An important part of this development effort involves establishing processes, best practices, and tooling for creating these workflows. The concept of Containerized Workflows aims to include everything necessary to help new users get started quickly. It also provides a proven starting point for expert users to build upon for more specialized tasks. To support this, documentation with examples, openly licensed models tested with the workflows, and example datasets will be available directly within the environment. This document provides an update on the status of the Containerized Workflows development and outlines the next steps in this ongoing effort.