Published June 22, 2026 | Version v1
Conference paper Open

A Microservice-based Architecture for Reproducible AI Pipelines

  • 1. ROR icon University of Cyprus
  • 2. UBITECH
  • 3. UNINOVA-CTS
  • 4. Associated Lab of Intelligent Systems (LASI)
  • 5. IDEA Institute

Description

As Artificial Intelligence (AI) systems move from prototypes to deployment, reliability and trustworthiness are increasingly limited by the data and AI pipeline lifecycle rather than by model training alone. Existing platforms offer strong support for individual lifecycle functions such as orchestration, experiment tracking, validation, or documentation, yet these capabilities are often adopted as separate tools, making provenance, observability, governance, and safe adaptation difficult to manage end-to-end. This paper presents the microservice-based reference architecture of the AI-DAPT project for reproducible AI pipelines whose core contribution is the unification of data- and model-centric operations within a single lifecycle-oriented platform. The architecture is organized around five iterative phases, namely data design, data sculpting, data generation, model delivery, and data/model optimization, while treating metadata, lineage, explainability, security, and human-oversight as native platform services rather than peripheral add-ons. The paper further contributes a trustworthiness-by-design architectural view that connects data preparation, valuation, synthetic data generation, orchestration, monitoring, adaptive retraining, and AI security through interoperable services. Finally, it presents the current realization of this architecture using widely adopted orchestration, storage, monitoring, and experimentation technologies, thereby offering a practical blueprint for building adaptable, observable, and governable AI pipelines in real-world settings.

Notes

The version of the paper here available is the author's Accepted Manuscript (AM).

Files

Architecture_IEEE_ICE-final.pdf

Files (819.9 kB)

Name Size Download all
md5:48d9102db67e8c186b7cde38f6293aa0
819.9 kB Preview Download

Additional details

Funding

European Commission
AI-DAPT - AI-Ops Framework for Automated, Intelligent and Reliable Data/AI Pipelines Lifecycle with Humans-in-the-Loop and Coupling of Hybrid Science-Guided and AI Models 101135826