Exploring Secure and Efficient Temporary Data Sharing between co-located Kubernetes Containers

Data-intensive workloads like distributed deep learning generate significant temporary data between pipeline stages, requiring efficient temporary data storage solutions. Platforms like Kubernetes abstract infrastructure complexities by using loosely-coupled containers that leverage disaggregated storage for sharing data and storing state. This design facilitates workload deployment and scaling in the Cloud-Continuum, but also becomes a bottleneck, particularly in resource-constrained edge environments. A tiered storage system for temporary data can alleviate this issue by exploiting data locality: leveraging local storage attached to the physical hosts to share data between co-located containers without the need of accessing remote storage. This article serves as a preliminary work to explore how to efficiently and securely share data between co-located containers on the same physical host for workloads deployed on Kubernetes. Our experiments demonstrate that using CSI shared local volume mounts can be used not only to share files in an efficient and secure way but also to create shared memory regions and pass file descriptors, providing different viable approaches to data sharing for layered storage systems in Kubernetes.