On the Design and Architecture of Deployment Pipelines in Cloud- and Service-Based Computing - A Model-Based Qualitative Study

DevOps and Continuous Delivery (CD) are becoming the de-facto standard for software deployment in the cloud. Deployment pipelines are a core artefact in such practices, but so far their design is largely discussed informally, and existing sources on DevOps and CD practices are often inconsistent or incomplete. The lack of a generic, complete, tool-agnostic, and application-independent treatment of deployment pipeline design and architecture impedes their understanding by human designers and the creation of generic tools that work across different technologies and applications. To alleviate this problem, we have performed a qualitative, in-depth study of 25 deployment practice descriptions by practitioners containing informal deployment pipeline models. From our study we derived a precisely specified model of deployment pipeline architectures. We can show that the formal model substantially increases the precision of the modelling compared to the informally modelled pipelines in the original sources.


I. INTRODUCTION
A core trend in cloud-and service-based computing is the steady increase of the frequency of change required in those systems: continuous releases are becoming the expected norm rather than the exception. Deployment pipelines [1] are the centrepiece of most DevOps and CD practices. So far the design of deployment pipelines is largely discussed informally (cf. [2], [3]). At a first glance, such informal models by practitioners give a good overview of all deployment structures. However, experience in the field and a detailed study of these informal descriptions immediately reveal that almost all realistic deployment pipelines require complex architectures of various tools and components, including deployment target environments and tool integration architectures (cf. [1]). The complexity of this overall architecture is usually not covered well in the informal models, nor are all aspects of the deployment pipeline itself. This impedes building generic tools operating on the whole pipeline, such as architecture consistency checkers or static analysis tools. Common abstractions, and consequently interfaces, are missing, which hinders understanding by human designers beyond a single set of technologies, environments, and applications.
To alleviate these problems, we have performed a qualitative, in-depth study of 25 deployment practice descriptions by practitioners containing informal deployment pipeline models. We followed the model-based qualitative research method described in [4], based on the established Grounded Theory (GT) [5] qualitative research method, together with methods for studying established practices like pattern mining and their combination with GT [6]. The knowledgemining procedure is applied in many iterations until reaching theoretical saturation [5], as is widely accepted in qualitative research. In our study, we decided to stop our analysis when 7 additional knowledge sources did not add anything new to our understanding of the research topic. This is a rather conservative operationalisation of theoretical saturation; our study converged already after 10 knowledge sources in the sense that no substantial new formal model elements were created. We used this method to examine the following research questions: RQ1 What are recurring established practices for designing deployment pipeline structures?
RQ2 What are the relevant environments in deployment pipelines?
RQ3 What are the architectural elements relevant for building a deployment pipeline infrastructure?
Our result is a precisely specified model comprising views for modelling deployment pipeline structures, deployment environments, and infrastructure architecture. We also precisely define the links between the model elements in each view, as well as consistent links between the different views. For each of the informal pipeline models studied, we contribute a precisely modelled instance of our model. Finally, in a preliminary evaluation, we can show that the formal models result in a total average improvement of 134.72% in modelling accuracy compared to the informally modelled pipelines in the original sources 1 .

II. RELATED WORK
Informal descriptions of deployment pipelines and associated architectures dominate the literature (cf. [7], [1], [3]), but they usually fail to fully cover either the complexity of these designs and architectures or all aspects of the deployment pipeline itself, and are often inconsistent. While many scientific works use and improve deployment pipelines (cf. [8], [9]), and first studies on deployment practices in organizations have emerged [10], [11], a generic, tool-agnostic, and application-independent treatment of deployment architectures is missing today. Our study aims to provide the first systematic and precise specification approach for CD architectures, laying the foundations for automated quality control of the design of deployment architectures. This would enable checking, e.g., whether a pipeline design performs too few or too manual quality controls, is missing important steps (like forgetting to model a commit trigger), links to the environment (like a cloud test environment that is launched but not torn down), or is performing time-consuming or resource-intensive steps too early. So far, such design issues have been identified in the literature as red flags [1], [2], but their automatic detection is not possible as it requires precise models of all elements of the deployment architectures.Many sources also point at the need to substantially alter the architecture of the target systems for supporting rapid releases [9], [10], [11]. For instance, decomposition of a system into microservices is extensively studied (cf. [8], [12], [13]), but the impact on the associated deployment architectures has not yet been studied in a systematic way. Our work aims to provide the groundwork needed to perform such studies, by enabling formal reasoning, validation, and verification through a precise and consistent modelling foundation for the CD parts. All the tuple elements are defined in the subsections below. We first discuss two prerequisites for modelling deployment pipelines: components of the deployment infrastructure and deployment environments. Then we discuss specifying the structure of the deployment pipeline. We have made our generated models available online 2 .
Each aspect of our model is discussed in two parts: First we discuss generic modelling notions that should suffice for modelling the elements of a CD model instance and their relations. Second, based on the recurring CD-specific elements found in our study, we specify CD-specific set members and rules (all summarized in Table I). For instance, 2 https://swa.univie.ac.at/cd-pipeline-models/cd-pipeline-models.zip we first define the generic notion of a pipeline node and then specify all the possible pipeline node types and their type hierarchy relations that we have observed in our study in Table I. We expect that the generic aspects will likely remain stable in the future, whereas the elements in Table I might require changes or extensions. Please note that we consider this list of elements in Table I complete with regard to the sources we have studied, but these sets and rules can be extended or redefined when using or applying our model (e.g., for modelling CD/DevOps aspects we have not yet covered in our study, or for future technologies).

A. Modelling the Deployment Infrastructure Architecture
An important result of our study was that the focus of informal descriptions of deployment pipelines is only in exceptional cases solely on the structure of the pipeline. Instead, almost always the components which represent the infrastructure of the deployment pipeline, such as continuous integration (CI) tools or deployment pipeline orchestration components, and their interconnections are described as well. To formally capture this, we first model component nodes and their connectors: CP is a finite set of component nodes. CN ⊆ CP × CP is a finite set of connector edges.
CD infrastructure components are typically categorized along their main function, which can be modelled using types in type hierarchies. For example, deployment pipeline orchestration and package tools are important recurring types of such components, and package tool is a subtype of development tool. In our model, component types are defined as follows: CP T is a finite set of component types. stype CP T : CP T → P(CP T ) is a function called component type hierarchy. stype CP T (cpt) (with cpt ∈ CP T ) is the set of direct supertypes of cpt; cpt is called the subtype of those supertypes. The transitive closure stype * CP T = ∞ i=0 stype i CP T defines the inheritance in the hierarchy such that stype * CP T (cpt) (with cpt ∈ CP T ) contains the direct and indirect supertypes of cpt. The inheritance hierarchy is cycle free, i.e., ∀cpt ∈ CP T : stype * CP T (cpt) ∩ {cpt} = ∅. dtype CP : CP → P(CP T ) is a function that maps each component node cp ∈ CP T to its set of direct component types. type CP : CP → P(CP T ) is a function that maps each component node cp ∈ CP T to its set of direct and transitive types, i.e., ∀cp ∈ CP, dt ∈ dtype CP (cp) : . CD infrastructure connectors have types and a type hierarchy, too; e.g., components can launch another component or read an artefact from another component, and here launch and read are connector types. In our model, CN T is a finite set of connector types. It has a type hierarchy definition exactly identical to the one of CP T (see specification above) with analogous function definitions for stype CN T , dtype CN , and type CN (omitted here for brevity).
In our study we found a number of recurring types of components and connectors used in infrastructure architectures of deployment pipelines. All component and connector types that were included in our study according to our inclusion criteria, as well as their relations in two type hierarchies, are formally defined in the first four rows of Table I.

B. Modelling Deployment Environments
A second prerequisite for precisely specifying a deployment pipeline, which is used in almost all our sources, is the notion of deployment environments. They are used first to model the deployment environments to which the pipeline deploys, such as a test environment in a virtual private cloud or a production environment in a public cloud. Second, they are used to describe the environments in which the deployment infrastructure (see previous section) itself is deployed. For instance, sometimes the deployment pipeline orchestrator or a continuous integration tool run in the same cloud environment the system is deployed to, or a local or server environment are distinguished from a cloud environment if both are used in a pipeline.
The main deployment environment elements of our model are the deployment nodes: DN is a finite set of deployment nodes. These can be connected with each other, such as a production and a test environment running on a cloud environment: DN R ⊆ DN × DN is a finite set of deployment node relations. Different types of relations might exist such as part-of, connects-to, or runs-on: DN RT is a finite set of deployment node relation types. type DN R : DN R → DN RT is a function that maps each deployment node relation dnr ∈ DN RT to its type.
Components of the deployment infrastructure have relations to these deployment nodes: DR ⊆ CP ×DN is a finite set of deployment relations. Different types of deployments exist such as deployed-on, uses, or launches: DRT is a finite set of deployment relation types. type DR : DR → DRT is a function that maps each deployment relation dr ∈ DRT to its type.
There are specific kinds of deployment nodes: DE ⊆ DN is a finite set of devices. EE ⊆ DN is a finite set of execution environments. EE is used to model the environments a system can be deployed to, which is modelled in a type hierarchy: EET is a finite set of execution environment types. It has a type hierarchy definition exactly identical to the one defined for CP T (see specification above in Section III-A) with analogous function definitions for stype EET , dtype EE , and type EE (omitted here for brevity).
Again, we have specified those CD-specific set members and type hierarchy rules that we have observed in our study as well. Rows 5-8 of Table I contain formal definitions for the environment types, their type hierarchy, and their relations that we have observed in the informal deployment pipeline descriptions analysed in this study.

C. Modelling Deployment Pipeline Structures
Deployment pipelines are often modelled as behaviour models resembling UML activities. As a basis for mod-elling pipeline specifics we thus have chosen abstractions resembling the basic elements of activities -but excluded all abstractions of activities in UML that we have not empirically observed in our study -to keep our model much simpler than UML activities: AN is a finite set of activity nodes. AE ⊆ AN × AN is a finite set of activity edges. CON ⊆ AN is a finite set of control nodes. IN ⊆ CON is a finite set of initial nodes. F IN ⊆ CON is a finite set of final nodes. F ON ⊆ CON is a finite set of fork nodes. JON ⊆ CON is a finite set of join nodes. DEN ⊆ CON is a finite set of decision nodes. M EN ⊆ CON is a finite set of merge nodes. ACT ⊆ AN is a finite set of actions. AEA ⊆ ACT is a finite set of accept event actions. AT A ⊆ AEA is a finite set of accept time event actions. SSA ⊆ ACT is a finite set of send signal actions.
All special kinds of nodes in a deployment pipeline are subsets of some of those activity nodes. In addition they are subsets of P E which is a finite set of pipeline elements.
For P E we define a number of functions used to specify important properties of pipeline elements. aut : P E → {T rue, F alse} is a function that determines whether a pipeline element is automatically processed in the pipeline or requires manual work. The following functions are used to specify important links to infrastructure components and environments (as defined in the previous sections): run : P E → CP is a function that determines the component this pipeline element runs in. inv : P E → {(l 1 , l 2 , ..., l n ) : l 1 ∈ CP, l 2 ∈ CP, ..., l n ∈ CP } is a function that determines the components this pipeline element can invoke. inp : P E → {(l 1 , l 2 , ..., l n ) : l 1 ∈ CP, l 2 ∈ CP, ..., l n ∈ CP } is a function that determines the components providing inputs to a pipeline element (like an artefact passed to a pipeline element). out : P E → {(l 1 , l 2 , ..., l n ) : l 1 ∈ CP, l 2 ∈ CP, ..., l n ∈ CP } is a function that determines the components providing outputs of a pipeline element (like an artefact produced by a pipeline element) env : P E → {(l 1 , l 2 , ..., l n ) : l 1 ∈ DN, l 2 ∈ DN, ..., l n ∈ DN } is a function that determines the deployment nodes used by a pipeline element.
The core element in a typical deployment pipeline are pipeline nodes, modelled as elements of P N (with P N ⊆ P E, P N ⊆ AN ) which is a finite set of pipeline nodes. P N T is a finite set of pipeline node types. Exemplary CD-specific P N members are pipeline nodes for building, packaging, unit testing, and so on. P N T has a type hierarchy definition exactly identical to the one defined for CP T (see specification above in Section III-A) with analogous function definitions for stype P N T , dtype P N , and type P N (omitted here for brevity).
Mainly for triggering the pipeline, we further define special accept event actions: P AE (with P AE ⊆ P E, P AE ⊆ AEA) is a finite set of pipeline accept event actions. P AET is a finite set of pipeline accept event action types; it is used to model for instance a trigger by a commit  event vs. a manual trigger. type P AE : P AE → P AET is a function that maps each pipeline accept event action pae ∈ P AET to its type.
To model commit events (which can also happen during a pipeline run), we model a special send signal action: P SS (with P SS ⊆ P E, P SS ⊆ SSA) is a finite set of pipeline send signal actions. P SST is a finite set of pipeline send signal action types. type P SS : P SS → P SST is a function that maps each pipeline send signal action pss ∈ P SST to its type.
Finally, for defining decision such as approval gates, we model a special decision node: P DN (with P DN ⊆ P E, P DN ⊆ M EN ) is a finite set of pipeline decision nodes. P DN T is a finite set of pipeline decision node types. type P DN : P DN → P DN T is a function that maps each pipeline decision node pdn ∈ P DN T to its type.
Those CD-specific set members and type hierarchy rules that we have observed for pipeline elements in our study are specified in rows 6-11 of Table I. They contain formal definitions for the environment types, their type hierarchy, and their relations we have observed in the informal deployment pipeline descriptions analysed in this study.

IV. THREATS TO VALIDITY
To increase internal validity we decided to use practitioner reports that were produced independent of our study. This avoids any bias, e.g. compared to interviews in which the practitioners would have known that their answers are used in a study. However, this introduces a different internal validity threat: Some important information might be miss-ing in the reports, which would have been revealed in an interview. We tried to mitigate this threat by looking at many more sources than needed to reach theoretical saturation, as it is unlikely that all different sources miss the same important information. The different members of the author team have cross-checked all models independently to minimize researcher bias. The threat to internal validity that the researcher team is biased in some sense remains, however. The experience and search-based procedure for finding knowledge sources may have introduced some kind of bias as well. However, this threat is mitigated to a large extent by the chosen research method, which requires just additional sources corresponding to the inclusion and exclusion criteria, not a specific distribution of sources. Due to the many included sources, it is likely our results can be generalized to a larger population of similar pipeline models and architectures. However, the threat to external validity remains that our results are only applicable to similar kinds of pipeline models and architectures; generalization to novel or unusual pipeline models and architectures might not be possible without modification of our models.

V. CONCLUSION
We have performed a qualitative study in which we have studied the design and architecture of deployment pipelines from 25 unique and independent sources. Our study led to a detailed model precisely describing the recurring pipeline structures and their links as an extension of mainly activity model abstractions to answer RQ1. To answer RQ2, we have extended mainly deployment model abstractions to specify the environment in which deployment pipelines run and to which they deploy. Finally, to model the deployment pipeline infrastructures for RQ3, we have extended mainly component model abstractions. In all three cases, we observed theoretical saturation relatively early and could precisely model almost all main concepts described in the original sources. This leads us conclude that the found models are very likely adequate representations of the original sources and can express almost all major concepts expressed therein. We have thoroughly cross-checked all models independently by the different researches in the author team to minimize researcher bias. In addition to the DevOps models as our major contribution, another contribution of our study is a set of 25 formal CD model instances. These might be useful as a basis for further research in the area. In this paper we have used formal models and detailed object notations to present the details of our approach; in practice the models should be rendered using more appealing notations e.g. akin to UML component, deployment, and activity diagrams, which is easily possible as an extension of our model-driven tools. As future work we plan to realize constraint checkers to implement static analysis tools for deployment pipeline architectures. For such tools, precise abstractions as provided in this paper are a necessary prerequisite.