Quality attributes use in architecture design decision methods: research and practice

Over the past 10 years software architecture has been perceived as the result of a set of architecture design decisions rather than the elements that form part of the software design. As quality attributes are considered major drivers of the design process to achieve high quality systems, the design decisions that drive the selection and use of specific quality properties and vice versa are closely related. Consequently, quality attributes must play a role for decision making processes and be documented alongside the decisions captured. Consequently, we conduct a systematic literature review to study the importance and impact of the relationships between quality attributes and architecture design decisions and to what extent existing architecture knowledge management methods and tools deal with the decisions that affect the quality of a system. We also report on the challenges and future research paths for architectural knowledge management methods and tools. Our results reveal important explicit relationships between both software artifacts, the role of uncertainty in decision making and empirical studies reporting the use of quality attributes in architecture knowledge management activities.


Introduction
Since the early 2000s, the software architecture community perceives design decisions as first-class artifacts that should be captured alongside the standard software architecture documentation [23].Today, many of the existing decision making approaches demand the capture and documentation of multiple alternatives that have to be evaluated during the development and evolution of the system.However, as making decisions often implies that competing requirements must be satisfied for different stakeholders' concerns, the evaluation of quality attributes (QAs) [8] in the architecture may trigger additional decisions that must also be evaluated.For many years, QAs [9] have been used in the architecture to describe the non-functional properties of systems and are primarily addressed in the early phases of the design activity.Although software architects make design decisions to depict the major functional parts in the design, we need to understand the impact of quality attributes in the architecture as well as important decision drivers [30,54,55].
Some authors [21,47,51] also highlight the role of design patterns, as a kind of architectural knowledge or proven design decisions, to address the quality requirements of a system.Although some of the existing architecture decision methods consider the role of the QAs implicitly or explicitly, satisfying the various trade-offs of QAs during any decision making process as well as the interplay of decisions and quality properties still remains a complex and challenging task.
Existing architecture design decision methods and tools have already been analyzed in other works but with a different focus and aim.For instance, Capilla et al. [13] provide an informal retrospective analysis of AK management research while Falessi et al. [19] examine various techniques for selecting design alternatives during decision making.Tofan et al. [46] conducted a systematic mapping study as an overview of existing architecture decision documentation approaches in which the authors discuss functional and nonfunctional requirements in the context of architecture decisions, but they do not analyze in detail the relationships between quality attributes and architectural decisions.
Although previous studies discuss the relationship between architectural design decisions (ADDs) and quality attributes (QAs), they do not provide a full perspective discussing dimensions such as uncertainty, dependency types between decisions, or QA trade-offs, and do not analyze in depth such characteristics we do in our study.Therefore, apart from analyzing different dimensions and challenges in the overlap of ADDs and QAs, we provide a deeper analysis classifying each selected paper according to different criteria.
To the best of our knowledge, the use and integration of QAs in existing AK management (AKM) methods and tools have been under-investigated so far.Consequently, our main contribution in this research is the investigation of the role of QAs in existing AK approaches as well as the support of QAs in existing architecture decision methods and tools.We also provide an integrated view of the relationships between QAs and AK based on the three aforementioned dimensions and supported by a deeper analysis of approaches that report industry case studies.
The paper is structured as follows.In Section 2, we clarify the two key terms relevant to our SLR, architecture design decision and quality attribute.In Section 3 we present the research method we have applied to extract the relevant literature.In particular, we introduce the rationale of our literature review, discuss our search strategy and process as well as the selection criteria we applied, and we state the three research questions under investigation.We go into the details of the research's results in Section 4-6.Section 7 discusses our findings and future research directions, as well as the limitations of our study, and, finally, in Section 8, we draw the conclusions of our work.

Terminology and Conceptual Model
Architecture Design Decision (ADD): According to ISO/IEC/ 42010:2011 [2], an ADD affects various architectural elements, pertains to or raises concerns, and is justified by architecture rationale.Ven et al. [49] understand design decisions as a first-class artifact that couples rationale with software architecture.Approaches using templates for capturing ADDs [48] and are supported by existing meta-models [54] and tools [40,44,46] set the focus on architecture knowledge management.
Quality Requirement, quality attribute and quality property: According to ISO/IEC/IEEE 25010:2011 [1] a quality requirement (QR) is defined as "a requirement that a software quality attribute be present in software," while a quality property (QP) is understood as a "measurable component of quality."In addition, quality attribute (QA) is defined by [9] as "a measurable or testable property of a system that is used to indicate how well the system satisfies the needs of its stakeholders." QAs are orthogonal to functionality [9] and the choice of a functionality in a design decision does not dictate the level of quality of such a functionality.Therefore, wheter ADDs are driving the selection of the qualities of the system or the other way round, QAs are seen as major decision drivers [9,30,54,55].
In order to define a conceptual relationship between QAs and ADDs, we describe the following conceptual model depicted in Fig. 1, which illustrates the main entities used to create the significant design decisions that require quality properties as decision drivers or even incomplete knowledge to make a decision.The decisions can be used to select the most suitable qualities of interest for a particular system (e.g.security and availability are major concerns for banking systems) or the other way around, where quality attributes drive the selection of a particular decision (e.g.select a middleware with high performance).As decisions are connected to other decisions, it is common that all the decisions made form a decisions network, and software architects can use this network to evaluate the impact of decision changes during evolution cycles.Finally, the set of decisions made should lead to a solution architecture that fulfills the requirements and constraints that motivated the decisions taken.

Review Method
Following the guidelines provided by Kitchenham et al. [26] we run a systematic review to investigate in depth the relationships and role of QAs and ADDs.According to Kitchenham et al. [25], systematic reviews comprise the following three phases: (1) planning i.e. stating the rationale of the review and editing of the review protocol, (2) execution i.e. conduction of the review according to the review protocol, and (3) reporting i.e. presentation of the review results.
As we found that the topic relating QAs and ADDs in AKM approaches has been poorly treated, we perceived the need to analyze which research works better highlight the role of QAs and their relationships in existing AKM approaches, tools, and case studies in order to derive meaningful findings.We summarize the main objectives of our study below: 1. Review and better understand the relationships of QAs and ADDs.
2. Review and better understand the focus of existing AKM methods and tools, especially with regard to the challenges they address.3. Find evidence about how the relationships between QAs and ADDs are modeled and documented.4. Review and better understand implications for researchers and practitioners interested in AKM in the current state of the art.

Research Questions
In the context of the literature review, based on the aforementioned objectives, we considered the following research questions: RQ1 What is the role and use of QAs and their relationships to ADDs in existing AKM methods and tools?RQ2 Which challenges related to managing QAs are frequently addressed by existing AKM methods and tools?RQ3 What is the size and scope of industry case studies of existing AKM methods and tools with regard to QAs?
In order to organize the results of the research questions, we describe the concepts and research questions in Fig. 2 to guide the reader through the results.For research question RQ1, we are interested in whether the QAs appear in the decision making process, in the documentation of the design decisions, or both.We also model different types of relationship between the decisions and the quality attributes (e.g.explicit, not explicit, etc.).The evaluation of ADDs using QAs can be supported in different ways such as the ones indicated in the values of the figure.Regarding research question RQ2, we first look for whether there is uncertainty using the QAs in existing architecture design decision methods.In addition, we searched for different types of dependency between the QAs and whether these appear explicitly as well as the trade-offs between QAs.Finally, research question RQ3 explores the size of the design space and scope of the different approaches using QAs and ADDs, and the evaluation method used.In the figure, We also explicitly model some interdependencies between different concepts belonging to the research questions.These are represented by solid arrows in Fig. 2. For instance, QA uncertainty from RQ2 has a clear impact on the evaluation of ADDs using QAs (cf., Research Question RQ1), as every time we evaluate a design decision we need to know if the qualities exhibit incomplete information.

Search Strategy
To conduct the systematic review, we queried four digital publication libraries: 1) the Association for Computing Machinery (ACM) Guide to Computing Fig. 2: Relationships between the main concepts and research questions that organize the results shown in the tables in Section 4-6 Literature1 , 2) the IEEE Xplore Digital Library2 , 3) DBLP3 , and 4) Springer 4 .The search string we used entails the most appropriate keywords related to the scope of this SLR, namely the terms "quality attributes", "non-functional requirements", "design decisions", and so on.The query we performed in the four digital libraries can be expressed as a Boolean formula: software architecture AND (quality attributes OR non-functional requirements OR nfr OR design decisions OR architecture knowledge OR architectural knowledge) The queries performed on the libraries refer only to the titles, as additionally searching abstracts and content would produce an enormous amount of matches, unmanageable to be inspected manually.However, as Springer does not allow the filtering of only the titles, in this case we searched the overall contents of the articles.As our search strategy might miss useful citations, we used backward and forward snowballing [50] to identify additional candidate citations.In backward snowballing, the references of the selected articles are checked for useful publications, while in forward snowballing the publications that cite the selected articles are examined.In both cases, snowballing finishes when a convergence is reached and no new articles can be found.Because different digital databases provide different search facilities, it was not possible to use one single search string in all the cases to identify all the relevant sources.We summarize in Table 1 the different queries we used in each digital library.

Selection Criteria
In order to extract the publications to be used as basis for answering our three research questions, we defined appropriate selection criteria.Inclusion/exclusion criteria were used to select publications focusing on methods and tools for ADD support and documentation with a focus on QAs, and to reject papers that focused on other forms of AKM or did not refer explicitly to ADDs or QAs.Therefore, the inclusion and exclusion criteria are as follows: Inclusion Criteria (IC) IC1: The primary study is a peer-reviewed scientific paper that introduces a technique, method, or approach about AKM or ADDs using QAs.IC2: The primary study reports a case study about the use of QAs in architecture decision making or on an AKM tool.IC3: The primary study is the most recent version.IC4: The paper is written in English.

Exclusion Criteria (EC)
EC1: The primary study does not deal with QAs in AKM explicitly or implicitly.EC2: The primary study does not consider the tandem QA-ADD, even if it belongs to the software architecture field.EC3: The primary study does not include an evaluation of QAs using ADDs.
EC4: The primary study is an older publication from the same authors about the same approach or a duplicated study.EC5: The primary study is a short paper with less than five pages.EC6: The primary study is a non-peer-reviewed publication.

Search Process and Selected Publications
The results of the queries we performed for the period between 01/2001 and 10/2018 reported the following numbers: ACM DL (1070 papers), IEEE (785 papers), DBLP (1509 papers), and Springer (2032 papers).In the case of the Springer database, we selected only those papers belonging to the software engineering discipline (i.e."Software Engineering" keyword).In addition, as depicted in Fig. 3, we merged and aggregated the results from the four databases.Once we have removed the duplicates, we came up with 4571 papers.We used the title to filter out the research works according to the inclusion/exclusion criteria which resulted in 73 papers.According to [50], we carried out a snowballing process to find additional papers.Using this technique, we carried out two backward-forward iterations in the proposed four databases, except for Springer where we needed three iterations, and we found only four additional papers.This proves that the selection of the queries was precise enough in order to find the relevant papers.After the iteration process, we manually filtered out the papers (77 papers) according to the inclusion/exclusion criteria, This was done by two of the paper's authors (Universidad Rey Juan Carlos -URJC), then a second time by the other two authors (University of Vienna -UV).We reviewed the contents of the papers in order to perform an accurate selection regarding those works relating ADDs and QAs.For instance, we did not consider the work of [27] on capturing software design decisions in our selection as the tool they introduce aims at annotating ADDs in software related artifacts rather than on architecture decision making and documentation, and does not consider the use of QAs.Other examples include the work of [42] which on the one hand discusses the prioritization of QAs for different stakeholders' concerns, but on the other hand, it does not contain any explicit reference to ADDs.Apart from that, we excluded works that referred to the same approaches and/or tools and kept only the most recent or most detailed publications (e.g.journal or conference papers instead of workshop papers).The papers were selected based on careful reading of the abstracts, introduction, and conclusions but we read in depth all the 77 papers selected for manual review to ensure their compliance to the inclusion/exclusion criteria and we finally selected 36 papers.All the papers selected were reviewed by the four authors of this research work.As a result, we consolidated a list of 36 publications (four journal, 28 conference and four workshop publications) and we classified the selected publications into three focus areas according to our research questions.
Table 1 in the Appendix5 lists the selected papers along with the publication year and place as well as the name of the method or tool used for architecture decision making and documentation.The majority of the publications (28) are from the years 2007-2018, which gives an indication of the increasing interest of the software architecture community in ADDs in recent years.In most of the cases, the proposals are accompanied by tools for software architects.
4 Role and Importance of QAs in ADD Tools and Methods (RQ1) The importance of QAs in decision making in software architecture [9,30,54,55] and the fact that good decisions affect the selection of which QAs are important for a system during architecture reviews [8] are widely recognized.
In order to answer RQ1, in this section we discuss the role of QAs in AK approaches.More specifically, we will investigate at what point in time the QAs appear in the decision making process or are just documented, the explicit and implicit relationships between ADDs and QAs, how decisions are evaluated using QAs, and how QAs are used to decide on the best alternative.In the following subsections, we discuss in particular the selected publications with regard to the following aspects: 1. Appearance of QAs: We distinguish between tools that integrate QAs in decision making, in ADD documentation, or in both.2. Relationships between ADDs and QAs: We investigate the relationships between QAs and ADDs and whether these are explicitly or implicitly documented.We distinguish between "not explicit" and "explicit" trace links support and identify those approaches that contain explicit links supported by some kind of evaluation method.3. Evaluation of decisions and qualities: Some approaches use QAs to evaluate the best or the optimal alternative decisions.However, other research works focus on how to make design decisions in order to select one or several quality requirements that are more important for the architecture and hence, fulfill the quality requirements of a system.

Appearance of QAs
It is of key importance to document the QAs that will drive the shape of the architecture and the decisions made during the architecting activity in order to prevent AK vaporization.In the primary studies, we observed three major trends regarding the "Appearance of QAs" (see RQ1).The values of the second column in Table 2 in the Appendix reflect these trends.The first trend is to document QAs in approaches using AK.The second trend is documenting QAs in the decision making process.Finally, the third set of research works document QAs in both documentation and decision making.In the first case, QAs are described explicitly in the documentation alongside the design decisions.This approach is taken by two of the papers examined excluding those that appear in both documentation and decision making.The second trend describes how the QAs are documented and used in the decision making process.Many of the works highlight the importance of QAs in the reasoning activity and how QAs act as major decision drivers (e.g.[5,6,31,55]).In some of these works, QAs play a key role for architectural trade-off decision making that is sometimes carried out using multiple attributes [32] and constraints [5].Different criteria and methods can be used to evaluate the role of QAs in the decision process based on e.g. using utility trees [6] and recommender systems [29].
Corresponding tool support can assist software architects in the decision making process using QAs for evaluating alternative decisions.A few tools provide automated support for managing the interactions between QAs and decisions such as the ArchiTech tool, an application of the Quark approach [5].For reusable architectural decision models, CoCoADvISE provides automated support for quality-attribute-based architectural decision making [31].In addition, Lopes Silva et al. introduce a tool to support the architecture design phase by recommending architectural styles given the quality attributes [29].Finally, almost 14% of the approaches under study document both the QAs in ADD documentations while at the same time they describe how QAs influence the reasoning activity on ADDs.For instance, the STREAM-ADD approach [16] suggests to explicitly document ADDs alongside NFRs, and to use QAs during architectural refinements in structural and technology decisions.In this approach, NFRs are used with soft-goals to evaluate the set of alternatives using architectural styles and the rationale for them.

Relationships between ADDs and QAs
Regarding explicit and implicit relationships (or links) between ADDs and QAs, we explored the types of links between decisions and quality attributes.These QAs map to high-quality requirements [10] as an important quality concern a system demands.This is important for the software architecture documentation because it helps software maintainers to identify the trace links between both artifacts when decisions change, and to reevaluate the decisions if QAs are modified.From our results we identified the following types of links between both artifacts: "not explicit", "explicit", and "supported but not explicit".For the sake of clarity, papers that categorize the relationship between ADDs and QAs as "not explicit" means that there is no direct and clear link described in a meta-model or via examples.Those works that we describe as containing "explicit" links between ADDs and QAs suggest an approach or method where QAs are used to support architecture decision making and in most cases such direct links are represented in a meta-model as well.Finally, works containing "supported but not explicit" relationship provide support for describing and evaluating the links between QAs and ADDs in an implicit way.Not all the research works selected provide explicit trace links between design decisions and quality attributes as these relationships are somehow hidden.
In approaches categorized as "not explicit" [21,35] the authors highlight that there is a relationship between ADDs and QAs but those links are not clearly described or represented.Such links are defined and used by tools like Software Architecture Warehouse (SAW), which handles collaborative decisions with a voting support facility [35].Nevertheless, the authors did not explicitly document these trace links as the focus of their approach is different, but this does not mean the trace links do not exist in models supporting tools like SAW.
From the approaches that model and define explicit ADD-QA links, we can highlight the one introduced by [15].In this work, the authors describe explicit connections in a meta-model linking decisions to design artifacts and quality attribute requirements, which helps support an automatic synthesis method of candidate architecture solutions.The decision-centric architecture design process uses the QAs to identify the issues that will lead to the issue solutions.These issue solutions will be used to synthesize the solution architecture from the decisions captured with other software engineering artifacts and the relationships among them.Other approaches [22,52,54] provide different but similar ways to relate several types of ADDs with QAs and they document such relationships explicitly with the use of meta-models.
Another way to relate ADDs with QAs is the use of architectural tacticssee for instance, [7,34] and [24] -or by combining requirements goal models with component diagrams [41].Finally, other works like CoCoADvISE [31] introduce a reusable architectural decision meta-model which integrates design solutions (i.e.ADDs) with decision drivers (i.e.QAs) and is subsequently used as a basis for generating questionnaires to support architectural decision making.This approach offers some degree of automation for representing the design decisions and QAs of interest, which are evaluated positively or negatively.More recent works like [14] describe explicit support of quality attributes to understand the ripple effect in decision making, but they do not provide explicit relationships between the quality attributes like in [38] or [39].

Evaluation of decisions and qualities
The final category suggests the following values to classify how QAs and ADDs are evaluated: "not supported", "partially used", and "fully used for evaluation" Fig. 4: Appearance of QAs in decision making, documentation, or both (left) and type of relationships between ADDs and QAs (right) (see Table 2 of the Appendix for the detailed results).The first value (i.e."not supported") refers to works where we could not find an evaluation of ADDs using QAs or the other way round.The second, "partially used", refers to those works where the evaluation is done only in one direction (e.g.QAs are used to evaluate ADDs) or the process is not described in enough detail.Finally, the last category, "fully used for evaluation", refers to those research works where the authors describe a method, approach, or tool supporting the evaluation and selection of architecture design decisions using quality attributes in both directions (compared to only one in the previous category).Fig. 5: Evaluation of ADDs using QAs and of QAs using ADDs Design decisions are used to select which QAs are more suitable to improve the quality of the architecture.Alternatively, decisions can be evaluated based on qualities, and on the pros and cons of each decision.In the first case, we decide on the QAs that must be present in a design.In the second case, the pros and cons are used to select the best or the optimal design decisions (e.g. a higher level of security is needed, so a requirement decision about security needs to be improved and hence, the necessary mechanisms to achieve the desired quality must be added).
For instance, [52] do not indicate how they evaluate decisions using quality attributes and the other way around.In most of these approaches, decisions and QAs are captured, but the evaluation is not that explicit or explained.Some other research works do not support or indicate any of the aforementioned types of evaluations using ADDs and QAs (e.g.[4,35,36,52,55]).
If we focus on those works rated as "fully used for evaluation", the approach described by [22] uses the notion of forces.The decision forces make the rela-tionships between design decisions and the factors that influence the decision makers explicit.Among these forces, we can find the quality attributes that are used to select the best decision alternatives in a competing manner.The authors use a table to connect these forces to other architecture views (e.g.view technology) and provide the explicit trace links between the forces and decisions, which are evaluated using a scale (i.e.?, -, +, ++).We can use the forces to rank the QAs during the selection of different technologies or use the decisions to select the QAs that better suit different business goals.
Other works suggest a multi-attribute decision making approach to make decision trade-offs, evaluate the most suitable QAs implemented in the system [32], and estimate the impact of a decision in software quality [5].The work presented by [11] highlights the role of design patterns, and their impact on good architecture design decisions and quality attributes is evaluated based on the selection of patterns.Most of the approaches categorized as "fully used for evaluation" put more emphasis on how decisions are selected based on a set of qualities that must be satisfied in a competing manner, while the rationale in the opposite direction (i.e.make decisions to select or reason on the best qualities that must be supported) is rarely seen, even if the trace links between both types of artifacts are explicit.In [33] the authors provide a catalogue of decisions for designing industrial IoT systems as they can be fully used for evaluation of different qualities.

QA-related Challenges Addressed by of ADD Tools and Methods (RQ2)
In this section, we discuss the QA-related challenges that are addressed in existing tools and methods for architecture decision making and documentation.In particular, after analyzing the selected articles of the literature survey, we identified the following main challenges that are addressed in several publications which are detailed in Table 3 of the Appendix.

QA uncertainty
Uncertainty is caused, among others, by imprecise and incomplete knowledge, or requirements that make decision making and QA trade-offs difficult.Uncertainty describes a situation in which QAs are not exactly known and can not be precisely quantified, therefore, they are evaluated using a verbal scale.The task of resolving this kind of uncertainty during decision making is on the one hand complex and on the other hand, time-consuming.Even though the inherent uncertainty of QAs is explicit in a few cases, the resolution of this kind of uncertainty is not supported by the majority of the existing tools and methods.For instance, Zdun expresses uncertainty of quality goals using approximated scores (++ for very positive influence, + for positive influence, o for no influence, -for a negative influence, and --for a very negative influence expected) [53].Similarly, in our previous work [31] we described such relationships at a meta-model level for reusable architectural decision models in order to express a positive or negative impact of design solutions on QAs.Bode and Riebisch [11] use a point system to express the impact of architectural styles on quality properties with a scale going from -2 (strong negative) to 2 (strong positive), very similar to the verbal scale in [53].The work of [39] provides a method to estimate uncertain parameters which are unknown during architecture design.We observe that a large majority of the 10 approaches under analysis which discuss uncertainty of QAs use either a verbal scale or a point system to express this uncertainty.Nonetheless, except for one paper (i.e.[37]) the works under study do not tackle resolving uncertainty during decision making.However, previous works have studied decision making in the architectural solution space under uncertainty by considering probability distributions of the parameters of an architecture model [28] or fuzzy values for the alternative solution properties [18].

QA interdependencies
The interactions between QAs are widely recognized in software architecture evaluation and a QA may have a positive or negative impact on other QAs [17].For instance, in a particular system context, system security might come at the cost of availability, whereas in other system contexts availability is a subgoal of security and thus the two are associated positively [20].In [31], QA interdependencies are described in a meta-model level, so a QA can be in synergy with or in contradiction with another QA.In another approach, the authors define subsets of QAs in the form of utility trees by distinguishing between quality attributes and quality factors [6] to establish links between the QAs and design decisions, while [11] also introduces a hierarchy of subcharacteristics and properties related to the quality goal "Evolvability" of software projects.Lopes Silva et al. express potential relationships between QAs, which can be conflictive (-), cooperative (+), or neutral (0) [29].
Another factor that may complicate QA evaluation is the resolution of priorities among the different qualities, as stated in [43].For instance, preferences on QAs are given by comparing them pairwise and giving quantitative weights in the quality-driven approach for decision making by [3].Also, [12] use prioritization of QAs using an ordinal scale to rank the design solutions with respect to the set of QAs of interest, and they define four types of criteria (i.e.ontocriteria, anticriteria, diacriteria, and pericriteria) based on classes of architectural design decisions.In the Quark approach described in [5], prioritization of QAs can be used by a decision inference system to provide a prioritized list of ADDs.Fig. 6 shows the percentage of the approaches under study which support uncertainty and QA interdependencies.In our analysis we observed that 21 of the 36 approaches under study support quality attribute evaluation trade-offs during decision making.Fig. 7 shows the percentage of the selected approaches supporting QA trade-offs as well as the different types of trade-off automation.While the largest number of the approaches support manual QA tradeoffs (10 out of 21), only five approaches described an automatic process.In other works, the level of automation is not indicated.The ADD+ method [32] supports automatic trade-offs between conflicting and incomplete quality scenarios and various stakeholders' concerns and preferences as a multi-attribute decision problem, in which the attributes represent the degree of satisfaction of conflicting quality scenarios.ArchDesigner considers making trade-offs a multi-attribute decision making problem and leverages the Analytic Hierarchy Process (AHP) to calculate value scores for design alternatives given their relative impact on QAs and relative stakeholders' preferences [3], while in [37] the authors perform multi-criteria decision analysis.The tool introduced by de Boer et al. can be used for automated trade-off analysis to rank the alternative solutions according to certain QA priorities [12].
Currently, many approaches integrate trade-offs in the architecture decision making process where quality scenarios are used to assist in making trade-offs, such as described in [6].In addition, the ArchPad (RADM) method of [55] provides reusable pattern-based decision models, which entail the information for resolving requirements at different levels [54].Also, [15] propose a method for assisting in the selection of and reasoning on architectural solutions, so architects can summarize the advantages and disadvantages of each architecture solution in order to make trade-offs.The STREAM-ADD approach supports manual trade-off analysis of alternatives considering the fulfillment and the priorities of softgoals and NFRs [16].In other cases, like in the Software Architecture Warehouse (SAW tool) approach, trade-offs are made in a collaborative context [35] and stakeholders discuss the advantages and disadvantages of each design alternative until they reach a consensus.More sophisticated approaches use constraint satisfaction algorithms [5] and expert systems [29] to reason about QA trade-offs.Finally, the authors in [34] suggest two negotiation strategies using agents to support trade-off analysis.Fig. 8 summarizes the different methods used for making QA trade-offs based on the analysis of the 21 aforementioned approaches.The demonstration and evaluation of the approaches in real-life-scenarios and empirical studies the authors investigated validate many of the proposals, and they guide practitioners who need to integrate QAs in architecture decision making processes.The rationale for including the design space size and scope in this research relies on the importance for each industrial and academic case study about the decision capturing effort, dependencies between decisions and other software artifacts, type and nature of the decisions, number of alternatives considered, and the QAs used.Fig. 9 reports the size of several industry cases in terms of scenarios and design issues, architectural patterns and tactics used as well as the size of the decisions network.More specifically, a portion of the proposals uses a small number of decisions (<10), except in the case of [54,55] where a big design space consisting of 300 reusable SOA-related decisions has been created.While the approach has been used in industrial projects, the validation of the corresponding QAs was not the focus of the validation in reusable ADDs.19 proposals report a case study, six a motivating example and five of them a real project or system, such as the ones described in [43,45,54].In other evaluation studies, industrial experts have participated in case studies (e.g. in [11,31]).
In addition, few approaches (such as [21]) do not perform any kind of evaluation or they just document types of decisions ranging from those based on concrete scenarios to structural, behavioral, and technology decisions [16].Also, the Software Architecture Warehouse [35] seems to be validated in various design workshops with students but this is poorly reported.The majority of the case studies report at least three QAs while only six publications do not report which QAs were used.The frequencies of the QAs used in the papers under study are shown in Table 2, as an indicator of the most relevant QAs used in decision making activities.Regarding the types of studies, we can conclude that the majority of them have introduced a technique or a tool in their case studies and examples.While 42% report a case study, 25% of them are only examples but in the context of industrial projects.Nevertheless, in the majority of the studies, the design space size and scope, as well as the number of QAs that are systematically studied are rather low.Fig. 10: Distribution of evaluation methods reported in analyzed ADD tools and methods

Discussion
In this literature review, we have studied the research works on QAs in ADD methods and tools over the past 17 years.Most of the papers examined investigate the role of QAs in architecture decision making approaches and how design decisions are used to select the most optimal QAs during architecture evaluation and design.The results also revealed that in the majority of cases, QAs are documented or used in a decision making process.Other findings show that a large majority of studies describe fully explicit relationships between QAs and ADDs, which highlights the importance of such trace links for documentation and knowledge capturing.With respect to the evaluation of QAs and ADDs, about one fifth of the studies do not focus on or describe such an evaluation.This fact leads us to assume the difficulty of carrying out such an evaluation activity.The QA-related challenges used in existing ADD methods show that 58% of the research works deal with QA trade-offs, which is close to the number of works using QAs to evaluate the design decisions.However, only 28% and 39% of the research works address QA uncertainty and QA interdependencies respectively, an indication that these research areas are still immature.There is only one publication covering all three QA-related challenges (i.e.[38]).Another interesting aspect is the automation level of approaches using QAs: the results reported in half of the approaches show that 10 of them are "manual" and six use semi-automatic processes to calculate the priority of the QAs for decision making and trade-off evaluation.The findings have revealed that there is a mature set of baseline approaches for supporting QAs in AK methods and tools.However, it seems to be difficult to achieve more sophisticated automation, e.g. in QA-based architectural decision making.

Implications for researchers and practitioners
The size and scope of the industrial use cases analyzed show an early adoption of methods and tools that support the interplay between ADDs and QAs, but the maturity of the approaches and their adoption are in an early adoption phase.Furthermore, the traditional QA trade-off evaluation in software architecture can benefit from approaches relating the decisions with the qualities of the system, so that software architects can better understand the underpinning reasons for selecting the best quality properties of a system as well as how the selection of a quality property influences the decisions for technology selection, and use these results to train novice architects.In addition, industry practitioners provided a certain amount of evidence for successful use of ADD entangled with QAs in specific domains or application cases (e.g.SOA decisions, control systems, smart systems, financial IT systems), but the size of the decision set tends to be small, maybe because of the cost for capturing them.
At the same time, the size of the studies show that the techniques analyzed in this literature review can be applied with relatively low effort compared to the size of real software projects and no significant investment other than learning and simplistic tool development is needed to start out.Our analysis has also revealed that a large majority of the studies (85%) report evaluation of the presented approaches.Even though some of the studies present industryrelated cases in the majority of the studies scientifically less rigorous evaluation approaches such as "non-industrial case studies", "example applications", and "motivating examples" have been used.In addition, the size of the decision sets in some of these industrial cases is sometimes low, which does not necessarily diminish the relevance of the evaluation approach adopted, as such decisions may belong to a subset of the architecture or system under study.Therefore, more real examples where significant sets of decisions are made, based on stringent quality requirements, are needed.
Finally, one interesting finding for academics and professional software engineers is the frequencies of appearance of the QAs found in AKM approaches (see Table 2).Of about 40 QAs identified, only seven show frequencies of six or more.In particular, performance, security, cost, reliability, usability, maintainability, and modifiability exhibit the highest frequencies of appearance.These results might indicate that a reduced set of QAs seem to be more important or useful than others even for different domains and applications, which are at least in the scope of publications analyzed.The QAs with a higher frequency are more likely to be considered in relation to the ADDs than the QAs with a lower frequency, therefore, the most frequent QAs should be considered first by software engineers when designing software-intensive systems.

Limitations
The limitations of our study are the following: In many cases, we needed to interpret the implicit use of QAs in the ADD related approaches, as some of the results reflect our understanding given the missing or blurry specification.Therefore, the extraction and evaluation results of the related information may have led to inaccuracies.As many of the tools we discussed are not available online, we had to base our findings only on the reported results in the publications under study.There might also be some bias of the authors during the selection and classification of the candidate papers, as they have worked for years in the field and are active researchers in the area.We tried to mitigate this risk by reading and interpreting the primary studies independently in several iterations, and by double-checking each other.Also, the fact that the authors are based in different institutions and have different backgrounds helped to mitigate this risk.Finally, our systematic review may have inevitably missed some relevant tools and methods that were excluded during the search process and the implementation of our inclusion/exclusion criteria.We tried to mitigate that risk by querying and aggregating results from four databases (i.e.ACM DL, IEEEXplore, DBLP, Springer), by using general search terms (e.g."software architecture") and alternative terms (e.g."quality attributes" and "non-functional requirements"), and by using forward and backward snowballing.

Conclusions and future trends
This paper reports a systematic literature review on QAs in architecture decision making and documentation approaches.While the findings of this literature review identify promising research and practice areas, there are still a number of factors that challenge a broader adoption of ADD methods using QAs.While the knowledge capturing problem and the relationships to other software artifacts like requirements seems solved, the majority of the approaches examined cannot provide more automatic procedures to evaluate QAs using decisions and make decisions for the selection of the required QAs.Only some prioritization mechanisms for QAs and alternative decisions as well as the making of trade-offs can be partially automated.In addition, as QA interdependencies have been poorly discussed by ADD approaches, we provided additional insight motivating the importance of bi-directional links between ADDs and QAs to show that decisions can be motivated by changes in the quality attributes or the selection and evaluation of a particular quality is driven by a design decision.Thus, we also identified the relevant works supporting the explicit and documented trace links between both artifacts and which of these approaches use the trace links to evaluate QAs using decisions and the other way around, but also to understand how a change in a QA may affect other related QAs.
Another challenging area for knowledge sharing and collaborative approaches where decisions are not made by one single person is Group Decision Making (GDM), as well as how GDM approaches can be used in agile development contexts.Moreover, making decisions under uncertainty is another challenging area and green field to train software architects in making decisions with incomplete information.Finally, future research should provide better support and tools for attribute evaluation at decision making and for making architectural decisions sustainable over time.

Fig. 1 :
Fig. 1: Conceptual model showing the main concepts and relationships between design decisions and quality attributes.Decision networks can adopt different topologies like DAG (Directed Acyclic Graphs) or QOC (Question-Option-Criteria) models.

Fig. 8 :
Fig. 8: Methods for making QA trade-offs and number of approaches employing these methods

Fig. 9 :
Fig. 9: Number of approaches with respect to the size and scope of reported industry cases studies

Table 2 :
Frequency of appearance of QAs