A Systematic Mapping Review of Software Quality Measurement: Research Trends, Model, and Method

Software quality is a key for the success in the business of information and technology. Hence, before be marketed, it needs the software quality measurement to fulfill the user requirements. Some methods of the software quality analysis have been tested in a different perspective, and we have presented the software method in the point of view of users and experts. This study aims to map the method of software quality measurement in any models of quality. Using the method of Systematic Mapping Study, we did a searching and filtering of papers using the inclusion and exclusion criteria. 42 relevant papers have been obtained then. The result of the mapping showed that though the model of ISO SQuaRE has been widely used since the last five years and experienced the dynamics, the researchers in Indonesia still used ISO9126 until the end of 2016.The most commonly used method of the software quality measurement Method is the empirical method, and some researchers have done an AHP and Fuzzy approach in measuring the software quality.


INTRODUCTION
Software product quality is a critical factor in business [1]. It is determined by to what extent the product could meet the user needs, and do its performance, and how many defects in that software [2]. Hence, it needs to do the quality checking before the software products is sent to the market as the reparation after delivery is viewed very costly and can affect the company credibility [3]. Some researchers have studied the software product quality in a different perspective such as Boehm model [4], Dromey model [5], McCall model [6], ISO/IEC 9126 [7] and ISO/IEC SQuaRE [8]. Based on the model defined, the quality of the software product is measured regarding its ability to fulfill the aim of the developers and the needs of the users [9]. For this, human as the user becomes the factor that needs to be involved in the method of the software quality measurement. Adopting the term of Usability Evaluation Method, then the analysis of the software is divided into two: analytical method (measurement by the experts) and empirical method (measurement by users) [10][11][12]. The experts include the academicians and practitioners in software engineering, content expert, and technical expert. Meanwhile, the users here include end user, including the manager, employees, lecturers, students or customers using the software. This research aims to see the trend of the model and the method of the measurement of the software in the last 10-year period. The pattern of the model and the method are used as the references in the subsequent researchers. This research used the method of Systematic Mapping Study based upon the research questions. This approach could provide a description of the research area, identify the number of research, type of the research and the results available IJECE ISSN: 2088-8708 

RESULTS AND ANALYSIS 4.1. Definition of Research Question
The first phase of this research was to define the Research Question (RQ) to identify the scope of review. The question was made by following the PICO structure (Population, Intervention, Comparison, Result) [19]. Tabel 1 shows the PICO structure from the research questions and Table 2 presents the description of the literature.

Searching of Papers
The second phase of Systematic Mapping Process was to search the paper correlated with the Research Questions. The best way was by making the search string that is by structuring the words based upon PICO as presented in Table 1.The keywords for the search string have been taken from each aspect from the structure. The string used to search the paper is as follows: ("software" OR "website" OR "application software") AND ("software quality model" OR "ISO/IEC 250n"OR "ISO/IEC SQuaRE"OR "ISO/IEC9126" OR "Boehm" OR "McCall" OR "Dromey")

Screening of Papers and Keywording of Abstract
Keyword was applied to search the paper based on the title, abstract and content that correspond to the research database namely IEEExplore (ieeexplore.ieee.org), Scopus (www.scopus.com) and Science Direct (www.sciencedirect.com). From the result of the automatic paper searching in the research database, 308 papers have been obtained with the following details: 77 IEEExplore, 211 Scopus, and 20 Science direct. Once taking the results, we applied the selection criteria to filter the candidates. All papers obtained from the research database would be selected based on the inclusion and exclusion criteria that were used to limit the scope area obtained from Systematic Mapping Study. The elimination process was conducted to reselect or for the exclusion of content in the inclusion area. After searching and filtering the paper, 42 relevant papers were obtained. Table 3 shows the inclusion and exclusion criteria used in this research. To determine the distribution of model to measure the Software Quality.

RQ 2
What kind of method is performed most often for Software Quality?
To determine the distribution of method to measure the Software Quality. To answer the Research Question 1, Table 4 was made. This table shows the distribution of paper discussing the implementation of Software Quality Model in last decade. In this table, it can be found out that the model of ISO/IEC 9126 and ISO/IEC SQuaRE had a percentage similar to the application of the Software Quality (40.5%). Terminology for the characteristics of software quality had been discussed in the International Consensus in 1991. In the period of 2001-2004 ISO issued the standard series 9126 extended into four parts including model and metrics of the software quality [20]. However, the standard starts to be widely used in the measurement of the software quality for some years after the existence of its revision. In Table 4, it can be seen that the emergence of ISO/IEC 9126 is a model that has mostly been discussed by the researchers; four of which come from Indonesia [21][22][23][24]. This might be likely that in that year the researchers from Indonesia have started the research on the quality of the software to be published in the international journal. It is proven that 2 of the researchers were the same research team that is [21], [23] discussing the implementation of model ISO 9126 in the software of inventory asset, and Learning Management System. If the revision of the model ISO 9126were just implemented two years after its emergence, then the review of ISO SQuaRE in 2011 would have needed one year to make the research from several researchers appeared. This then shows that the use of the new standard has started to attract the attention of the researchers to apply it to the measurement of the software quality. Through the use of model ISO 9126 and ISO SQuaRE have experienced a dynamics since 2015 and 2016 the use of model ISO 9126 is only used by the researchers from Indonesia [25][26][27]. This shows that the model is quite popular in Indonesia.
In the trend of the model in the last decade, Boehm's Model and McCall's Model commonly were combined with some other models [6], [28][29][30], and gradually were no longer used after 2012. Meanwhile, Dromey's Model is not found in all papers regarding the implementation of the software quality. This is in line with the statement of Miguel stating that there is no discussion on how the model is used in practice but the theoretical model is used to build other more accurate models [14].  Figure 2 shows that the empirical model is mostly used in measuring the software quality (67%). The method includes survey [26], [48], [49], [59], [63], observation [21], [27], [31][32][33], [35], [37], [39], [42][43][44], [52], [53], [64] and usability testing [23], [65] involving the assessment of users. This achievement is followed by the analytical method at 17% (includes heuristics evaluation [34], [40], exploratory case study [50], and evaluation group [30], [45]). Some researchers measuring the software quality without user judgments, they are [29], [52], [61][62], [66] classified into other method. The empirical method is mostly used for being cheaper, close with the experience of the users, simple, and providing a new perspective, direct feedback from users and, verbal data easily collected and processed. The form of the model of the software quality measurement that is hierarchical and the method of measurement involving the elements of user and expert judgment has encouraged the researchers to use the approach of Analytical Hierarchy Process and fuzzy logics. Table 5 shows that the research approach used AHP at the measurement of the software quality has been conducted by [48], [34], and [44], but the research still used the crisp number in weighting the score of the assessment from the users; as a consequence, its weighting seemed to be less representative. They also added a number of sub-characteristics in the model of the measurement of the software quality such as compatibility, modularity, complexity, reusability, availability, customizability, or traceability.

RQ 2: What kind of method is performed most often for Software Quality?
Given the more complex issues to be settled, the use of fuzzy logics concept then is used to cope with any the lack of the use of crisp numbers. The approach of Fuzzy Multi-Criteria was revealed by Challa, he considering the weight of the fuzzy calculation based upon three perspectives of the users: developer, manager, and user [9]. Challa provided a conclusion from the measurement of the software quality with the labels of Very Good, Good, Average, Poor, Very Poor. Yuen [38] used the Fuzzy AHP to compare the different software to select the best software [37]. Yang [42] used the approach of Fuzzy Choquet Integral to measure the software quality based on the different perspective of users showing the influence between the criteria results. Ying Xing [30]used the method of Fuzzy Statistic Method to provide a quantitative method for the measurement of software quality in the digitalization system of antique ethical resources. The model was suitable for the system of three layers of the software quality of Walters and McCall. Chen [31] proposed a model of software quality to measure the quality of the software in the system of DVR (digital video recorder) during the phase of its development to minimize the gap in the assessment of the developers and end users and evaluators of the third party. The characteristics of the users giving the assessment of the software quality are various. Some papers do not mention the number of its evaluator, but the average number is not more than 30 respondents. The lack of the method approaches above is how to do optimization of the score weighting from the user perception to obtain the maximum results. This score weighting would be used for the ranking of the issue on the quality that is very urgent to be improved in the design phase.

CONCLUSION AND FUTURE WORK
Based on the research above, it can be concluded that the model of ISO SQuaREis a model of the measurement of software quality that is mostly used in the last five years. The use of ISO 9126 model is still applied in the measurement of the software quality by the researchers in Indonesia until the end of 2016. This indicates that the model is still quite relevant to be used. The model of Boehm and McCall gradually started to be left behind, and their application is mostly combined with another model since the last decade. The most widely used method of measurement of the software quality is the empirical method that is a measurement involving the users. The method approach using AHP and Fuzzy has been widely used but still needs to be studied further in this case regarding the optimization of the weighting of the user assessment and the ranking of the software quality problem that must be improved by the developers based on the user perception. Therefore, it needs to suggest the taxonomy model, framework process, determination of the highest weighting from the indicator of the ranking of the software quality, and to recommend the improvement of the design based on the result of the measurement.