Understanding the Landscape of Software Modelling Assistants: A Systematic Mapping - Raw data and Protocol

This repository contains the protocol and raw data from the paper "Modelling Assistance in Model Driven Software Engineering Tools: A Systematic Mapping Study and a State of the Practice." This paper is currently under review in the Information and Software Technology Journal.

How to use: 

Inside this repository, you will find five (5) files: DataExtraction.xlsx; MENTI-RESULTS-1.png; MENTI-RESULTS-2.png; StudySelection.xlsx; and Tool-Market-review.xlsx. We describe each file in the following paragraphs. 

DataExtraction.xlsx. MS Excel file containing the 44 proposals (a.k.a. primary studies) included as a result of our systematic mapping study. This file has one tab named "PrimaryStudies", comprising the following columns: 

• Code: Contains the unique code we use for each proposal. We generate such code following the following template: First and second author last name + two first last name letters of the remaining authors (if present) + year.
• Ref: Contains the reference number for each proposal used in the paper.
• Title: Contains the proposal's original title.
• Status: Always "included"
• RQ1: Kind of approach. Contains the clusters to answer RQ1: Which strategies do existing proposals follow to assist modelling in MDSE tools?
• RQ1: Specific Word. Contains the specific words authors use for identifiying their proposal.
• RQ1: Other? Wich? Contains additional keywords authors use for identifiying their proposal.
• RQ2: Cluster Limitation. Contains the limitation clusters for answering RQ2: What goals (G) and limitations (L) do the proposals have?
• RQ2: Limitations (keywords). Contains the keywords authors use to describe their proposals' limitations.
• RQ2: Limitation(s). Contains full descriptions where authors described their proposals' limitations.
• RQ2. Goal Cluster. Contains the goal clusters for answering RQ2: What goals (G) and limitations (L) do the proposals have?
• RQ2. Goal (keywords). Contains the keywords authors use to describe their proposals' goals.
• RQ2. Goal(s). Contains full descriptions where authors described their proposals' goals.
• RQ3. Empirically evaluated? Answers yes or no depending on if the proposal was evaluated.
• RQ3. Evaluation metric cluster. Contains the evaluation metric cluster for answering RQ3: Which evaluation metrics (M) and target users (U) do the proposals have?
• RQ3. Evaluated metric keywords. Contains the keywords authors use to describe their proposals' evaluation metrics.
• RQ3. Evaluated metrics. Contains additional information on evaluated metrics for each proposal.
• RQ3. They define a user? Answers yes or no depending on if the proposal has a target user.
• RQ3. Cluster by target user. Contains the target user clusters for answering RQ3: Which evaluation metrics (M) and target users (U) do the proposals have?
• R3. Target user keywords. Contains the keywords authors use to describe their proposals' target users.
• QA: is the proposal clear? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No
• QA: are the limitation(s) clear? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No
• QA: Are the goal(s) clear? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No
• QA: Are the publications tools downloadable? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No
• QA: Is there a clear case study or example illustrating the proposal? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No
• QA: Is the whole proposal empirically validated? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No
• QA: Are the users clearly described? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No
• QA: Are the results clearly explained? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No
• CORE/JCR: Objective quality assessment question containing CORE and JCR indexes.
• Citations: Number of citations.
• Quality index: Results of calculating the quality index based on the Quality Assessment table (See Table 1 in the paper)

MENTI-RESULTS-1.png Contains the results of consulting a set of proposed research questions with the potential target audience of our paper. By asking this question, we aim to answer which RQ target audience found more critical and exciting for dividing the MRQ? This file contains specifically the question:  "Which question do you find more important and interesting to solve via a systematic mapping?" 

MENTI-RESULTS-2.png Contains the results of consulting, which other research questions could be solved via a systematic mapping study. By asking this question, we aim to answer what other RQ would target audience propose to divide the MRQ? This file contains specifically the question: "What other questions would you like to answer via a systematic mapping?"

StudySelection.xlsx. MS Excel file containing 2613 records involved in title, abstract, and full text review for selecting primary studies. This file has two tabs: 

• Database-search-data. Contains the results of the screening title, abstract, and full text using the inclusion and exclusion criteria defined in our paper. Moreover, contains the traceability of each decision made for each record and who took that decision (i.e., reviewer). This tab contains the following columns:
  • Year. Contains the year of publication of each record (only included).
  • Strategy. Contains the snowballing strategy if it is forward or backward snowballing. If it is empty, the record comes from the database search.
  • Source code. Contains the study from it comes during the snowballing.
  • First reviewer. Contains the name of the first reviewer.
  • Second review. Contains the name of the second reviewer.
  • Code. Contains the unique code we use for each proposal. We generate such code following the following template: First and second author last name + two first last name letters of the remaining authors (if present) + year.
  • Title. Contains the proposal's original title.
  • Status. Contains the proposals status: included, excluded-after-tr, excluded, and duplicated
  • Include after title screening? Contains Yes if it is included and No if it is excluded.
  • Include after abstract screening? Contains Yes if it is included and No if it is excluded.
  • Include after full-text review? Contains Yes if it is included and No if it is excluded.
  • Include after the second reviewer's opinion? Contains Yes if it is included and No if it is excluded.
  • Include after discussion? Contains Yes if it is included and No if it is excluded.
  • Not fulfilled inclusion/exclusion criteria. Contains the not fulfilled inclusion or fulfilled exclusion criteria as follows:
    • (I1) Does the paper define a specific proposal for assisting users during modelling in MDSE tools? Compilation of proposals like literature reviews, systematic mappings, or systematic literature reviews does not fulfil I1.
    • (I2) Is the proposal designed to assist users during modelling in MDSE tools? We focus on proposals that assist users during modelling in MDSE tools, including—but not limited to—modelling, model tracing, model debugging, model repair, and model validation, among others. Proposals focused on assisting in developing MDSE tools do not fulfil I2.
    • (E1) The proposal's main contribution is not on assisting users during modelling in MDSE tools. If assisting users during modelling is not the main contribution, we exclude the proposal using E1—e.g., a proposal showing a new MDSE tool where they superficially mention assisting users during modelling.
    • (E2) The proposal is not related to software engineering.
    • (E3) The proposal is not written in English
    • (E4) The proposal is not a peer-reviewed publication
    • (E5) The proposal's full text is not available.
• CK. This tab contains the K statistic calculation based on the first and second review agreements.

Tool-Market-Review.xlsx. MS File containing the 17 MDSE tools from the Gartner Magic Quadrant reviewed for answering RQ4. What is the state of the practice of modelling assistance? This file contains two tabs: 

• Data. Contains the extracted data for answering RQ4. This tab contains the following columns:
  • MDSE tool. Contains the name of the tool we extract the data from. Moreover, include the GMQ classification inside parenthesis as follows: LE: Leaders; V: Visionaries; C: Challengers; NP: Niche Players.
  • Kewywords. Contains the name MDSE authors refers to their proposals in their tool documentation.
  • Link. Contains the link to the proposal documentation.
  • Quotes. Contains textual quotes extracted from tools documentation.
• GQM. Contains a picture of the Gartner Magic Quadrant for Enterprise Low-Code Application Platforms 2023.

Triangulation-v.xlsx. MS File containing the triangulation performed for each RQ cluster in the results. This file contains six tabs: five tabs containing the clusters and the decision after triangulation; and one tab for calculating the K-statistic between the first review and result after discussion on including the studies on the final clusters:

• RQ#-{Cluster_name}. These five tabs comprise the same structure and are named depending on the RQ and the cluster. For example, the first cluster about strategies that answer RQ1 represents a tab with the name RQ1-Strategies. Each tab contains the following columns: 
  • Cluster. Contains the cluster name. 
  • Cluster definition. Contains the cluster definition based on the authors' observation of the extracted data. 
  • Cluster keyword.  Contains the specific keyword for each proposal included in each cluster. 
  • Paper specific word. Contains the paper-specific word to associate with a specific keyword for each proposal in each cluster. 
  • Review decision. Contains the decision made during the first review of the clusters: can take two values "agree/disagree." 
  • Disagree reason. Contains the disagree reason in case the previous column was "disagree." 
  • Conflict resolution. Contains the strategy to solve the conflict in case it was a "disagree" decision after the first review. 
  • Solution. Contains the result after the second review and applying the conflict resolution. Can take two values "Accept" or "Discuss." Accept means both reviews agree on the change to solve the first "disagree" decision. Discuss means the disagreement persists, but there is an explanation for it. 
  • Link. Contains the link to the specific proposal. 
  • Code. Contains the code to reference the extracted data: a unique code we use for each proposal. We generate such code following the following template: First and second author last name + two first last name letters of the remaining authors (if present) + year.
• K-statistic. This tab contains the K statistic calculation based on the first and second review agreements during triangulation.

If you have any doubts, contact mosq@zhaw.ch / ruiz@zhaw.ch