Testing Context-Aware Software Systems in the Automotive Domain: A Multi Vocal Literature Review Protocol and Dataset

Testing Context-Aware Software Systems in the Automotive Domain: A Multi Vocal Literature Review Protocol and Dataset

A Multi-Vocal Literature Review (MVLR) is a form of a systematic review that includes grey literature in addition to peer-review literature (Garousi, Felderer, and Mäntylä 2019). The decision to justify an MVLR is drawn from the results of recent literature reviews, in particular the recent results from (Matalonga et al. 2022)  where it is shown that there is little evidence in the white literature on the approaches to testing non-academic CASS Systems.

Previous academic works (including our Quasi-Systematic Literature Reviews and Rapid Reviews) operate under the following assumptions and observations:

• Assumption 1. CASS are widespread and being deployed for commercial or industrial use.
• Observation 1. The software engineering and software testing communities have had time to adopt (or develop new) techniques to deal with the context and effects of testing software systems.
• Observation 2. Academics have been able to work with software organizations to transfer or study the approaches used to test CASS, yet the published case studies we are aware of describe a partial picture of the overall adoption and approach of the problem.

In spite of these assumptions and the availability of systematic literature review studies, there is little evidence of how software organizations are testing CASS.

Research Goals

Aim: To uncover evidence on how the automotive industry reports their working with the dynamic testing process regarding CASS.

We use the term industry to broaden our scope to include stakeholders with an interest in the quality of CASS like NGOs, standard-setting organizations and regulation-setting organizations who can influence how software must be treated in different domains.

The following research questions convey the general interest of our enquiries. These are driven by our previous research and expectations on the sources.

• RQ1 Are there sources to support the understanding and indicate directions to deal with the problem of testing CASS?
• RQ2 What are the challenges of using these dynamic testing process solutions?
• RQ3 How are the dynamic testing processes that deal with the context of CASS described in the sources?

Dataset 

This dataset contains the following artifacts:

1. Testing CASS MVLR Protocol.pdf: protocol containing the methodological details of performing the MVLR
2. Sources Identification, Selection and Data Extraction.xlsx: spreadsheet used to record and control discovered and selected sources
3. Extraction_Documents.zip: compressed file containing all data collection forms filled with data extracted from the
4. MVLR_Analysis-Codebook.xlsx: listing of codes emerging from the collected data