Id,index,Title,Year,Publisher,RQ3_Contributions,Type_of_Venue,Venue,WB_BB,RQ5_comments,RQ6_Kind_of_Testing,RQ6_Comments,RQ7_Type_of_Case_studies,Location_of_Case_studies,RQ7_New,RQ7_New_Location,RQ7_New_Source,Language_of_Case_studies,RQ8_Name,RQ8_Language_of_Implementation,RQ8_Open_Source,Types,Techniques,Inputs,Metrics,RQ9_non_research_tools,RQ9_non_testing_or_REST,RQ9_cited,RQ8_Location_of_Implementation,RQ10_features,RQ10_Input,RQ10_Output,RQ10_Auth,RQ10_DBReset,RQ10_ExternalServices,RQ10_AutomatedInstrumentation,RQ10_Other,RQ11_addressed_challenges,Found_during_snowballing,After_1_March,Comments,,2ndCategory,RQ11_Coverage,RQ11_Compare,RQ11_Fault,RQ11_Abstract,RQ11_Summary,UseEMB,AuthoredByUs,RQ12_Summary,RQ12_Detailed
P_01,1,Automated black-box testing of nominal and error scenarios in RESTful APIs,2022,Wiley,NA: New automated approach and its extension,Journal,STVR,Black Box,Extract operation dependency graph,"robustness,system",,Industrial,Hosted on the Internet,Internet,Internet,,,RestTestGen,Java,Yes,"System testing, mutation testing",model-based,"OpenAPI, dic",,Swagger Schema Validator,,"Postman, SoapUI, vREST NG, HttpMaster, ApiFortress, RestAssured, APIFuzzer, Fuzz-lightyear,TnT-Fuzzer",https://github.com/SeUniVr/RestTestGen,Input:OpenAPI>Output:JUnit test case>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:,OpenAPI,JUnit test case,TRUE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,FALSE,FALSE,TRUE,"black-box automated test based on OpenAPI, using oracles",,FALSE,FALSE,Security testing,Adopt the metrics introduced by Martin-Lopez et al. [46] as a future extension>Extend the testing capability of RESTTESTGEN to try and assess the presence of security defects in the implementation of REST APIs>Extend RESTTESTGEN with oracles taking into account security related aspects as a future work
P_02,2,Automated Specification-Based Testing of REST APIs,2021,Web of Science,NA: New automated approach and its extension,Journal,Sensors,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,RESTApiTester,Java,Yes,,,,,,,"SoapUI, Moesif API Analytics, Tcases",https://github.com/RobertGyalai/RESTApiTester/,Input:OpenAPI>Output:Unspecified test case>Auth:No>DB_Reset:No>ExternalServices:No>Other:,OpenAPI,Unspecified test case,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,adding user interaction,Integrating user interactions,FALSE,FALSE,"Load testing,Tool support","Extend the number of specification languages that can be interpreted>Enable having an option to export the generated test cases in one or more testing frameworks, such as JUnit or Serenity>"
P_03,3,Resource and dependency based test case generation for RESTful Web services,2021,Springer,NA: New automated approach and its extension,Journal,EMSE,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,informed-search,,,,,Spring?,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",TRUE,No,No,,,,TRUE,TRUE,FALSE,"Detect dependencies, Propose sampling strategies and sampling methods",Handling resource dependencies,TRUE,TRUE,"Resource handling,Having more REST case-studies",Do byte-code instrumentation on the source code of the SUT to analyze and handle these further constraints>Conduct additional experiments with more case studies>Investigate novel solutions for doing a more intelligent analysis of resources in web services that are not fully following the REST guidelines>Involve database operations to directly manipulate resources in the RESTful APIs
P_04,4,A Discrete Dynamic Artificial Bee Colony with Hyper-Scout for RESTful web service API test suite generation,2021,Science Direct,NA: New automated approach and its extension,Journal,Applied Soft Computing,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,EvoMaster,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,Enhance ABC algorithm by integrating a hyper-scout unit,,TRUE,FALSE,,Investigate an enhanced algorithm that incorporates appropriate mechanisms to resolve the characteristic structure of the problems
P_05,5,A Black Box Tool for Robustness Testing of REST Services,2021,IEEE,NA: New automated approach and its extension,Journal,IEEE Access,Black Box,,"robustness,system",,Industrial,Hosted on the Internet,Internet,Internet,,,bBOXRT,Java,Yes,,,,,SpotBugs,,"RestAssured,SpotBugs",https://git.dei.uc.pt/cnl/bBOXRT/,Input:OpenAPI>Output:Unspecified test case>Auth:No>DB_Reset:No>ExternalServices:No>Other:Command Line Interface,OpenAPI,Unspecified test case,FALSE,FALSE,FALSE,FALSE,Command Line Interface,FALSE,No,No,https://eden.dei.uc.pt/~cnl/papers/2020-access.zip,,,FALSE,FALSE,FALSE,Robustness test based on OpenAPI,,FALSE,FALSE,"Tool support,Classifying test results",Study the applicability of machine learning algorithms to handle large diversity of responses>Improve upon its extensibility in order to ease the process of adding new functionalities>Add support for additional payload types
P_06,6,Automated Black- and White-Box Testing of RESTful APIs With EvoMaster,2021,IEEE,NA: New automated approach and its extension,Journal,IEEE Software,Both,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,,,Postman,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",FALSE,No,No,,,,FALSE,FALSE,FALSE,Evaluate EvoMaster,,TRUE,TRUE,"Handling external services,Classifying test results ",handle external services>Classify the found faults
P_07,7,Black-Box and White-Box Test Case Generation for RESTful APIs: Enemies or Allies?,2021,IEEE,NA: New automated approach and its extension,Conference,ISSRE,Hybrid,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,"RESTest,EvoMaster",,No,,,,,"Postman,RestAssured",,"RestAssured,SpotBugs,Postman",,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,TRUE,TRUE,TRUE,Integrate black-box and white-box approaches,,FALSE,TRUE,"Having more REST case-studies,White-box heuristics","Investigate whether the developers are willing to do
some manual work to improve the testing process>Higher degree of automation>Inference of realistic test inputs>Advanced white-box heuristics>Test oracles"
P_08,8,Deep Learning-Based Prediction of Test Input Validity for RESTful APIs,2021,IEEE,NA: New automated approach and its extension,Workshop,DeepTest,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,DeepLearningBasedTool,Python,Yes,,,,,,keras,,https://anonymous.4open.science/r/8954c607-8d6c-4348-a23c-d57c920cdc22/README.md,Input:API Call>Output:Validity of API call>Auth:No>DB_Reset:No>ExternalServices:No>Other:,API Call,Validity of API call,FALSE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,FALSE,FALSE,FALSE,Use deep-learning to predict validity of API requests,,FALSE,FALSE,Having more REST case-studies,Perform a more thorough evaluation including different datasets and more APIs>Explore the use of different machine learning techniques for the automated inference of inter-parameter dependencies in RESTful APIs
P_09,9,Empirical Comparison of Black-box Test Case Generation Tools for RESTful APIs,2021,IEEE,EA: Empirical study on various automated approaches,Conference,SCAM,Black Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,"RESTest,Restler,bBOXRT,RestTestGen",,No,,,,,,,"RestAssured,APIFuzzer,SoapUI,Postman,Fuzz-lightyear,TnT-Fuzzer",,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,TRUE,FALSE,FALSE,,,FALSE,FALSE,"Tool support,Classifying test results","Evolve the comparison with new testing tools, along with the updated versions of the already considered tools>Investigate how the adopted specification-based coverage correlates with code coverage>Extend the experimental framework to measure defect detection capabilities"
P_10,10,Improving Test Case Generation for REST APIs Through Hierarchical Clustering,2021,IEEE,NA: New automated approach and its extension,Conference,ASE,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,EvoMaster,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,TRUE,TRUE,TRUE,Consider sequences of API requests using AHC,Handling resource dependencies,TRUE,FALSE,,"Conduct a larger empirical evaluation>Investigate more learning algorithms within the hierarchical clustering category>Investigate other categories of machine learning methods alternative to hierarchical clustering, such as Bayesian Network>Investigate alternative, more adaptive mechanisms to decide whether the linkage tree model needs to be retrained or not>Implement and apply linkage learning to unit-test case generation as well"
P_11,11,RESTest: automated black-box testing of RESTful web APIs,2021,ACM,NA: New automated approach and its extension,Conference,ISSTA,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,RESTest,Java,Yes,,,,,"Postman,RestAssured",IDLReasoner,"Postman, RestAssured, ReadyAPI, API Fortress",https://github.com/isa-group/RESTest,"Input:OpenAPI>Output:JUnit test case>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:JAR file, Modification of Operations Under Test, Weights for parameters",OpenAPI,JUnit test case,TRUE,FALSE,FALSE,FALSE,"JAR file, Modification of Operations Under Test, Weights for parameters",FALSE,No,No,,,,FALSE,FALSE,FALSE,Support different techniques such as fuzzing and constraint-based techniques,,FALSE,FALSE,"Tool support,Non-functional testing,Metamorphic testing",Integrate search-based techniques for the generation of test cases>Automate metamorphic testing of RESTful APIs based on existing metamorphic relation patterns>Support non-functional testing of RESTful APIs by leveraging existing specifications such as SLA4OAI
P_12,12,"A simple, lightweight framework for testing RESTful services with TTCN-3",2020,IEEE,NA: New automated approach and its extension,Conference,QRS,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,,,No,formal verification,,,,,RJ-Plugin,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,,,FALSE,FALSE,Tool support,Support the automatic generation of the TTCN-3 type system for OpenAPI v2/v3 specifications and adding XML support
P_13,13,AI-Driven Web API Testing,2020,IEEE,PI: Proposal or idea,Conference,ICSE,Black Box,,system,,No Case-studies,No Case-studies,No Case Studies,No Case Studies,,,,,No,,,,,,,"RestAssured,Postman",,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,FALSE,FALSE,FALSE,,,FALSE,FALSE,,
P_14,14,Checking Security Properties of Cloud Service REST APIs,2020,IEEE,NA: New automated approach and its extension,Conference,ICST,Black Box,,security,,Industrial,Hosted on the Internet,Internet,Internet,,,Restler,"Python, F\#",Yes,,,,,,,"TnT-Fuzzer,APIFuzzer, Burp Suite, Sulley, BooFuzz, or the commercial AppSpider and Qualys’s WAS",https://github.com/microsoft/restler-fuzzer,Input:OpenAPI>Output:Unspecified test case>Auth:Yes>DB_Reset:No>ExternalServices:>Other:,OpenAPI,Unspecified test case,TRUE,FALSE,FALSE,FALSE,,FALSE,No,No,maybe not,,,FALSE,FALSE,FALSE,Add active property-checkers to a stateful fuzzer (security rules),Oracle problem,FALSE,FALSE,Having more REST case-studies,Explore general symbolic-execution-based approaches>Fuzz more services through their REST APIs and check more properties to detect different kinds of bugs and security vulnerabilities>
P_15,15,Increasing Reliability of Programming Interfaces Based on Fuzz Testing,2020,IEEE,NA: New automated approach and its extension,Conference,DESSERT,Black Box,,system,,,,Local Artificial,Local,Artificial,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,,,FALSE,FALSE,,
P_16,16,QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs,2020,IEEE,NA: New automated approach and its extension,Conference,ICST,Black Box,,system,,Mixed (open source and industrial),Hosted locally,Local (Open-Source and Closed-Source),Local,Open-source and Closed-source,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,FALSE,FALSE,TRUE,"Use automatic property-based test produced from OpenAPI, Automatically expose misalignment of specification and implementation",,FALSE,FALSE,Having more REST case-studies,"To be applied to a larger set of SUTs>Developer friendly>Shrinking>Mutate the specification>Iterate over parameters>Before going stateful, make sure API handles input>Stateful shrinking might not be accurate>Augment with manually created model"
P_17,17,RESTTESTGEN: Automated Black-Box Testing of RESTful APIs,2020,IEEE,NA: New automated approach and its extension,Conference,ICST,Black Box,,"robustness,system",,Industrial,Hosted on the Internet,Internet,Internet,,,RestTestGen,Java,Yes,,,,,Swagger Schema Validator,,Swagger Schema Validator,https://github.com/SeUniVr/RestTestGen,Input:OpenAPI>Output:JUnit test case>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:,OpenAPI,JUnit test case,TRUE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,FALSE,FALSE,FALSE,"black-box automated test based on OpenAPI, using oracles",,FALSE,FALSE,"Having more REST case-studies,Security testing,Tool support ","Extend the testing capability of RESTTESTGEN to try and assess the presence of security defects in the implementation of REST APIs>Extend our implementation to  support authentication, to perform a wider empirical validation, possibly including more complex and critical REST APIs, such as those related to the FinTech domain"
P_18,18,Intelligent REST API data fuzzing,2020,ACM,NA: New automated approach and its extension,Conference,ESEC/FSE,Black Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,Restler,"Python, F\#",Yes,,,,,,,"TnT-Fuzzer,APIFuzzer, RestAssured, Postman, SoapUI, and others [1, 4–6, 30], Peach [28] and SPIKE [32], [12, 13, 16, 17, 20, 21, 29, 33, 36, 37]",,Input:OpenAPI>Output:Unspecified test case>Auth:Yes>DB_Reset:No>ExternalServices:>Other:,OpenAPI,Unspecified test case,TRUE,FALSE,FALSE,FALSE,,FALSE,No,No,black-box data fuzzing rest api payloads to detect bugs in cloud services,,,FALSE,FALSE,FALSE,,,FALSE,FALSE,,Explore how to extend our implementation to check for mass assignment vulnerabilities
P_19,19,Semantic-oriented Automatic Generation Method of REST Interface Test Cases,2020,ACM,NA: New automated approach and its extension,Conference,ISCSIC,Black Box,,system,,No Case-studies,,No Case Studies,No Case Studies,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,maybe not,,,FALSE,FALSE,FALSE,Generate semantic-supported REST interface test cases,,FALSE,FALSE,,
P_20,20,Model-Driven Integration Testing of Hypermedia Systems,2019,Web of Science,NA: New automated approach and its extension,Journal,JWE,Black Box,,"integration,system",,Sample,,Local Artificial,Local,Artificial,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,FALSE,FALSE,FALSE,"Hypermedia testing, especially with different user roles and error cases",,FALSE,FALSE,,"Address the practicality of automating server updates for hypermedia client testing>Generate different types of negative testing, such as applying not allowed methods, trying to access a (sub-)resource after  deleting it or forcing the HTTP client to send wrong resource representations> Extend the model crawler to extract more possible test cases to achieve larger test coverage>Discuss more about how the application should behave in case of invalid requests>Address the client-side hypermedia testing"
P_21,21,RESTler: Stateful REST API Fuzzing,2019,IEEE,NA: New automated approach and its extension,Conference,ICSE,Black Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,Restler,"Python, F\#",Yes,,,,,,,"APIFuzzer, AppSpider, Qualys WAS, BooFuzz, TnT-Fuzzer, Burp, Sulley",https://github.com/microsoft/restler-fuzzer,Input:OpenAPI>Output:Unspecified test case>Auth:Yes>DB_Reset:No>ExternalServices:>Other:,OpenAPI,Unspecified test case,TRUE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,FALSE,FALSE,TRUE,"Infer dependencies among requests, Analyze dynamic feedback from prior executions",Handling resource dependencies,FALSE,FALSE,"Having more REST case-studies,Security testing",Fuzz more services through their REST APIs and check more properties to detect different kinds of bugs and security vulnerabilities
P_22,22,Generating Mock Skeletons for Lightweight Web-Service Testing,2019,IEEE,NP: New analysis approach and potential solution,Conference,APSEC,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,,,No,,,,,,,"Parasoft [10], CA [11], Wiremock [12], Hoverfly [13]",,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,Use symbolic machine learning to synthesise HTTP services’ mock skeletons from network traffic recordings,Mocking,FALSE,FALSE,,Conduct empirical study with end-users
P_23,23,RESTful API Automated Test Case Generation with EvoMaster,2019,ACM,NA: New automated approach and its extension,Journal,TOSEM,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,RestAssured,,RestAssured,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",TRUE,No,No,,,,TRUE,TRUE,TRUE,,,TRUE,TRUE,"Database handling,Tool support","Design novel heuristics to address such issues-- Handle other popular languages in which RESTful web services are often written, such as JavaScript/NodeJS and C#"
P_24,24,Test coverage criteria for RESTful web APIs,2019,ACM,NP: New analysis approach and potential solution,Workshop,A-TEST,Black Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java,,,No,,,,,,,SmartBear,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,Define test coverage levels,Defining coverage criteria,TRUE,FALSE,"Having more REST case-studies,Tool support",Further evaluations of the proposed criteria and the TCM with other APIs and test generation techniques>Provide tool support that allows to analyse test suites and find out their maximum TCL>Elaborate on the definition of the operation flows criterion>Use search-based techniques to generate test suites that maximise API coverage
P_25,25,Resource-based test case generation for RESTful web services,2019,ACM,NA: New automated approach and its extension,Conference,GECCO,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,,,,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",TRUE,No,No,,,,TRUE,TRUE,FALSE,Implement a set of effective templates based on the semantics of HTTP methods,Handling resource dependencies,TRUE,TRUE,"Having more REST case-studies,Resource handling","Do byte-code instrumentation on the source code of the SUT>Conduct additional experiments with
more case studies>Investigate novel solutions for doing a more intelligent analysis of resources of web services that are not fully following the REST guidelines>Investigate better strategies for more specialized and effective mutation operators"
P_26,26,Automatic Generation of Test Cases for REST APIs: A Specification-Based Approach,2018,IEEE,NA: New automated approach and its extension,Conference,EDOC,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,Api Tester,Java,Yes,,,,,,,"Dredd,SoapUI, Got-swag",https://github.com/opendata-for-all/api-tester,Input:OpenAPI>Output:>JUnit test case>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:,OpenAPI,JUnit test case,TRUE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,FALSE,FALSE,TRUE,Generate specification based test cases,,FALSE,FALSE,Tool support,"Increase the coverage levels by improving the parameter inference technique, for instance, by using natural language processing techniques to infer parameter values and constraints from the parameter descriptions>Support OpenAPI v3.0"
P_27,27,Metamorphic testing of RESTful web APIs,2018,IEEE,NA: New automated approach and its extension,Journal,IEEE TSE,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,,,No,,property-based,,,,,"API Fortress, RestAssured, Postman, HttpMaster, vREST, SoapUI",,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,No,No,,,,FALSE,FALSE,TRUE,Exploit metamorphic relations,Oracle problem,FALSE,FALSE,,Automated discovery of metamorphic relations beyond numerical programs
P_28,28,Quantitative security assurance metrics: REST API case studies,2018,ACM,NP: New analysis approach and potential solution,Conference,ECSA,Black Box,,security,,Industrial,Hosted on the Internet,Internet,Internet,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,not sure,,,FALSE,FALSE,FALSE,Quantify security assurance,,FALSE,FALSE,Security testing,Automation of test cases and the generation of test cases definition based on abstract application and threat models
P_29,29,Model-driven Testing of RESTful APIs,2015,ACM,NA: New automated approach and its extension,Conference,ICWWW,Black Box,,system,,Sample,Hosted locally,Local Artificial,Local,Artificial,,,,No,,,,,,,"RestAssured,SoapUI",,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,model-based test case generation,,FALSE,FALSE,"Security testing,Performance testing",Focus on this topic in order to enable test cases on a more abstract level>Aiming at a framework which offers an independent model to describe any existing REST API
P_30,30,Towards property-based testing of RESTful web services,2013,ACM,NA: New automated approach and its extension,Workshop,ACM SIGPLAN workshop on Erlang,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,,,FALSE,FALSE,"Security testing,Tool support","Automate the process>Include tools to focus on common bugs that are already well known: buffer overruns, problems with encoding, etc."
P_31,31,Connectedness testing of RESTful web-services,2010,ACM,NA: New automated approach and its extension,Conference,ISEC,Black Box,,system,connectedness,Sample,Hosted locally,Local Artificial,Local,Artificial,,,,No,,,,,,,Fiddler,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,not bad,,,FALSE,FALSE,FALSE,Connectedness testing using formal specifications,Handling resource dependencies,FALSE,FALSE,,Test if the deletion of certain resources does not result in creation of orphaned resources or broken links
P_32,32,Test-the-REST: An Approach to Testing RESTful Web-Services,2009,IEEE,NP: New analysis approach and potential solution,Conference,Computation World,Black Box,,system,,Sample,Hosted locally,Local Artificial,Local,Artificial,,,c#,No,,,,,,,"SoapUI, Fiddler, WebInject, JMeter",,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,,,FALSE,FALSE,,defining a formal web-service specification language>designing algorithms to automatically generate TTR test cases from web service specifications
P_33,33,Restats: A Test Coverage Tool for RESTful APIs,2021,IEEE,TD: Tool implementation or demonstration,Conference,ICSM,Black Box,,system,,Sample,Hosted locally,Local Artificial,Local,Artificial,,Restats,Python,Yes,,,,,Burp Suite,,"PortSWigger, SoapUI, Postman, RestAssured",https://github.com/SeUniVr/restats,"Input:OpenAPI,I/O Traces>Output:Test coverage metrics, Test coverage level>Auth:No>DB_Reset:No>ExternalServices:No>Other:","OpenAPI,I/O Traces","Test coverage metrics, Test coverage level",FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,Measure test coverage based on OpenAPI,Defining coverage criteria,FALSE,FALSE,,
P_34,34,Applying Change Impact Analysis Test to Migration Test Case Extraction Based on IDAU and Graph Analysis Techniques,2019,IEEE,NA: New automated approach and its extension,Workshop,ICSTW,White Box,,"unit,system",,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java,,C & Shell & Perl...,Yes,,,,,,Soot,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,GitHub - takedatmh/git: A fork of Git containing Windows-specific patches.,,,FALSE,FALSE,FALSE,Test-case extraction for a part of system modification and transition,,FALSE,FALSE,,Reaching the goal of useful test case reduction model by increasing the target application and source codes to generate complete model
P_35,35,Study on REST API Test Model Supporting Web Service Integration,2017,IEEE,TD: Tool implementation or demonstration,Conference,BIGDATASECURITY-HPSC-IDS,Hybrid,,integration,,No Case-studies,,No Case Studies,No Case Studies,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,"multi-dimensional API validation requirements, call sequencing and organization of test-cases, data-dependency between test cases",Handling resource dependencies,FALSE,FALSE,,
P_36,36,Differential regression testing for REST APIs,2020,ACM,NA: New automated approach and its extension,Conference,ISSTA,Black Box,,regression,,Industrial,Hosted on the Internet,Internet,Internet,,,Restler,"Python, F\#",Yes,,,,,,,"Tnt-Fuzzer, Sulley, QualysWAS, Burp Suite, Boofuzz, AppSpider, APIFuzzer, SoapUI, Postman",,Input:OpenAPI>Output:Unspecified test case>Auth:Yes>DB_Reset:No>ExternalServices:>Other:,OpenAPI,Unspecified test case,TRUE,FALSE,FALSE,FALSE,,FALSE,No,No,the goal is to find breaking changes in different versions of an API,,,FALSE,FALSE,FALSE,"find regressions in the API specifications, find regressions in the service",,FALSE,FALSE,,
P_37,37,Static analysis for web service security - Tools & techniques for a secure development life cycle,2015,IEEE,NP: New analysis approach and potential solution,Conference,IEEE-HST,Static Analysis,,security,,No Case-studies,,No Case Studies,No Case Studies,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,Discusses vulnerabilities of web services,,,FALSE,FALSE,FALSE,Static analysis techniques to overcome OWASP top 10 vulnerabilities,,FALSE,FALSE,Tool support,A sample toolkit for vulnerability analysis and perform comprehensive threat models for cloud environments and service oriented architectures
P_38,38,Adaptive Hypermutation for Search-Based System Test Generation: A Study on REST APIs with EvoMaster,2021,ACM,NA: New automated approach and its extension,Journal,TOSEM,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,EvoMaster,,No,,,,,RestAssured,,RestAssured,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,Select and mutate genes adaptively based on their fitness impact and mutation history during search,,TRUE,TRUE,Having more REST case-studies,"Conduct additional experiments with more evolutionary algorithms on more case studies>Conduct experiments with more budget settings
and study the performance with different budgets on different case studies"
P_39,39,Handling SQL Databases in Automated System Test Generation,2020,ACM,NA: New automated approach and its extension,Journal,TOSEM,White Box,,system,"could be used in other contexts, like GUI testing",Mixed (open source and industrial),Hosted locally,Local (Open-Source and Closed-Source),Local,Open-source and Closed-source,java & kotlin,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,RestAssured,Antlr,RestAssured,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",FALSE,No,No,,,,TRUE,FALSE,TRUE,Handle database,Handling database,TRUE,TRUE,Database handling,Design novel techniques to minimize negative effects of handling SQL heuristics as secondary objectives
P40,40,Open Problems in Fuzzing RESTful APIs: A Comparison of Tools,2022,arXiv,EA: Empirical study on various automated approaches,Open-access Repository,arXiv,Both,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java & kotlin,"EvoMaster,RESTest,Restler,bBOXRT,Schemathesis,RestTestGen,RestCT",,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),Yes,,,,TRUE,FALSE,FALSE,Study current limitations and effectiveness of fuzzer tools,,TRUE,TRUE,"Handling external services,Underspecified schemas","how to deal with underspecified schemas>how to deal with interactions with external services (e.g., other APIs and databases)"
P41,41,ARTE: Automated Generation of Realistic Test Inputs for Web APIs,2022,IEEE,NA: New automated approach and its extension,Journal,IEEE TSE,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,RESTest,Java,Yes,,,,,,,,https://github.com/isa-group/RESTest,"Input:OpenAPI>Output:JUnit test case>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:JAR file, Modification of Operations Under Test, Weights for parameters",OpenAPI,JUnit test case,TRUE,FALSE,FALSE,FALSE,"JAR file, Modification of Operations Under Test, Weights for parameters",FALSE,Yes(citations),No,https://github.com/isa-group/RESTest/wiki/Test-configuration-files#semanticgenerator-arte,,,FALSE,TRUE,TRUE,"Use natural language processing, search based and knowledge extraction techniques",,FALSE,FALSE,Tool support,Explore new heuristics for the search of more effective predicates>Refining the feedback loop for generating regular expressions>Develop a web API to ease the integration of ARTE into third party tools
P42,42,An OpenAPI-based Testing Framework to Monitor Non-Functional Properties of REST APIs,2020,Springer,NA: New automated approach and its extension,Conference,ICWE,Black Box,,system,non functional props,No Case-studies,No Case-studies,No Case Studies,No Case Studies,,,Gadolinium,"TypeScript, JavaScript",Yes,,,,,SoapUI,,SoapUI,https://github.com/opendata-for-all/gadolinium,"Input:OpenAPI>Output:OpenAPI,Visualized data (charts)>Auth:No>DB_Reset:No>ExternalServices:No>Other:Non-functional testing",OpenAPI,"OpenAPI,Visualized data (charts)",FALSE,FALSE,FALSE,FALSE,Non-functional testing,FALSE,Yes(related-work),No,,,,FALSE,FALSE,FALSE,Test non-functional properties by leveraging OpenAPI,,FALSE,FALSE,Tool support,"Support additional NFPs (e.g., throughput and reliability)>Support other cloud platforms to improve the geographical coverage>Exploring new visualization techniques to help developers study how structural properties (e.g., the size or structure of the payload of the operations) may affect the NFPs."
P43,43,Combinatorial Testing of RESTful APIs,2022,ACM,NA: New automated approach and its extension,Conference,ICSE,Black Box,,system,combinatorial,Industrial,Hosted on the Internet,Internet,Internet,,,RestCT,Python,Yes,,,,,,spacy,,https://github.com/GIST-NJU/RestCT,Input:OpenAPI>Output:Test results (statistics.csv and Swagger)>Auth:No>DB_Reset:No>ExternalServices:No>Other:,OpenAPI,Test results (statistics.csv and Swagger),FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),Yes,,,,FALSE,FALSE,FALSE,Cover interactions of input parameters in every single operation,Inferring inter-parameter dependencies,FALSE,FALSE,,
P44,44,Automated Test Generation for REST APIs: No Time to Rest Yet,2022,arXiv,EA: Empirical study on various automated approaches,Open-access Repository,arXiv,Both,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,various languages,"EvoMaster,RESTest,Restler,bBOXRT,Schemathesis,RestTestGen",Java,Yes,,,,,"APIFuzzer,Dredd,Tcases",,"RestAssured, API Fortress, APIFuzzer, Cats, Got-swag, Postman,TnT-Fuzzer",https://github.com/codingsoo/REST_GO,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),Yes,,,,TRUE,FALSE,TRUE,,,FALSE,FALSE,"Tool support,Resource handling",Extend evaluation by using mutation to further evaluate the tools’ fault-detection ability>leverage the insights gained from our study and preliminary investigations to develop new techniques for testing REST APIs that perform better input parameter generation and consider dependencies among operations>Investigate ways to extract meaningful input values from the API specification and server logs>study the application of symbolic analysis to extract relevant information from the code>research the use of NLP-based techniques to infer producer consumer dependencies between operations-- research the use of NLP-based techniques to infer producer-consumer dependencies between operations
P45,45,Enhancing Resource-Based Test Case Generation for RESTful APIs with SQL Handling,2021,Springer,NA: New automated approach and its extension,Conference,SSBSE,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java & kotlin,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,,,,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",,Yes(citations),No,,,,FALSE,FALSE,FALSE,,Handling database,TRUE,TRUE,"Having more REST case-studies,Resource handling","Having more REST case-studies,Resource handling"
P46,46,Model-based testing of restful web services using uml protocol state machines,2013,Penn State,NA: New automated approach and its extension,Conference,SAST,Black Box,,system,,No Case-studies,No Case-studies,No Case Studies,No Case Studies,,,,Java,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(related-work),No,,,,FALSE,FALSE,FALSE,Use UML protocol state machine as the formal behavioral model,,FALSE,FALSE,Tool support,"Extend the tool to include automatic test data generation and other coverage criteria>Deal with common features in most Web services like authentication, authorization, and resource representations (e.g, JSON and XML)>Investigate how the approach can be applied to test the
composition of RESTful Web services>Conducting experimental evaluations"
P48,48,foREST: A Tree-based Approach for Fuzzing RESTful APIs,2022,arXiv,NA: New automated approach and its extension,Open-access Repository,arXiv,Black Box,,system,,Open source industrial,Hosted locally,Local Closed-Source,Local,Closed-source,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),Yes,,,,TRUE,TRUE,FALSE,a tree-based approach to capture the resource dependencies,Handling resource dependencies,FALSE,FALSE,,
P49,49,Automatic Detection of Access Control Vulnerabilities via API Specification Processing,2022,arXiv,NA: New automated approach and its extension,Open-access Repository,arXiv,Black Box,,security,,,,Local Artificial,Local,Artificial,,,Python,No,,,,,"Autorize,Fuzz-lightyear ",,"Autorize,Fuzz-lightyear ",,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,,Oracle problem,FALSE,FALSE,Tool support,Handle cases where OpenAPI specification is not available>Trustworthiness of the API specification>Output tool has potential vulnerabilities
P50,50,Enhancing Search-based Testing with Testability Transformations for Existing APIs,2021,ACM,NA: New automated approach and its extension,Journal,TOSEM,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java & kotlin,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,RestAssured,,RestAssured,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",TRUE,Yes(citations),No,,,,TRUE,FALSE,TRUE,testability transformations to solve flag problem,White-box heuristics,TRUE,TRUE,White-box heuristics,"Provide more transformations for other common API methods>Integrate taint analysis, seeding and Dynamic Symbolic Execution in the search algorithm>Get high code coverage results and higher fault detection"
P51,51,On the Faults Found in REST APIs by Automated Test Generation,2022,ACM,NP: New analysis approach and potential solution,Journal,TOSEM,White Box,,system,,,,Local (Open-Source and Closed-Source),Local,Open-source and Closed-source,,EvoMaster,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),Yes,,,,FALSE,TRUE,TRUE,,,TRUE,TRUE,,Improving clustering>Industrial Validation
P52,52,REST API Fuzzing by Coverage Level Guided Blackbox Testing,2021,IEEE,NA: New automated approach and its extension,Conference,QRS,Black Box,,system,,Open source industrial,Hosted locally,Local Closed-Source,Local,Closed-source,"Go, Ruby, PHP (Real World is not that industrial)",Hsuan-Fuzz,Go,Yes,,,,,Go-fuzz,,"AFL, LibFuzzer, Go-fuzz",https://github.com/iasthc/hsuan-fuzz,"Input:OpenAPI,Path dependencies>Output:Unspecified test case>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:","OpenAPI,Path dependencies",Unspecified test case,TRUE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),No,"performs BB, however examines grey box in empirical studies to compare",,,TRUE,FALSE,FALSE,Resolve blind mutations issue by leveraging Test Coverage Level feedback,,FALSE,FALSE,"Security testing,Tool support,Classifying test results",Increasing Coverage Level>Manual Analysis and Automatic Identification>Supporting IoT and CoAP
P53,53,ICSE: G: Automated Management of Inter-Parameter Dependencies in Web APIs,2021,ACM,NP: New analysis approach and potential solution,Conference,ICSE,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,RESTest,Java,Yes,,,,,,"IDLReasoner, Kong",OpenAPI Generator,https://github.com/isa-group/RESTest,"Input:OpenAPI>Output:JUnit test case>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:JAR file, Modification of Operations Under Test, Weights for parameters",OpenAPI,JUnit test case,TRUE,FALSE,FALSE,FALSE,"JAR file, Modification of Operations Under Test, Weights for parameters",FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,Automated management of inter-parameter dependencies,Inferring inter-parameter dependencies,FALSE,FALSE,,
P54,54,More Effective Test Case Generation with Multiple Tribes of AI,2022,ACM,NA: New automated approach and its extension,Conference,ICSE,Black Box,,system,,,,Local Open-Source,Local,Open-source,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),Yes,,,,TRUE,TRUE,TRUE,Use grammar-based fuzzing and machine learning,,FALSE,FALSE,,
P55,55,Deriving Semantics-Aware Fuzzers from Web API Schemas,2021,ACM,NA: New automated approach and its extension,Conference,ICSE,Black Box,,system,,Open source industrial,Hosted locally,Local Closed-Source,Local,Closed-source,,Schemathesis,Python,Yes,,,,,"APIFuzzer,Fuzz-lightyear, Got-Swag, Cats, TnT-Fuzzer, Swagger-conform, Fuzzy-swagger, Swagger Fuzzer",,"HypoFuzz, Atheris, hypothesis, APIFuzzer,Fuzz-lightyear, Got-Swag, Cats, TnT-Fuzzer, Swagger-conform, Fuzzy-swagger, Swagger Fuzzer",https://github.com/schemathesis/schemathesis,"Input:OpenAPI>Output:Python test case>Auth:No>DB_Reset:No>ExternalServices:No>Other:Multi-worker test execution, Storing and replaying tests, ASGI/WSGI Support, Generated Code Samples, Docker Image, Customizable checks, Available as a Service, Available as CLI, Available as a Python Library,GraphQL",OpenAPI,Python test case,FALSE,FALSE,FALSE,FALSE,"Multi-worker test execution, Storing and replaying tests, ASGI/WSGI Support, Generated Code Samples, Docker Image, Customizable checks, Available as a Service, Available as CLI, Available as a Python Library,GraphQL",FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,Use property-based testing tools,,FALSE,FALSE,Tool support,Expand and share benchmark suite>Build defect analysis into fuzzers>Support the long tail of schema features>Investigate code-coverage-guided fuzzing for web APIs>Improved request-sequence testing without “links”
P56,56,Specification and automated analysis of inter-parameter dependencies in web APIs,2021,IEEE,NP: New analysis approach and potential solution,Journal,IEEE TSC,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,,,No,,,,,,IDLReasoner,PICT,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(related-work),No,,,,FALSE,FALSE,TRUE,Specify and analysis inter-parameter dependencies,Inferring inter-parameter dependencies,FALSE,FALSE,,
P57,57,Quality Assurance of Micro-Services - When to trust your micro-service test results?,2021,IEEE,NP: New analysis approach and potential solution,Conference,QRS,Black Box,,integration,,,,Local Artificial,Local,Artificial,,Instance Identification,Go,Yes,,,,,,,Postman,https://github.com/theovassiliou/instanceidentification,,,,TRUE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,instance identification,Instance identification,FALSE,FALSE,,"how IIDs can be used in professionally developed, large scale-microservice applications>extend the applicability to asynchronous communication paradigms"
P58,58,Solving the Instance Identification Problem in Micro-service Testing,2022,Springer,NP: New analysis approach and potential solution,Conference,ICTSS,Black Box,,integration,,No Case-studies,No Case-studies,No Case Studies,No Case Studies,,,,,No,,,,,,,Postman,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),Yes,,,,FALSE,FALSE,FALSE,instance identification,Instance identification,FALSE,FALSE,,"how IIDs can be used in professionally developed, large scale-microservice applications>extend the applicability to asynchronous communication paradigms"
P60,60,Jsongen: a QuickCheck Based Library for Testing JSON Web Services,2014,ACM,NA: New automated approach and its extension,Conference,Erlang,Black Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,Jsongen,Erlang,Yes,,,,,,,,https://github.com/fredlund/jsongen,Input:JSON schema>Output:QuickCheck generator>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:,JSON schema,QuickCheck generator,TRUE,FALSE,FALSE,FALSE,,FALSE,Yes(related-work),No,,,,FALSE,FALSE,FALSE,Use a quick-check property based testing tool,,FALSE,FALSE,Automated oracles,Extend the automatic checks done for pure RESTful web services>Embed schemas to characterize URI patterns
P61,61,Analyzing OpenAPI Specifications for Security Design Issues,2021,IEEE,NA: New automated approach and its extension,Conference,SecDev,Black Box,,security,,Open source industrial,No run,Local Closed-Source,Local,Closed-source,,,,No,,,,,,,ZAP,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,a semi-automatic approach to infer various key information about that API specification’s security issues,,FALSE,FALSE,,
P62,62,Efficient penetration of API sequences to test stateful RESTful services,2021,IEEE,NA: New automated approach and its extension,Conference,ICWS,Black Box,,system,combinatorial,Open source industrial,Hosted locally,Local Closed-Source,Local,Closed-source,,,Python,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),No,,,,,TRUE,FALSE,parse OpenAPI and create a directed tree that represents a partial order of calls of APIs to collect values demanded by each target API,,FALSE,FALSE,,Mix qualified-name strategy and unqualified-name strategy of values to associate API pairs that are overlooked by the current work
P63,63,Morest: Model-based RESTful API Testing with Execution Feedback,2022,arXiv,NA: New automated approach and its extension,Open-access Repository,arXiv,Black Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,,,No,,,,,,"PHPCoverage,CoverBand,JaCoCo",,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),Yes,,,,,FALSE,TRUE,a model-based RESTful API testing technique that builds and maintains a dynamically updating RESTful-service Property Graph (RPG) to model the behaviors of RESTful-services and guide the call sequence generation (dependencies),Handling resource dependencies,FALSE,FALSE,Tool support,Advanced input parameter generation>Fine-tuning of configurable options>Support undocumented APIs
P64,64,Pythia: Grammar-based fuzzing of REST APIs with coverage-guided feedback and learning-based mutations,2020,arXiv,NA: New automated approach and its extension,Open-access Repository,arXiv,White Box,,system,,Open source industrial,Hosted locally,Local Closed-Source,Local,Closed-source,"ruby, ...",,,No,,,,,,,"BooFuzz, Burp Suite, Peach Fuzzer",,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),No,"neither BB, nor WB: grammar-based",,,TRUE,TRUE,TRUE,augment grammar-based fuzzing with coverage-guided feedback and a learning-based mutation strategy,,FALSE,FALSE,,
P65,65,Bootstrapping Automated Testing for RESTful Web Services,2022,IEEE,NP: New analysis approach and potential solution,Journal,IEEE TSE,Black Box,,system,to remedy type collapse problem in API parameters,Industrial,Hosted on the Internet,Internet,Internet,,,,,No,,,,,"TnT-Fuzzer, Burp Suite, Fuzzapi",,"TnT-Fuzzer, Burp Suite, Fuzzapi",,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),Yes,,14 June,,FALSE,TRUE,TRUE,introduce FET techniques (... inter-parameter dependency inference),Inferring inter-parameter dependencies,FALSE,FALSE,"Handling external services,Tool support","Handle Inter-API Dependency>Update the FET Lattice
"
P66,66,Exploiting Input Sanitization for Regex Denial of Service,2022,ACM,NA: New automated approach and its extension,Conference,ICSE,Black Box,,system,detecting ReDos vulnerabilities,,Hosted on the Internet,Internet,Internet,,,Restler,,No,,,,,,,Prism (to validate requests and generate mock responses),,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),Yes,,may,,FALSE,FALSE,TRUE,apply Consistent Sanitization Assumption,,FALSE,FALSE,"Security testing,Tool support","Security testing,Tool support"
P67,67,Call Sequence List Distiller for Practical Stateful API Testing.,2020,KSIResearch,NP: New analysis approach and potential solution,Conference,SEKE,Black Box,,system,,,,Internet,,,types of apis are not specified,,Python,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),No,,,,FALSE,TRUE,FALSE,"Reduce API call sequences without reducing coverage, measure adequacy of an API sequence",,TRUE,FALSE,,More practical testing of stateful service APIs
P68,68,Hazard Integrated: Understanding Security Risks in App Extensions to Team Chat Systems,2022,KSIResearch,NA: New automated approach and its extension,Conference,NDSS Symposium,Black Box,,security,,Industrial,Hosted on the Internet,Internet,Internet,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),Yes,,april,,FALSE,FALSE,FALSE,,,FALSE,FALSE,,Data collection and vulnerability confirmation (particularly the issues unrelated to TACT APIs)>Support other policies and improve automation level
P69,69,A Framework for Automated API Fuzzing at Enterprise Scale,2022,IEEE,NA: New automated approach and its extension,Conference,ICST,Black Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,,,No,,,,,ZAP,"WSFuzzer, JaCoCo","Clusterfuzz, FuzzBuzz, Fuzzit, APIFuzzer, TnT-Fuzzer, ZAP, Burp Suite",,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),Yes,,8 June,,FALSE,FALSE,FALSE,utilize API specifications and combine them with cluster computing elasticity to fuzz multiple apps at once,,FALSE,FALSE,"Tool support,Performance testing","Allow users to provide their fuzzers
via our API>Support non-web specifications>Support combinatorial and performance testing"
P70,70,Testability transformations for existing APIs,2020,IEEE,NA: New automated approach and its extension,Conference,ICST,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java & kotlin,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,RestAssured,,,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",TRUE,Yes(citations),No,,,,TRUE,FALSE,FALSE,testability transformations to solve flag problem,White-box heuristics,TRUE,TRUE,White-box heuristics,"Provide more transformations for other common API methods>Integrate techniques and ideas from taint analysis, seeding and dynamic symbolic execution in the search algorithms to improve performance"
P71,71,SQL data generation to enhance search-based system testing,2019,ACM,NA: New automated approach and its extension,Conference,GECCO,White Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,java & kotlin,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,RestAssured,,RestAssured,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",FALSE,Yes(citations),No,,,,TRUE,FALSE,FALSE,Handle database,Handling database,TRUE,TRUE,Database handling,"Investigate other approaches to generate such type of SQL data, like for example by using Dynamic Symbolic Execution>support other kinds of databases, like NoSQL"
P72,72,Exploratory test agents for stateful software systems,2019,ACM,PI: Proposal or idea,Conference,ESEC/FSE,Black Box,,system,,No Case-studies,,No Case Studies,No Case Studies,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,Use exploratory test agents,,FALSE,FALSE,,
P73,73,EvoMaster: A search-based system test generation tool,2021,Open Journals,TD: Tool implementation or demonstration,Journal,JOSS,Both,,system,,No Case-studies,,No Case Studies,No Case Studies,,,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,,,,https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,SBST,,FALSE,TRUE,,
P74,74,REST API composition for effectively testing the Cloud,2022,unam.mx,TD: Tool implementation or demonstration,Journal,JART,Black Box,,system,,,,,,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),Yes,,22 March,,FALSE,FALSE,FALSE,,,FALSE,FALSE,Tool support,Tool support
P75,75,Design and Implementation of a Test Tool for PSD2 Compliant Interfaces,2021,Semantic Scholar,TD: Tool implementation or demonstration,Conference,ICEIS,Black Box,,security,,Sample,Hosted locally,Local Artificial,Local,Artificial,,,,No,,,,,RestAssured,,RestAssured,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,,,FALSE,FALSE,Having more REST case-studies,Evaluate the test tool in a real-world setting>Extract general guidelines and findings on testing standardized interfaces and evaluate them by transferring them to other industries with interface standards
P76,76,MockRest—A Generic Approach for Automated Mock Framework for REST APIs Generation,2020,Springer,NP: New analysis approach and potential solution,Conference,ICICCT,Black Box,,unit,,Sample,Hosted locally,Local Artificial,Local,Artificial,java,,,No,mock for unit test,,,,,,SoapUI,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,present a mock framework,Mocking,FALSE,FALSE,,
P77,77,Development of Vulnerable Web Application Based on OWASP API Security Risks,2021,IEEE,PI: Proposal or idea,Conference,IES,None,,security,,No Case-studies,,No Case Studies,No Case Studies,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,proppose an API security learning environment called Vulnerable Academic Information System (VAIS) based on the OWASP API Security Risks,,FALSE,FALSE,"Security testing,Tool support",Implementation of VAIS by using popular microframeworks and database management system based on the proposed design>Evaluation and Analysis of each API Security Risks in VAIS challenges>Stress Test Analysis on containerized VAIS againts common web attack vectors>Analysis VAIS game based existing gamification frameworks
P78,78,Software Tool to Perform Metamorphic Testing on RESTful Web APIs,2022,Springer,NA: New automated approach and its extension,Conference,ICICSE,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,,react.js and node.js,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,deploy metamorphic testing using metamorphic relation in production phase,Oracle problem,FALSE,FALSE,Tool support,Extend the tool to support multiple APIs at a time>Support non REST APIs
P79,79,CPN Model Based Standard Feature Verification Method for REST Service Architecture,2018,IEEE,NA: New automated approach and its extension,Conference,CollaborateCom,Black Box,,architecture,,Sample,,Local Artificial,Local,Artificial,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,propose a verification method based on Colored Petri Nets,,FALSE,FALSE,Having more REST case-studies,Strengthen the formal description and validation of the core process of the verification method>Apply the method to practical objects with more typical REST services
P80,80,Automation of Integration Testing of RESTful Hypermedia Systems: A Model-driven Approach,2018,Semantic Scholar,NA: New automated approach and its extension,Conference,Webist,Black Box,,integration,,Sample,,Local Artificial,Local,Artificial,sample case study?!,,,No,,,,,,,Selenium,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),No,,,,FALSE,FALSE,FALSE,propose a Model-driven Testing (MDT) approach for hypermedia systems,,FALSE,FALSE,Automated oracles,"Extend the model crawler to extract more possibilities to generate different types of negative testing, such as applying not allowed methods, trying to access a (sub-)resource after deleting it or forcing the HTTP client to send wrong resource representations>Discuss more about how the application should behave in case of invalid requests"
P81,81,RESTful API automated test case generation,2017,IEEE,NA: New automated approach and its extension,Conference,QRS,White Box,,system,,Mixed (open source and industrial),Hosted locally,Local (Open-Source and Closed-Source),Local,Open-source and Closed-source,,EvoMaster,"Java, Kotlin, C\#, TypeScript, JavaScript",Yes,,,,,RestAssured,,"RestAssured, Wiremock",https://github.com/EMResearch/EvoMaster,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",TRUE,Yes(citations),No,,,,TRUE,TRUE,TRUE,SBST,,FALSE,TRUE,"Handling external services,Tool support,White-box heuristics,Database handling","handling of string constraints, accesses to database and to other external services"
P82,82,Unit and integration testing of modular cloud services,2017,IEEE,TD: Tool implementation or demonstration,Conference,AINA,Both,,"unit,integration",,,,,,,,,,No,,,,,,,"Selenium Testing, ...",,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,two fold strategy (unit and integration testing),,FALSE,FALSE,Performance testing,Testing of more complex scenarios regarding performance metrics and multi tenancy>Increase the number of TestCast test cases
P83,83,Tool Demonstration: Testing JSON Web Services Using JSONGEN,2018,sistedes.es,TD: Tool implementation or demonstration,Conference,PROLE,Black Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,Jsongen,Erlang,Yes,,,,,,,"JSON Schema Faker, Schematic Ipsum, json-schema-random",https://github.com/fredlund/jsongen,Input:JSON schema>Output:QuickCheck generator>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:,JSON schema,QuickCheck generator,TRUE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,Use a quick-check property based testing tool,,FALSE,FALSE,Automated oracles,Extend the automatic checks done for pure RESTful web services
P84,84,Towards automated testing of web service choreographies,2011,ACM,PI: Proposal or idea,Conference,AST,Black Box,,"unit,integration,acceptance",,Sample,Hosted locally,Local Artificial,Local,Artificial,,,,No,,,,,,,"SoapUI, WebInject",,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,adapt automated testing techniques to SOA,,FALSE,FALSE,,"Deal with the lack of observability> Some inherent characteristics of service compositions such as dynamicity, adaptiveness, and governance issues must also be solved to automate the integration tests>Make the framework applicable in any choreography>some environment issues such as the non consolidation of choreography service standards, the decentralized flow of information, multiple party communication, and parallelism must be treated"
P85,85,Optimized Test Data Generation for RESTful Web Service,2017,IEEE,NA: New automated approach and its extension,Conference,ASPEC,Black Box,,system,,Sample,Hosted locally,Local Artificial,Local,Artificial,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,"SBST, use Hadoop to implement parallelized test data generation",,FALSE,FALSE,,
P86,86,REST service testing based on inferred XML schemas,2014,Macrothink Institute,NA: New automated approach and its extension,Journal,Network Protocols and Algorithms,Black Box,,system,,Mixed(sample and industrial),Mixed,Internet and Local Artificial,Both,Artificial,,,,No,,,,,SoapUI,,"RestAssured,SoapUI",,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,statistical analysis over responses produced at multiple service invocations,,FALSE,FALSE,Tool support,Possibility of including as input the XSD expected and comparing it with the obtained one to detect anomalous elements>Apply the same processing to responses in JSON or other formats>Improve anomaly detector module and its diagnosis
P87,87,Predictive Simulation for Building Trust Within Service-Based Ecosystems,2022,IEEE,NP: New analysis approach and potential solution,Workshop,PerCom Workshops,Black Box,,,,No Case-studies,,No Case Studies,No Case Studies,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),Yes,,6 May,,FALSE,FALSE,FALSE,,,FALSE,FALSE,,
P88,88,ExVivoMicroTest: ExVivo Testing of Microservices,2022,Wiley,NA: New automated approach and its extension,Journal,Journal of Software: Evolution and Process,Black Box,,regression,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,,ExVivoMicroTest,Python,Yes,,,,,,,,https://gitlab.com/learnERC/exvivomicrotest,Input:I/O Traces>Output:Python test case>Auth:No>DB_Reset:No>ExternalServices:No>Other:,I/O Traces,Python test case,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),Yes,,12 April,,FALSE,FALSE,FALSE,"analyzes the execution of deployed services at run-time, ",,FALSE,FALSE,Security testing,Study and compare the fault-detection capability of in house and ex vivo tests>Investigate the usage and definition of obfuscation techniques that can hide the actual collected data without changing the executions
P90,90,Verification of microservices using metamorphic testing,2019,Springer,NA: New automated approach and its extension,Conference,ICA3PP,Black Box,,system,,Open Source,Hosted locally,Local Open-Source,Local,Open-source,"java, JavaScript",,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,TRUE,Yes(citations),No,,,,FALSE,FALSE,TRUE,apply metamorphic testing into the verification of microservices-based applications,Oracle problem,FALSE,FALSE,Having more REST case-studies,Evaluate the metamorphic testing against much larger real-world microservice based applications to identify further research direction in applying metamorphic testing to aid the verification of microservices-based applications
P91,91,Testing cloud services using the TestCast tool,2018,Springer,NP: New analysis approach and potential solution,Conference,ITNG,Both,,"unit,integration",,No Case-studies,No Case-studies,No Case Studies,No Case Studies,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,,,FALSE,FALSE,,
P92,92,RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs,2020,Springer,NA: New automated approach and its extension,Conference,ICSOC,Black Box,,system,,Industrial,Hosted on the Internet,Internet,Internet,,,RESTest,Java,Yes,,,,,RestAssured,,RestAssured,https://github.com/isa-group/RESTest,"Input:OpenAPI>Output:JUnit test case>Auth:Yes>DB_Reset:No>ExternalServices:No>Other:JAR file, Modification of Operations Under Test, Weights for parameters",OpenAPI,JUnit test case,TRUE,FALSE,FALSE,FALSE,"JAR file, Modification of Operations Under Test, Weights for parameters",TRUE,Yes(citations),No,,https://github.com/isa-group/icsoc-2020-supplementary-material,,FALSE,FALSE,TRUE,Specify and analysis inter-parameter dependencies,Inferring inter-parameter dependencies,FALSE,FALSE,"Tool support,Non-functional testing","Implement currently missing features in RESTest, such as testing of sequences of operations and search-based online testing approaches>Make RESTest SLA-aware with SLA4OAI"
P94,94,Verification of Hypermedia Characteristic of RESTful Finite-State Machines,2018,ACM,NP: New analysis approach and potential solution,Conference,The Web Conference,Black Box,,system,,,,No Case Studies,No Case Studies,,,,,,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,No,No,,,,FALSE,FALSE,FALSE,,,FALSE,FALSE,Automated oracles,Generate robust RESTful APIs along with tests to assure Fielding’s hypermedia constraint
P97,97,A catalogue of inter-parameter dependencies in RESTful web APIs,2019,Springer,NP: New analysis approach and potential solution,Conference,ICSOC,Black Box,,system,,Industrial Case-studies,,Internet,Internet,,,,,No,,,,,,,,,,,,FALSE,FALSE,FALSE,FALSE,,FALSE,Yes(citations),No,,,,FALSE,FALSE,FALSE,study on the presence of dependency constraints among input parameters in web APIs in industry,Inferring inter-parameter dependencies,FALSE,FALSE,Tool support,Modelling and analysis of input constraints in web APIs>Perform an empirical study assessing the validity of the conclusions drawn from our investigation
P98,98,JavaScript Instrumentation for Search-Based Software Testing: A Study with RESTful APIs,2022,IEEE,NA: New automated approach and its extension,Conference,ICST,White Box,,system,,,,Local Open-Source,Local,Open-source,JavaScript,EvoMaster,,Yes,,,,,RestAssured,,RestAssured,,"Input:OpenAPI>Output:JUnit test case,xUnit test case,Jest test case>Auth:Yes>DB_Reset:Yes>ExternalServices:No>Other:Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",OpenAPI,"JUnit test case,xUnit test case,Jest test case",TRUE,TRUE,FALSE,TRUE,"Both BB & WB, installers (.msi for Windows, .dmg for OSX, .deb for Linux) also a jar file, Self-contained tests,Fault Detection,Automated Code Instrumentation",TRUE,Yes,Yes,,,,,,,SBST,White-box heuristics,TRUE,TRUE,Tool support,Extend with Testability Transformations