FragGen Replication Package

- VM Folder structure - - /home/fraggen/fraggen (project home) - dockerApps (subjects) - GroundTruth (ground truth models) - pythonCode (contains scripts to run experiments) - jars (required jars) - results (Final results after experiments available here) - saveJsons (folder to save manual classification jsons - out (output of crawls and tests) - paper-crawls (Contains crawls reported in the paper) * - test-results_mac (regression tests in mac server) * - test-results_rhel (regression tests in mac server) * - oracle_analysis (manually labelled oracle failures) * * - to be extracted from FragGenData separately - RQ1 -folder -/home/TSE_RQ1/tse_rq1 - readme.md : contains additional instructions - TSE_Fraggen.py : contains comments to help run the code - Need to download dataset_rq1 separately and use the location in TSE_Fraggen.py - To get results reported in paper - python3 TSE_Fraggen.py - Setup/instructions before starting experiments - - Make sure VM has working internet connection - Set the screen resolution to atleast 1400x900 - Update chrome browser if necessary for your experiments - Memory Requirements : recommended 8GB - Hard Disk : recommended (100GB) - Always save manually classified json files into /home/fraggen/fraggen/saveJsons - Using Crawls reported in the paper for RQ2, RQ3 - Download the provided "FragGenData.7z" from zenodo - Place the extracted data : /home/fraggen/fraggen/paper-crawls (crawl, mutationRun) : /home/fraggen/fraggen/test-results_mac (e1) : /home/fraggen/fraggen/test-results_rhel (e2, e3) : /home/fraggen/fraggen/oracle_analysis - RQ2 - cd /home/fraggen/fraggen/pythonCode - python3 analyzeCrawl.py - set ALL_CRAWLS = "../paper-crawls" - raw stats for all crawls - /home/fraggen/fraggen/paper_crawls/results/analysis_BestCrawls_60min.csv - RQ3a - - cd /home/fraggen/fraggen/pythonCode - python3 analyzeTestRun_Regression.py - use the function "analyzePaperTestRuns()" - results generated in results folder - raw results for all tests - /home/fraggen/fraggen/results/testRunStats.csv - python3 analyzeTestRun_Regression.py - use the function "testAnalyzeManualTestOracles()" - prints the results to stdout - manually classified 100 failures per oracle - RQ3b - - cd /home/fraggen/fraggen/pythonCode - python3 analyzeTestRun_alt.py - ALL_CRAWLS = "/home/fraggen/fraggen/paper-crawls" - mutation analysis for all the oracles. - results generated in results folder - test run stats (no of tests passed, failed etc) - /home/fraggen/fraggen/results/mutationStats.csv - raw results for all oracles - /home/fraggen/fraggen/results/mutationOracleStats.csv To Run New Experiments : - Running Crawls - - cd /home/fraggen/fraggen/pythonCode - python3 runCrawljaxBatch.py - set "DRY_RUN = False" to get the expected runtime - set the runtime as required - Runs the crawls for all the subject apps - generates the crawls in "/home/fraggen/fraggen/out" - RQ2 - - Familiarize yourself with groundtruth models in /home/fraggen/fraggen/GroundTruth - cd /home/fraggen/fraggen/pythonCode - python3 analyzeCrawl.py - set ALL_CRAWLS = "../out" - creates classification.html for each crawl produced by runCrawljaxBatch.py - Assigns bin values and classifications for state-pairs in new crawls where possible. - manually classify state-pairs using web interface in each crawl. - /home/fraggen/fraggen/out/subject/algo/host/crawl/html_classification/classification.html - subject : eight subject apps - algo: e.g., addressbook_DOM_RTED_0.0_60mins - host: e.g., localhost - crawl: e.g, crawl0 - python3 analyzeCrawl.py - analyzes the classification for new crawls w.r.t ground truth and produces results - /home/fraggen/fraggen/results/analysis_BestCrawls_60min.csv - RQ3a - - cd /home/fraggen/fraggen/pythonCode - python3 runGeneratedTests.py - set ALL_CRAWLS = "/home/fraggen/fraggen/out" - set (MUTATION=False) - runs tests in each of the generated tests for new technique - python3 analyzeTestRun_Regression.py - use the function "analyzeNewTestRuns()" - generates HTML to analyze failed Test events - manually classify state-pairs using web interface in each crawl. - python3 analyzeTestRun_Regression.py - results generated in results folder - raw results for all tests - /home/fraggen/fraggen/results/testRunStats.csv - RQ3b - - cd /home/fraggen/fraggen/pythonCode - python3 runGeneratedTests.py - set ALL_CRAWLS = "/home/fraggen/fraggen/out" - set (MUTATION=True) - runs tests in each of the generated tests for new technique - python3 analyzeTestRun_alt.py - set ALL_CRAWLS = "/home/fraggen/fraggen/out" - mutation analysis for all the oracles. - results generated in results folder - test run stats (no of tests passed, failed etc) - /home/fraggen/fraggen/results/mutationStats.csv - raw results for all oracles - /home/fraggen/fraggen/results/mutationOracleStats.csv - Known Issues - - Selenium may fail to start a new chrome browser. - possible reasons include internet connectivity - Rerun fixes the issue if there is a working internet connection - Existing crawljax testGenerator may produce multiple methods with the same name - runGeneratedTests will fail if this is the case - fix the compilation errors manually in GeneratedTests.java in corresponding crawl to continue