Dataset Open Access

How Developers Locate Performance Bugs — Supplementary Material

Baltes, Sebastian; Moseler, Oliver; Beck, Fabian; Diehl, Stephan

Abstract:

Background: Performance bugs can lead to severe issues regarding computation efficiency, power consumption, and user experience. Locating these bugs is a difficult task because developers have to judge for every costly operation whether runtime is consumed necessarily or unnecessarily. Objective: We wanted to investigate how developers, when locating performance bugs, navigate through the code, understand the program, and communicate the detected issues.

Method: We performed a qualitative user study observing twelve developers trying to fix documented performance bugs in two open source projects. The developers worked with a profiling and analysis tool that visually depicts runtime information in a list representation and embedded into the source code view.

Results: We identified typical navigation strategies developers used for pinpointing the bug, for instance, following method calls based on runtime consumption. The integration of visualization and code helped developers to understand the bug. Sketches visualizing data structures and algorithms turned out to be valuable for externalizing and communicating the comprehension process for complex bugs.

Conclusion: Fixing a performance bug is a code comprehension and navigation problem. Flexible navigation features based on executed methods and a close integration of source code and performance information support the process.

Dataset:

  1. Tutorial: We provide the slides (PDF) and the video (MP4) we used in the tutorial phase of our study.

  2. Locating Bugs: We also provide supplementary material for each research question. We provide the advices we prepared for each bug in case a team got stuck (PDF); the questions we asked after each bug fixing session can be found on the introduction slides (PDF).

    • RQ1: Navigating and Understanding

      • RQ1.1: How was information from the profiling tool or other parts of the IDE used to locate the performance bug? Cross-case analysis (in German) (XLSX+ODS)

      • RQ1.2: Is the in-situ visualization of the profiling data beneficial compared to a traditional list representation? Cross-case analysis (in German) (XLSX+ODS)

      • RQ1.3: What navigation strategies do developers pursue to locate a specific performance bug? Interaction logs (TXT), Navigation visualizations (SVG), Screen recordings for Bug 3 (MP4, without audio because of confidentiality)

    • RQ2: Understanding and Communicating

      • RQ2.1: How do developers communicate with each other when locating a performance bug? Coding (XLSX+ODS), Sketches (PDF), Screen recordings for Bug 3 (MP4, without audio because of confidentiality)

      • RQ2.2: Could sketches help to understand and communicate a performance bug? Coding (XLSX+ODS), Sketches (PDF), Cross-case analysis (in German) (XLSX+ODS), Sketching videos for Bug 3 (MP4, without audio because of confidentiality)

  3. Questionnaire: The questionnaire that the participants filled out at the end of the study can be found here (PDF).

Files (1.5 GB)
Name Size
coding_bug3.ods
md5:3c54b63b8d7374f76adcbae75d7681e9
34.4 kB Download
coding_bug3.xlsx
md5:a63da567f5f9b98c1389d0d3ab1e4d1c
50.6 kB Download
cross_case_analysis.ods
md5:57ce930e8122155a12dbeb540a0f3fed
10.6 kB Download
cross_case_analysis.xlsx
md5:cee28c7fd109a55380157b05afb352bd
18.1 kB Download
group1_bug3_screencapture_video_only.mp4
md5:8b2eb6439a1b6ea16fb053ab5da772af
97.7 MB Download
group1_bug3_sketching_video_only.mp4
md5:46eea210dc9511b0ae763d2ba46d4a47
130.9 MB Download
group2_bug3_screencapture_video_only.mp4
md5:a8df4dcdc0d8254323699a56f931b3e8
71.6 MB Download
group2_bug3_sketching_video_only.mp4
md5:9fc83d92a60ee0a8ec2125c7bc2c7fb3
72.7 MB Download
group3_bug3_screencapture_video_only.mp4
md5:0ac484c284cccb3555a0b5fe4b3c5398
76.4 MB Download
group3_bug3_sketching_video_only.mp4
md5:42a687c902b65ebbab90bbdafbf1e1ee
89.9 MB Download
group4_bug3_screencapture_video_only.mp4
md5:c561e6f5c35c14186d330290f51bc402
125.8 MB Download
group4_bug3_sketching_video_only.mp4
md5:28a366cf402310daf78240710114d059
200.0 MB Download
group5_bug3_screencapture_video_only.mp4
md5:6cb648bb93f2e065af03c297d59fd090
125.9 MB Download
group5_bug3_sketching_video_only.mp4
md5:c951398286adca689c3957e477b4506d
112.7 MB Download
group6_bug3_screencapture_video_only.mp4
md5:fb7284bb485f7bfd8da876df9f7857f0
148.9 MB Download
group6_bug3_sketching_video_only.mp4
md5:d833faa49a785db059ee26db5cd0cbad
244.0 MB Download
hints_performance_test_1.pdf
md5:7dce095553efc643f3d197a5716ac6b9
77.2 kB Download
hints_performance_test_2.pdf
md5:64fabbbdfabd924a72175fc2e8e3d866
78.1 kB Download
hints_performance_test_3.pdf
md5:e1640e62e6e848e0726bc7d63d90855e
77.1 kB Download
hints_performance_test_4.pdf
md5:ec02d087f3664c3dc35e188747cb65ea
54.6 kB Download
interaction-logs_group1-1.log
md5:e8b79428f2402436fbb337cab10a01d3
9.4 kB Download
interaction-logs_group1-2.log
md5:506b0dab31be7da95d1c3a01086262cb
38.3 kB Download
interaction-logs_group1-3.log
md5:13f9f72de5c86737f166a42253ec2cd9
12.3 kB Download
interaction-logs_group2.log
md5:6d6a5d16309222d736bdd89e58c80835
40.9 kB Download
interaction-logs_group3.log
md5:45a1644e52bb1402061ac079eef98c29
52.5 kB Download
interaction-logs_group4.log
md5:7cfcc232610d4705561e13f779bfcebe
34.0 kB Download
interaction-logs_group5.log
md5:a63993646cfb6450025ea7a0c16f4155
52.7 kB Download
interaction-logs_group6.log
md5:8c5d0f633f536c0b156be837e32b38fd
26.8 kB Download
introduction_to_sampling_approach.mp4
md5:ab4d77ef1eb8fd4d288af94c9aab60c9
5.8 MB Download
navigation-vis_group1_perf_bug_3.svg
md5:4c42fb02142d5074985a0fc7869eb987
21.1 kB Download
navigation-vis_group2_perf_bug_3.svg
md5:4ecb6f2e277b6435786a9b03122ed7e0
17.8 kB Download
navigation-vis_group3_perf_bug_3.svg
md5:9d3b594ae6de362b047752419938da66
22.6 kB Download
navigation-vis_group4_perf_bug_3.svg
md5:02af9fb4fe8da0f694034f1017f29d41
17.2 kB Download
navigation-vis_group5_perf_bug_3.svg
md5:2e221bb90a7f336b484891bf7f97678c
22.6 kB Download
navigation-vis_group6_perf_bug_3.svg
md5:5176c70fc067988789fa2cceadf04f03
10.5 kB Download
questionnaire.pdf
md5:db302be7ea40f36d23967e5345ad7ec5
65.0 kB Download
sketches_group1.pdf
md5:1d7f7e4b1ae304e05f72c9584d1492f1
3.6 MB Download
sketches_group2.pdf
md5:e46fe0e8e51594d81704de4500cc8289
2.5 MB Download
sketches_group3.pdf
md5:526a2cba11283fd740fec334790aada9
2.5 MB Download
sketches_group4.pdf
md5:41cc7fe20b1c6a60d431c766c7ac9c86
1.1 MB Download
sketches_group5.pdf
md5:aeec6bb9b198676e0f51a2879d5e61ea
1.8 MB Download
sketches_group6.pdf
md5:e4d00e8efc7fefd9831adc0594dbd830
2.8 MB Download
tutorial.pdf
md5:adadf3983ce27074a445394ff1d08e0f
1.5 MB Download
  • Baltes et al. — Navigate, Understand, Communicate: How Developers Locate Performance Bugs (ESEM 2015): http://ieeexplore.ieee.org/document/7321208/
  • Baltes et al. — Navigate, Understand, Communicate: How Developers Locate Performance Bugs (ESEM 2015): https://sbaltes.github.io/assets/pdf/esem15-debugging.pdf
19
85
views
downloads
All versions This version
Views 1919
Downloads 8585
Data volume 1.5 GB1.5 GB
Unique views 1414
Unique downloads 1111

Share

Cite as