How Developers Locate Performance Bugs — Supplementary Material

Abstract:

Background: Performance bugs can lead to severe issues regarding computation efficiency, power consumption, and user experience. Locating these bugs is a difficult task because developers have to judge for every costly operation whether runtime is consumed necessarily or unnecessarily. Objective: We wanted to investigate how developers, when locating performance bugs, navigate through the code, understand the program, and communicate the detected issues.

Method: We performed a qualitative user study observing twelve developers trying to fix documented performance bugs in two open source projects. The developers worked with a profiling and analysis tool that visually depicts runtime information in a list representation and embedded into the source code view.

Results: We identified typical navigation strategies developers used for pinpointing the bug, for instance, following method calls based on runtime consumption. The integration of visualization and code helped developers to understand the bug. Sketches visualizing data structures and algorithms turned out to be valuable for externalizing and communicating the comprehension process for complex bugs.

Conclusion: Fixing a performance bug is a code comprehension and navigation problem. Flexible navigation features based on executed methods and a close integration of source code and performance information support the process.

Dataset:

1. Tutorial: We provide the slides (PDF) and the video (MP4) we used in the tutorial phase of our study.

2. Locating Bugs: We also provide supplementary material for each research question. We provide the advices we prepared for each bug in case a team got stuck (PDF); the questions we asked after each bug fixing session can be found on the introduction slides (PDF).

   • RQ1: Navigating and Understanding

     • RQ1.1: How was information from the profiling tool or other parts of the IDE used to locate the performance bug? Cross-case analysis (in German) (XLSX+ODS)

     • RQ1.2: Is the in-situ visualization of the profiling data beneficial compared to a traditional list representation? Cross-case analysis (in German) (XLSX+ODS)

     • RQ1.3: What navigation strategies do developers pursue to locate a specific performance bug? Interaction logs (TXT), Navigation visualizations (SVG), Screen recordings for Bug 3 (MP4, without audio because of confidentiality)

   • RQ2: Understanding and Communicating

     • RQ2.1: How do developers communicate with each other when locating a performance bug? Coding (XLSX+ODS), Sketches (PDF), Screen recordings for Bug 3 (MP4, without audio because of confidentiality)

     • RQ2.2: Could sketches help to understand and communicate a performance bug? Coding (XLSX+ODS), Sketches (PDF), Cross-case analysis (in German) (XLSX+ODS), Sketching videos for Bug 3 (MP4, without audio because of confidentiality)

3. Questionnaire: The questionnaire that the participants filled out at the end of the study can be found here (PDF).