Detecting and Characterizing Bots that Commit Code

Preprint of a paper accepted in MSR 2020 conference.

Abstract:

  Background: Some developer activity traditionally performed
  manually, such as making code commits, opening, managing, or
  closing issues is increasingly subject to automation in many OSS
  projects. Specifically, such activity is often performed by tools that
  react to events or run at specific times. We refer to such
  automation tools as bots and, in many software mining scenarios
  related to developer productivity or code quality it is desirable
  to identify bots in order to separate their actions from actions
  of individuals.
  Aim: Find an automated way of identifying bots and
  code committed by these bots, and to characterize the types of
  bots based on their activity patterns.  
  Method and Result: We propose BIMAN, a systematic approach to detect bots using author names, commit messages, files modified by the commit, and projects associated with the commits. For our test data, the value for AUC-ROC was 0.9. We also characterized these bots based on the time
  patterns of their code commits and the types of files
  modified, and found that they primarily work with documentation files and web pages, and these files are most prevalent in HTML and JavaScript ecosystems. We have compiled a shareable dataset containing detailed
  information about 461 bots we found (all of whom have more than 1000 commits) and 13,762,430 commits they created.