Detecting and Characterizing Bots that Commit Code

Background: Some developer activity traditionally performed manually, such as making code commits, opening, managing, or closing issues is increasingly subject to automation in many OSS projects. Specifically, such activity is often performed by tools that react to events or run at specific times. We refer to such automation tools as bots and, in many software mining scenarios related to developer productivity or code quality it is desirable to identify bots in order to separate their actions from actions of individuals. Aim: Find an automated way of identifying bots and code committed by these bots, and to characterize the types of bots based on their activity patterns. Method: Starting with a dataset containing 13,150 bots and 13,150 humans detected using simple heuristics, we trained a Random Forest model using variables describing the activity patterns of the authors (or bots), and another model that identified bots based on a heuristics used to detect whether the commit message was generated from a template. We also characterized these bots based on the time patterns of their code commits and the types of files modified.  We have compiled a shareable dataset containing information about the bots we found and the commits they created.