Below we describe the analysis done for the paper An Empirical Study of Bots in Software Development: Characteristics and Challenges from a Practitioner’s Perspective. Note that this R Markdown file can be turned into an R script to be modified in case you wish to analyse something different in our data.

We divide our analysis into several parts, and details regarding motivation and findings for each part are found in the manuscript. Here, we focus on presenting (i) the goals and (ii) steps for each analysis as reproducible scripts.

Setup: Loading dependencies and the dataset:

This will clean your environment to avoid loading unnecessary functions or variables.

rm(list = ls())

We begin by installing and loading the required packages.

# You can install packages in case they are not installed already.
# install.packages(c("tidyverse", "likert", "gridExtra") 
library(tidyverse)
library(likert)
library(gridExtra)

Next, we load the results from the survey into a data frame.

survey_answers <- read.csv2("complete_survey_responses.csv", header = T)

# We also create a graphical object that has the default theme for ggplot throughout this script.
PLOT_THEME <- theme_minimal(base_size = 14) + 
        theme(plot.title = element_text(size = 14, face = "bold", hjust = 0.5),
        panel.grid = element_blank(), axis.line = element_line(size = 1), axis.ticks = element_line(size = 1),
        strip.background = element_rect(colour = "white", fill = "aliceblue"), legend.position = "bottom")

Part 1: What tools do participants perceive as bots?

Goal: Get an overview of the participants’ answers about bot definitions.

We begin by analysing the questions regarding bot definition. Our goal is to see: (i) which tool descriptions practitioners perceive as bots, and (ii) map our participants to the defined personas.

The first step is to select the corresponding questions (columns) from the survey data. Since the survey data is quite extensive, this will make data wrangling easier. We also clean the column names to use short IDs and have cleaner plots. This helps when printing the data. The original text for questions can be found in the CSV file and also in Table 2 of the paper.

DEFINITION_QUESTIONS_COL <- 10:41

#Selects columns and turns values into factors to be used as likert scales.
bot_definitions_df <- select(survey_answers,DEFINITION_QUESTIONS_COL) %>% mutate_all(factor)
## Note: Using an external vector in selections is ambiguous.
## ℹ Use `all_of(DEFINITION_QUESTIONS_COL)` instead of `DEFINITION_QUESTIONS_COL` to silence this message.
## ℹ See <https://tidyselect.r-lib.org/reference/faq-external-vector.html>.
## This message is displayed once per session.
colnames(bot_definitions_df) <- paste("Q",1:ncol(bot_definitions_df), sep="") #Replaces questions text for IDs

print(paste("Total number of 'bot definition' questions:",ncol(bot_definitions_df),
            "; Total number of participants:",nrow(bot_definitions_df)))
## [1] "Total number of 'bot definition' questions: 32 ; Total number of participants: 111"

We create a diverging plot to get an overview of how participants perceive the presented tools as bots. Note that the answers correspond to the following scale:

  1. Definitely not a bot
  2. More ‘not a bot’ than ‘bot’
  3. It’s unclear from the description
  4. More ‘bot’ than ‘not a bot’
  5. Definitely a bot.
likert_bot_definition <- likert(bot_definitions_df)
plot(likert_bot_definition) + ggtitle("Is the following tool a bot?") + PLOT_THEME

Part 2: How do participants map to different personas?

Goal: Calculate persona scores for each participant.

We manually mapped each question to a persona into a CSV file used for this part of our analysis.

persona_map_df <- read.csv2("persona_mapping.csv", header = T)
persona_map_df

As described in the paper, we assign a persona score between [-2,2] to each participant by (i) rewarding them whenever they give a high likert score to a question of a corresponding persona, and (ii) punishing them for giving high scores for questions of another persona. For instance, a participant that gave high scores for Sam (smartness) questions, but low scores for Alex (autonomy) will have a higher Sam score than another participant that gave high score for Sam but also high score for Alex.

We begin by extracting the questions for each corresponding personas, and the questions that will punish each persona score (i.e., questions of opposing personas).

#Questions that increase each participant's persona scores
qid_alex_rewards    <- persona_map_df %>% filter(Alex == "x") %>% pull(QuestionID) %>% as.character
qid_charlie_rewards <- persona_map_df %>% filter(Charlie == "x") %>% pull(QuestionID) %>% as.character
qid_sam_rewards     <- persona_map_df %>% filter(Sam == "x")  %>% pull(QuestionID) %>% as.character

#Questions that reduce from each participant's persona scores
qid_alex_punish   <- persona_map_df %>% filter(Alex == "") %>% pull(QuestionID) %>% as.character
qid_charlie_punish<- persona_map_df %>% filter(Charlie == "") %>% pull(QuestionID) %>% as.character
qid_sam_punish    <- persona_map_df %>% filter(Sam == "") %>% pull(QuestionID) %>% as.character

Finally, using the responses from all participants, we create individual data frames for each persona’s reward. We do that by converting the likert scales to the corresponding reward score (details in the paper). For each participant we sum their reward scores for each persona.

# Function that convert the likert scores from [1,5] to [-2,2]
# Recall that before we converted the numerical values into factors.
convert_to_reward <- function(x) {case_when(
  x ==1 ~ -2,
  x ==2 ~ -1,
  x ==3 ~ 0,
  x ==4 ~ 1,
  x ==5 ~ 2)}

#Data frames for the reward scores. Adds a Sum column with all scores
alex_reward_df <- bot_definitions_df %>% select(!!qid_alex_rewards) %>% 
  mutate_all(.funs = convert_to_reward) %>% mutate(Sum = rowSums(., na.rm=T))

charlie_reward_df <- bot_definitions_df %>% select(!!qid_charlie_rewards) %>% 
  mutate_all(.funs = convert_to_reward) %>% mutate(Sum = rowSums(., na.rm=T))

sam_reward_df <- bot_definitions_df %>% select(!!qid_sam_rewards) %>% 
  mutate_all(.funs = convert_to_reward) %>% mutate(Sum = rowSums(., na.rm=T))

We do the same for the remaining data frames with punishing scores.

# Function that convert the likert scores from [1,5] to [2,-2]
convert_to_punish <- function(x) {case_when(
  x ==1 ~ 2,
  x ==2 ~ 1,
  x ==3 ~ 0,
  x ==4 ~ -1,
  x ==5 ~ -2)}

#Data frames for the punishing scores.
alex_punish_df   <- bot_definitions_df %>% select(!!qid_alex_punish) %>% 
  mutate_all(.funs = convert_to_punish) %>% mutate(Sum = rowSums(., na.rm=T))

charlie_punish_df<- bot_definitions_df %>% select(!!qid_charlie_punish) %>% 
  mutate_all(.funs = convert_to_punish) %>% mutate(Sum = rowSums(., na.rm=T))

sam_punish_df    <- bot_definitions_df %>% select(!!qid_sam_punish) %>% 
  mutate_all(.funs = convert_to_punish) %>% mutate(Sum = rowSums(., na.rm=T))

Lastly, we aggregate the neutral scores, i.e., participants that answered “Unclear” or that skipped defining one of the tools. This will be used later to obtain the participant’s average score in each persona.

count_neutral_values <- function(row_values){
  #Participant either "Skiped" (NA), or "Unclear" (score == 3)
  neutral_values <- which(is.na(row_values) | row_values == 3)
  return(length(neutral_values))
}

aggregate_neutral_scores <- function(persona_df) {
  modified_df <- persona_df %>%
    rowwise() %>% do(row_values = as_tibble(.)) %>%
    mutate(NeutralValues = count_neutral_values(row_values)) %>% unnest(cols = c(row_values))
  return(modified_df)
}

neutral_scores <- aggregate_neutral_scores(bot_definitions_df) %>% select(NeutralValues)

Using all three information (reward, punishing and neutral scores), we calculate the corresponding persona association score for each participant. This final score is an average obtained by summing the reward and punishing scores and then dividing it by the number of valid scores given by the participant.

Ultimately, each participant is scored to each persona as a value between -2 (i.e., unlikely to belong to that persona) to 2 (i.e., likely to belong to that persona). We will use these values throughout the rest of our analysis to characterize our sample and analyse correlations between participant’s answers. The resulting scores_df dataframe will be the baseline for the remained of our analysis.

num_of_questions <- ncol(bot_definitions_df)
charlie_scores<- (charlie_reward_df$Sum + charlie_punish_df$Sum) / (num_of_questions - neutral_scores$NeutralValues)
sam_scores    <- (sam_reward_df$Sum + sam_punish_df$Sum) / (num_of_questions - neutral_scores$NeutralValues)
alex_scores   <- (alex_reward_df$Sum + alex_punish_df$Sum) / (num_of_questions - neutral_scores$NeutralValues)

scores_df <- data.frame(Charlie = charlie_scores, Alex = alex_scores, Sam = sam_scores)
scores_df

Part 2: How common are different personas?

Goal: Identify whether a specific persona is more prominent among our survey respondants.

The result of the previous steps (scores_df) contains the persona association score for each participant. In order to control for the participants’ varied experience in using bots, we join this data frame with the original survey data to identify which participants actually used bots. This will be stored in a column named BotUsers.

# Constant values used for plotting and filtering of data.
BOT_USAGE_COLUMN    <- 43
BOT_EXPERIENCE_LABEL<- "Bot Experience"
NON_BOT_USER_LABEL  <- "No Bots Used"

#Copy the scores_df into a new datafram to keep this part of the analysis self-contained
personas_df <- scores_df
  
personas_df <- cbind(BotUsers = survey_answers[,BOT_USAGE_COLUMN], scores_df);
personas_df <- personas_df %>% mutate_at(.vars = c("BotUsers"), .funs = function(x){if_else(x == 1,BOT_EXPERIENCE_LABEL,NON_BOT_USER_LABEL)})

We now evaluate the prevalence of each persona in the data. Therefore, we initially only evaluate which score is highest for each participant. If all scores are below a threshold (in our case, threshold = 0) we count the participant as belonging to no specific persona (“None”).

# Function to classify a participant (row) based on its scores.
identify_main_persona <- function(participant){
  # Set threshold for participants belonging to no persona
  threshold <- 0 
  hasPersona <- participant$Charlie > threshold | participant$Alex > threshold | participant$Charlie > threshold
  main_persona <- if_else(hasPersona, colnames(participant)[max.col(participant)], "None")
  return(main_persona)
}

main_personas_df <- personas_df %>% select(-BotUsers) %>% #Temporarily drop the BotUsers column
  rowwise() %>% do(participant = as_tibble(.)) %>%
  mutate(MainPersona=identify_main_persona(participant)) %>% unnest(cols = c(participant))

#Bring back the BotUsers column and count occurencies of each main persona
main_personas_df <- cbind(BotUsers = personas_df$BotUsers, main_personas_df)
personas_frequency <- main_personas_df %>% group_by(BotUsers,MainPersona) %>% tally(name = "Frequency")
personas_frequency

Now we plot the results into simple bar charts to verify if we have a prominent persona among our participants.

Note that the plot below reveals that participants that have used bots fit more under the Charlie persona, where chatting or communicating with a bot is a relevant aspect to define a bot. In constrast, participants that have not used a bot before match the Sam persona where the smartness of a bot is one of its main defining charactersitics. We further discuss the implications of those results in connection to our findings in the paper.

max_frequency <- max(personas_frequency$Frequency) #Saves highest value to set plot's scales' limits.
persona_palette <- c("None" = "#AED9D6","Sam" = "#5BB4AC", "Charlie" = "#985f99", "Alex" = "#D9B466")

ggplot(personas_frequency, aes(x= MainPersona, y=Frequency, fill=MainPersona))+
  geom_bar(width = 1, stat = "identity", colour = "black") + coord_flip() +
  scale_y_continuous(limits = c(0, max_frequency), breaks = seq(0,max_frequency, by=2)) +
  PLOT_THEME + scale_fill_manual(values = persona_palette) +
  theme(legend.position = "None") +
  facet_grid(personas_frequency$BotUsers ~ .)

Part 3: How are the different personas distributed and correlated?

Goal: Understand the distribution of personas in our data and investigate possible correlation between personas.

Finally, we also want to investigate how the persona scores are actually distributed. Based on the personas_df dataframe from Part 2, we will use a density plot to see whether a specific persona is more prominent in our data.

Similar to our previous step, we divide our sample between participants that have used bots and those that have not. However, unlike our previous step, we do not associate a persona to a specific persona based on its highest scores. Instead, we analyse each participant based on all personas.

#Converts to a long table format to plot density.
melted_df <- personas_df %>% gather(key="Personas", value = "Scores", -c(BotUsers))
ggplot(melted_df, aes(x = Scores, fill = Personas)) + geom_density() +
  scale_fill_manual(values = persona_palette) + xlim(-2,2) + PLOT_THEME + 
  theme(legend.position = "None") +
  facet_grid(rows = vars(BotUsers), cols = vars(Personas))

We can also investigate the correlation between association scores (not documented in the paper). We observe that Sam and Alex are strongly correlated, while Charlie and Sam/Alex are not.

cor_charlie_sam <- ggplot(personas_df, aes(x=Charlie, y=Sam)) + geom_point() + geom_smooth(method=lm) +
  facet_grid( rows = vars(BotUsers) ) + PLOT_THEME +
  scale_y_continuous(limits = c(-2,2), breaks=seq(-2,2,by=0.5)) +
  scale_x_continuous(limits = c(-2,2), breaks=seq(-2,2,by=0.5)) 

cor_charlie_alex <- ggplot(personas_df, aes(x=Alex, y=Charlie)) + geom_point()+ geom_smooth(method=lm) + 
  facet_grid( rows = vars(BotUsers) ) + PLOT_THEME +
  scale_y_continuous(limits = c(-2,2), breaks=seq(-2,2,by=0.5)) +
  scale_x_continuous(limits = c(-2,2), breaks=seq(-2,2,by=0.5)) 

cor_sam_alex<- ggplot(personas_df, aes(x=Sam, y=Alex)) + geom_point() + geom_smooth(method=lm) +
  facet_grid( rows = vars(BotUsers) ) + PLOT_THEME +
  scale_y_continuous(limits = c(-2,2), breaks=seq(-2,2,by=0.5)) +
  scale_x_continuous(limits = c(-2,2), breaks=seq(-2,2,by=0.5)) 

grid.arrange(cor_charlie_sam,cor_charlie_alex,cor_sam_alex, ncol = 3)
## `geom_smooth()` using formula 'y ~ x'
## `geom_smooth()` using formula 'y ~ x'
## `geom_smooth()` using formula 'y ~ x'

Part 4: What are the main reasons to use bots?

Goal: Get an overview of how participants’ rate different reasons to use a bot in software development.

We also asked participants (that have used a bot in software development) to rate the reasons for using bots. The list of reasons used on our survey were obtained from our interview data. Participants rated each reason according to the following likert scale:

  1. Disagree
  2. Slightly disagree
  3. Unsure
  4. Slightly agree
  5. Agree

Below, we extract that data from our survey data and create diverging plots to see which are the main reasons that motivate participants to use Bots in software development.

# Gets columns where participants scored the reasons and removes participants that never used a bot.
BOT_REASONS_COL <- c(45:54)
bot_reasons_df <- survey_answers %>% select(BOT_REASONS_COL) %>% 
  filter_all(any_vars(!is.na(.))) %>% mutate_all(factor)
## Note: Using an external vector in selections is ambiguous.
## ℹ Use `all_of(BOT_REASONS_COL)` instead of `BOT_REASONS_COL` to silence this message.
## ℹ See <https://tidyselect.r-lib.org/reference/faq-external-vector.html>.
## This message is displayed once per session.
# Similar to the questions text, we summarize the reasons to better plot the data. 
# The complete text for each reason can be found in the CSV data.
short_descriptions <- c("Reduce Boring Tasks", "Information Integration","Health Monitoring",
                        "Integration Quality","Common Information Gateway", "ChatOps", 
                        "Automatic Resource Cleanup", "Analyse Big Data","Problem Notifications", 
                        "Maintain Dependencies")
colnames(bot_reasons_df) <- short_descriptions

likert_reasons <- likert(bot_reasons_df)
plot(likert_reasons) + ggtitle("Reasons to Use a Bot") + PLOT_THEME