---
output:
  pdf_document: default
  html_document: default
---
# Analysis of the Carpentries' Long-Term Feedback Survey
Authors: [Kari L. Jordan](https://github.com/kariljordan), [Ben Marwick](https://github.com/benmarwick), [Belinda Weaver](https://github.com/weaverbel), [Naupaka Zimmerman](https://github.com/naupaka), [Jason Williams](https://github.com/JasonJWilliamsNY), [Tracy Teal](https://github.com/tracykteal), [Erin Becker](https://github.com/ErinBecker), [Jonah Duckles](https://github.com/jduckles), [Beth Duckles](https://github.com/bduckles), [Elizabeth Wickes](https://github.com/elliewix)  

`r format(Sys.Date(), "%B %Y")`

```{r setup, include=FALSE}
library(knitr)
opts_chunk$set(echo = FALSE,
               message = FALSE,
               warning = FALSE)
library(tidyverse)
library(ggplot2)
library(dplyr)
library(tidyr)
library(readr)
library(purrr)
library(tibble)
library(DBI)
library(ggmap)
library(likert)
library(mapproj)
library(RColorBrewer)
library(srvyr)
```


```{r include=FALSE}
data <- readr::read_csv("https://raw.githubusercontent.com/carpentries/assessment/master/carpentries/long-term-survey/data.csv")
```


```{r include = FALSE}
# Highlights in the data
# Use prop.table() to calculate percentages on a single column.

#Change in Confidence
Confidence_Change <- round(prop.table(table(data$`Change-In-Confidence`)) * 100)

#Programming Usage Before Workshop
Programming_Usage_Before <- round(prop.table(table(data$`Programming-Usage-Before-Workshop`)) * 100)

#Programming Usage Since Workshop
Programming_Usage_Since <- round(prop.table(table(data$`Programming-Usage-Since-Workshop`)) * 100)

#Motivated to Seek Knowledge
Motivated_Seek_Knowledge <- round(prop.table(table(data$`Motivated-Seek-Knowledge`))* 100)

#Made Analyses Reproducible
Made_Analyses_Reproducible <- round(prop.table(table(data$`Made-Analyses-Reproducible`))* 100)

#Improved Coding Practices
Improved_Coding_Practices <- round(prop.table(table(data$`Improved Coding Practices`))* 100)

#Gained Confidence with Data
Gained_Confidence_Data <- round(prop.table(table(data$`Gained-Confidence-With-Data`))* 100)
Gained_Confidence_Data

#Recommend our Workshops
Recommended <- round(prop.table(table(data$Recommended))* 100)

# Function that makes a table of counts and percentages
# question_n is set by default to the number of respondents in the survey - 476. This value may have to be set by question. 
question_n <- nrow(data)

tally_and_perc <- function(df, colname, na.rm = FALSE, question_n){
  quo_colname <- enquo(colname)

  df %>% 
    group_by(!!quo_colname) %>% 
    tally() %>% 
    filter(if_else(rep(na.rm, nrow(.)),
                  !is.na(!!quo_colname),
                  as.logical(rep(1, nrow(.))))) %>% 
    mutate(`%` = round(n / question_n * 100, 1)) 
}

# function to compute number of non-NA responses to a question

n_responses_to_the_question <- function(df, from_colname, to_colname) {
  
  quo_from_colname <- enquo(from_colname)
  quo_to_colname <- enquo(to_colname)
  
  rowsums <- 
df %>%
  select(UQ(quo_from_colname):UQ(quo_to_colname)) %>% 
  # check that each row has a value for at least one col
  # convert to numeric, if NA, then put a zero, otherwise 1
  map_df(., ~ifelse(is.na(.x), 0, 1)) %>% 
  # sum to see if any rows have no reponses
  rowSums() 

# for all behaviours
idx <- ifelse(rowsums == 0, FALSE, TRUE)
sum(idx)
}

```
# Introduction

Software Carpentry is a worldwide volunteer organization whose mission is to make scientists more productive, and their findings more reliable, by teaching them foundational computing skills. Established in 1998, it runs short, intensive workshops that cover task automation using the Unix shell, structured programming in Python and R, and version control using tools such as Git. Its sibling organization, Data Carpentry, teaches foundational data science skills. To date, the majority of Software and Data Carpentry workshops have been run in the United States, Canada and the UK. However, there is growing interest elsewhere, and there are active ‘Carpentries’ communities in Australia, New Zealand, South Africa, the Netherlands, Norway and in other countries in Africa and in Central and South America.

While most workshops are favorably assessed by learners at the time of delivery, no systematic, long-term follow-up study has previously been done on the efficacy of the training delivered, nor of the short or longer term impact that such training might have had on learners’ work practices, further skills acquisition, or subsequent career paths. There has also been no useful demographic profiling of learners.

Why should that matter? 

We are outcome-driven organizations interested in continually improving the workshop experience for both our learners and our instructors. Additionally, we are largely volunteers. In order to continue our valuable work of teaching skills to researchers, we need supportive funding either from grant-making bodies or from member institutions, or - ideally - from both. Our case for funding is strengthened if we can provide impartial evidence that proves our workshops have the outcomes we claim. While it is wonderful to have amassed what must now amount to container-sized loads of positive 'sticky note' workshop feedback over the years, funders generally require more solid evidence of achievement before they are willing to cut a check. Establishing value is not just important for funding though: it is important for our growing community of trainers, instructors, lesson maintainers and helpers, all of whom volunteer their time because they believe in the importance of what we do. We owe it to them to show that their time is not wasted - that we are genuinely furthering the cause of efficiently organized, reproducible science. Lastly, we need to demonstrate to our learners that precious time carved out to master computational and data science skills will pay off many times over in time saved further down the track.
<!-- 

BM: Where do the numbersin this paragraph come from? Better to have them as inline variables rather than hard-coded in the text.
KJ: I removed the numbers because we don't necessarily need them in this report.
-->
Data Carpentry's post-workshop survey [results](https://carpentries.github.io/assessment/data-carpentry/postworkshop/report.html) tell us that our respondents are either enthusiastically or very involved in our workshops, they learn a great deal of practical knowledge, and agree that they can immediately apply what they learned at the workshop. Our interest is in establishing, long-term, what impact workshops are having on learners’ confidence in the skills they were taught. 

Therefore, through the generosity of the Moore Data Driven Discovery Initiative, assessment work has been undertaken across both Carpentries to help build an evidence base to complement the large body of existing anecdotal evidence that Carpentry-style training is both useful and effective in improving researchers’ work practices.

To gather the required evidence, Data and Software Carpentry launched a [long-term assessment survey](https://www.surveymonkey.com/r/carpentrieslongtermassessment) in March 2017. We first engaged in a community consultation to determine how best to design and word this survey. The responses from that consultation then guided the development of our long-term assessment strategy. You can read more about the consultation process in this [blog post](http://www.datacarpentry.org/blog/long-term-assessment-strategy/). 

The main goal of the resulting survey was to ask our learners to describe concrete changes they had implemented to their research practices as a result of completing a Carpentries workshop. We also asked whether they now had greater confidence in the tools they had been taught, and whether they had progressed in their careers as a result. The inclusion of multiple choice questions around programming in R or Python helped make the evidence of training efficacy more concrete, comparable and measurable, taking it out of the realm of ‘opinion’ or ‘feeling’, which, while interesting, is not as robust or reliable a marker of success as demonstrable evidence. For instance, ‘I can write a FOR loop to rename and move a batch of files’ is a much more reliable metric of achievement than ‘I can use the shell’.

All of the data collected in this survey was self-reported. It should be noted that there are disadvantages to self-reported surveys. For one, respondents may exaggerate their achievements. Additionally, a respondent’s state of mind while taking the survey may affect her/his answers. Survey results can potentially be biased because those feeling most positive are also those more likely to respond, while learners whose experience was less positive – or even negative - may not bother to answer. To account for this, we compared the results of the long-term survey with that of Data Carpentry’s [post-workshop survey results](https://carpentries.github.io/assessment/data-carpentry/postworkshop/report.html) and Software Carpentry's [post-workshop survey results](https://carpentries.github.io/assessment/software-carpentry/postreport.html). We found consistent patterns of increased confidence and self-efficacy in our learners.

The survey questions and the data used in this analysis are located in the [carpentries/assessment](https://github.com/carpentries/assessment) repo on GitHub. We have already received several pull requests from community members interested in this data. Feel free to use the data and [tell us](mailto: kariljordan@carpentries.org) about your findings.

This analysis includes `r question_n` observations. Not all respondents answered every one of the 26 questions.

# Highlights

The long-term survey assessed confidence, motivation, and other outcomes **more than six months** after respondents attended a Carpentry workshop. Provided below are a few highlights from the data.

+ `r Confidence_Change[3]`% of respondents reported being more confident **in the tools** that were covered during their workshop compared to before the workshop. 
+ `r Made_Analyses_Reproducible[1] + Made_Analyses_Reproducible[4]`% of respondents have **made their analyses more reproducible** as a result of completing a workshop.
+ `r Gained_Confidence_Data[1] + Gained_Confidence_Data[4]`% of respondents have gained confidence in **working with data** as a result of completing a workshop.
+ `r Recommended[3]`% of respondents have **recommended our workshops** to a friend or colleague.

# Respondent Demographics
```{r include=FALSE}
# Ben's tip to use 'gather' to go from wide to long format
# Responses are in columns 'Field' through 'Column12'

n_responses_to_field_question <- 
  n_responses_to_the_question(data, 
                              from_colname = Field, 
                              to_colname = Column12)
                              
field_perc <- 
data %>%
  select(Field:Column12) %>% 
  gather(col, field_perc) %>% 
  group_by(field_perc) %>% 
  tally_and_perc(field_perc, 
                 na.rm = TRUE, 
                 question_n = n_responses_to_field_question) %>%
  filter(!is.na(field_perc)) %>% 
  arrange(desc(n)) %>%
  rename(Field = field_perc) 
```

Carpentry learners represent a wide range of disciplines ranging from the sciences to engineering. Respondents were asked to indicate their field of research, work, or study by checking all that apply from a list of various disciplines. A breakdown of their responses is provided in the table below. Many of the respondents work in Life Sciences.

```{r}
kable(field_perc, 
      format = "markdown", 
      digits = getOption("digits"), 
      row.names = NA, 
      col.names = NA, 
    caption = NULL, 
    escape = TRUE)
```

```{r}
# Code for Status of Respondents
position = c("Undergraduate Student", "Graduate Student", "Postdoc", "Faculty", "Industry", "Academic Research Staff", "Other Academic Staff", "Other (please specify)")
position = factor(position)

data$Position = factor(data$Position, levels = position)

data_position_tally <- 
  data %>% 
  group_by(Position) %>% 
  tally() %>% 
  filter(!is.na(Position)) %>% 
  mutate(perc = round(n/sum(n) * 100, 0))

# Use the line below to include a table

#kable(data_position_tally, format = "markdown", row.names = FALSE, col.names = c("Position", "n"))

ggplot(data_position_tally, 
       aes(Position, y = 100 * (n/sum(n)),
           n)) +
  geom_bar(stat = "identity", fill="orange") +
  geom_text(aes(label=n), size= 4, vjust=-0.25) +
  scale_x_discrete(labels = function(x) lapply(strwrap(x, width = 10, simplify = FALSE), paste, collapse="\n")) +
  theme_classic() +
  xlab("Status") +
  ylab("% Respondents") +
  ggtitle("Majority of Respondents were Graduate Students") +
  theme(plot.title = element_text(hjust = 0.5)) +
  theme_classic(base_size = 14)
ggsave("figures/status.png")
```

```{r include=FALSE}
# Use code below for in-line text.
Graduate_Student <- 
data_position_tally %>% 
  filter(Position == 'Graduate Student') %>% 
  pull(perc)
```

Carpentries workshops are open to individuals from all backgrounds and fields. Attendees vary from students (undergraduate and graduate) and faculty to staff and persons working in industry. `r Graduate_Student`% of our respondents were graduate students.

Below is a breakdown of our respondents by the country in which they attended a Carpentries workshop.
 
```{r}
# Plot includes Ben's tip to add a percent column to the data_country_tally data frame 
# before it goes into the ggplot() function. Then you should be able to use reorder on that column name in the ggplot() function
data_country_tally <-
data %>%
  group_by(Country) %>%
  tally(sort = TRUE) %>%
  mutate(perc = round(100 * (n/sum(n)), 1)) %>% # add the % col
  filter(!is.na(Country)) %>%
  arrange(desc(n))

ggplot(data_country_tally,
       aes(reorder(Country, perc),
           perc)) +
  geom_bar(stat = "identity", fill = "orange") +
  theme_classic() +
  xlab("") +
  ylab("Percentage of all participants") +
  coord_flip() +
  ggtitle("Carpentry Workshop Respondents by Country") +
  theme(plot.title = element_text(hjust = 0.5)) +
  theme_bw(base_size = 14)
ggsave("figures/percent_by_country.png")

# Use code below for in-line text.
USA <- data_country_tally[data_country_tally$Country == 'USA', ]$n
USA_perc <- filter(data_country_tally, Country == "USA") %>% pull(perc)
Canada <- data_country_tally[data_country_tally$Country == 'Canada', ]$n
Canada_perc <- filter(data_country_tally, Country == "Canada") %>% pull(perc)
UK <- data_country_tally[data_country_tally$Country == 'UK', ]$n
UK_perc <-  filter(data_country_tally, Country == "UK") %>% pull(perc)
Australia <- data_country_tally[data_country_tally$Country == 'Australia', ]$n
Australia_perc <-  filter(data_country_tally, Country == "Australia") %>% pull(perc)

```

A large portion of Carpentry learners responding to the survey attended a workshop in the United States (`r USA_perc`%), followed by Canada (`r Canada_perc`%) the UK (`r UK_perc`%), and Australia (`r Australia_perc`%).

```{r include=FALSE}
# If interested in seeing the open-ended responses for Position: Other, run this code.
data_position_other_tally <- 
  data %>% 
  group_by(`Position-Other`) %>% 
  tally() %>% 
  filter(!is.na(`Position-Other`))
kable(data_position_other_tally, format = "markdown", row.names = FALSE, col.names = c("Position", "# Respondents"))
```

The Carpentries constantly strive to improve our workshop content and operations, which means a workshop run today might be different in some ways from a workshop run six months ago or run last year. If we know how many workshops respondents attended, and how long it has been since they completed that workshop or workshops, we can take those changes into account when we assess learner responses. Locating workshops temporally allows us to account for spikes in data trends that may be a result of changes in our workshop operations.

```{r include=FALSE}
#Code for in-line text
#How long ago did respondents attend a workshop?
How_Long_Ago <- round(prop.table(table(data$`How-Long-Ago`))* 100)
How_Long_Ago

#How many workshops have respondents attended?
How_Many_Workshops <- round(prop.table(table(data$`How-Many-Workshops`))* 100)
How_Many_Workshops
```

`r How_Long_Ago[3]`% of respondents participated in a Carpentry workshop more than one year ago, and `r How_Many_Workshops[1]`% of respondents have attended only one Carpentry workshop. 

```{r include=FALSE}
Carpentry_Involvement <- 
data %>%
  select(`Carpentry-Involvement`:Column51) %>% 
  gather(col, Carpentry_Involvement) %>% 
  group_by(Carpentry_Involvement) %>% 
  tally() %>% 
  filter(!is.na(Carpentry_Involvement)) %>% 
  arrange(desc(n)) %>% 
  rename(`Involvement Since Attending a Carpentry Workshop` = Carpentry_Involvement)

Carpentry_Involvement[1,2]
```

The response rate from learners who attended a workshop more than a year ago speaks to their level of involvement with the Carpentries. `r Carpentry_Involvement[1,2]` of the survey respondents are subscribed to the Carpentries newsletter, indicating their ongoing interest in our work. In survey research, it can be difficult to collect responses from participants a whole year after their involvement in an event. It is great to see that learners feel positively enough about their experience to, firstly, be receptive to our email communication and, secondly, to have taken the time to complete the survey in full. As mentioned in the introduction, this could be a potential disadvantage to self-reported data, as those who may have negative experiences may not have completed the survey.

```{r}
 data_howlong_tally <- 
  data %>% 
  group_by(`How-Long-Ago`) %>% 
  tally() %>% 
  filter(!is.na(`How-Long-Ago`)) 

# Use the code below to include a table of how long ago respondents attended a workshop
# kable(data_howlong_tally, format = "markdown", row.names = FALSE, col.names = c("Time", "n"), caption = "How Recently Did Respondents Complete a Workshop?")

ggplot(data_howlong_tally, 
       aes(`How-Long-Ago`, y = 100 * (n/sum(n)),
           n)) +
  geom_bar(stat = "identity", fill="orange") +
  geom_text(aes(label=n), size= 4 , vjust=-0.25) +
  scale_x_discrete(labels = function(x) lapply(strwrap(x, width = 10, simplify = FALSE), paste, collapse="\n")) +
  theme_classic() +
  xlab("Time Since Attending Carpentry Workshop") +
  ylab("% Respondents") +
  ggtitle("Majority Attended Workshop Over a Year Ago") +
  theme(plot.title = element_text(hjust = 0.5)) +
  theme_classic(base_size = 14)
ggsave("figures/time_since_attending_workshop.png")
```

```{r}
data_workshops_tally <- 
  data %>% 
  group_by(`How-Many-Workshops`) %>% 
  tally() %>% 
  filter(!is.na(`How-Many-Workshops`))

# Use the code below to include a table of the number of workshops respondents attended
#kable(data_workshops_tally, format = "markdown", row.names = FALSE, col.names = c("# Wrkshps", "n"), caption = "Number of Workshops Attended")

ggplot(data_workshops_tally, 
       aes(`How-Many-Workshops`, y = 100 * (n/sum(n)),
           n)) +
  geom_bar(stat = "identity", fill="orange") +
  geom_text(aes(label=n), size= 4, vjust=-0.25) +
  scale_x_discrete(labels = function(x) lapply(strwrap(x, width = 10, simplify = FALSE), paste, collapse="\n")) +
  theme_classic() +
  xlab("# Workshops Attended") +
  ylab("% Respondents") +
  ggtitle("Majority Attended One Carpentry Workshop") +
  theme(plot.title = element_text(hjust = 0.5)) +
  theme_classic(base_size = 14)
ggsave("figures/number_workshops_attended.png")
```

### Workshop Content

Data Carpentry's [lessons](http://www.datacarpentry.org/lessons/) include data organization in spreadsheets, data cleaning with OpenRefine, data management with SQL, and data analysis and visualization in R and Python.

Software Carpentry's [lessons](https://software-carpentry.org) include the Unix Shell, version control with Git and Mercurial, programming with Python, R, and MATLAB, databases with SQL, and Automation and Make. Provided below is a breakdown of the tools respondents identified as being taught in the workshop they attended.

```{r}
# Responses are in columns 'Content' through 'Column22'
# Ben's tip to use 'gather' to go from wide to long format
workshop_tools <- 
 data %>%
 select(`Content`:Column22) %>% 
 gather(col, workshop_tools) %>% 
 group_by(workshop_tools) %>% 
 tally() %>% 
 filter(!is.na(workshop_tools)) %>% 
 arrange(desc(n)) %>%
 rename(`Content` = workshop_tools)

# Use the line below to provide a table of the tools covered. 
# Respondents were asked to check all that apply.
# kable(workshop_tools, caption = "Tools Covered") 

# Ben's tip to use in-line text
# This code produces results to use in the text of the report
Git <- workshop_tools[workshop_tools$Content == 'Git', ]$n
Python <- workshop_tools[workshop_tools$Content == 'Python', ]$n
Unix_Shell <- workshop_tools[workshop_tools$Content == 'Unix Shell', ]$n
R <- workshop_tools[workshop_tools$Content == 'R', ]$n
SQL <- workshop_tools[workshop_tools$Content == 'SQL', ]$n
OpenRefine <- workshop_tools[workshop_tools$Content == 'OpenRefine', ]$n
Spreadsheets <- workshop_tools[workshop_tools$Content == 'Spreadsheets', ]$n
Cloud_Computing <- workshop_tools[workshop_tools$Content == 'Cloud Computing', ]$n
MATLAB <-  workshop_tools[workshop_tools$Content == 'MATLAB', ]$n
Mercurial <-  workshop_tools[workshop_tools$Content == 'Mercurial', ]$n
```

A large majority of respondents learned Git (n = `r Git`), Python (n = `r Python`), and the Unix Shell (n = `r Unix_Shell`). On the low end were spreadsheets (n = `r Spreadsheets`), cloud computing (n = `r Cloud_Computing`), MATLAB (n = `r MATLAB`), and Mercurial (n = `r Mercurial`). The fact that OpenRefine, spreadsheets, and cloud computing were on the low end is an indicator that the majority of our respondents attended a Software Carpentry workshop. This is most likely because Data Carpentry is a newer organization, starting in 2014, so there have been fewer workshops held.

```{r}
# The code segment below from Ben Marwick will show the most combinations of tools covered in our workshops.

n_responses_to_tools_question <- 
  n_responses_to_the_question(data, 
                              from_colname = Content, 
                              to_colname = Column22)

tools_cols <- 
data %>%
  select(Content:Column22)

list_of_tools_per_person <- list()
for(i in seq_len(nrow(tools_cols))) {
  ii <- quo(i)
  
  list_of_tools_per_person[[i]] <- 
  tools_cols %>% 
    slice(!!ii) %>% 
          c(., recursive=TRUE) %>%
          unname %>% 
    na.omit() %>% 
    as.vector()
}

# The code segment below from Ben Marwick will get the tally of combinations of tools and produce a table.
tool_combs <- 
purrr::map_chr(list_of_tools_per_person,
               ~paste0(.x, collapse = " "))

tool_combs_df <- 
tool_combs %>% 
  as_data_frame() %>% 
  group_by(value) %>% 
  tally() %>% 
  mutate(`%` = round(n / sum(n) * 100, 1)) %>% 
  arrange(desc(n)) %>% 
  filter(value != "")

# Top combinations as entered by respondants
colnames <- c("Frequency of Tools Covered", "n", "%")
kable(tool_combs_df[1:10, ], row.names = NA, col.names = colnames, caption = "Combination of Tools Covered")
```

In examining which combinations of tools stood out in the data, we can see from the matrix below that Git was frequently taught alongside Python, R, and/or the Unix Shell. Additionally, SQL was often taught with Git and/or the Unix Shell. OpenRefine, Spreadsheets, and Cloud Computing were on the low end, a clear indicator that the majority of survey respondents attended a Software Carpentry workshop, rather than a Data Carpentry workshop. 

```{r}
# The code segment below from Ben Marwick will give a matrix of the combinations of tools used.
m <- as.matrix(tools_cols) 
# The unique values in the matrix
vals <- sort(unique(as.vector(m)))

# Rearrange the data so that each value is a column
bigm <- t(apply(m, 1, function(row) match(vals, row, nomatch=0)))
colnames(bigm) <- vals

# Count the co-occurences of each value (diagonal is total number of rows with that value)
tool_co_occurences  <- as.data.frame(crossprod(bigm>0))
kable(tool_co_occurences, row.names = TRUE, caption = "Matrix of Common Tools Covered")
```

```{r include=FALSE}
# Data for in-line text on programming usage
Programming_Usage_Before <- round(prop.table(table(data$`Programming-Usage-Before-Workshop`)) * 100)
Programming_Usage_Before

#Programming Usage Since Workshop
Programming_Usage_Since <- round(prop.table(table(data$`Programming-Usage-Since-Workshop`)) * 100)
Programming_Usage_Since
```

### Programming Usage Pre- and Post Workshop

Understanding respondents’ programming usage both before and after attending a Carpentries workshop was one goal of this assessment study. Our hope is that the workshop learners attended favorably influenced their use of the programming tools they learned.

`r Programming_Usage_Before[2]`% of the learners who responded to our survey had not been using the tools covered in their Carpentries workshop before they attended the workshop. This decreased to `r Programming_Usage_Since[2]`% post-workshop.

The plot below is a comparison of respondents' usage of the tools covered in their workshop both pre- and post-workshop.

```{r include=FALSE}
# The code below can be used to get a table and plot of the number of respondents for programming usage
# pre workshop
# Naupaka's tip so pre- and post-responses match
data_paired_plot <- data
levels(data$`Programming-Usage-Before-Workshop`)[2] <- "I have not been using tools like these."
 
# Programming Usage Pre-Carpentry Workshop [Absolute Plot]
programming = c("I had not been using tools like these.", "Less than once per year.", "Several times per year.", "Monthly.", "Weekly.", "Daily.")
programming = factor(programming)
 
  data_usage_tally <- 
   data %>% 
   group_by(`Programming-Usage-Before-Workshop`) %>% 
   tally() %>% 
   filter(!is.na(`Programming-Usage-Before-Workshop`)) 

  kable(data_usage_tally, caption = "Prior Programming of Respondents")
 
  ggplot(data_usage_tally, 
        aes(`Programming-Usage-Before-Workshop`, n)) +
   geom_bar(stat = "identity", fill="orange") +
   geom_text(aes(label=n), size= 4) +
   scale_x_discrete(labels = function(x) lapply(strwrap(x, width = 10, simplify = FALSE), paste, collapse="\n")) +
   theme_classic() +
   xlab("Programming Usage") +
   ylab("# Respondents") +
   ggtitle("Programming Usage Pre-Workshop Varies") +
   theme(plot.title = element_text(hjust = 0.5)) +
   theme_classic(base_size = 14)
```

```{r include=FALSE}
# The code below is for pre-workshop programming usage reported as a percentage.
# Programming Pre-Carpentry Workshop [Percentage Plot]
 data %>%
  select(`Programming-Usage-Before-Workshop`) %>%
   group_by(`Programming-Usage-Before-Workshop`) %>%
   tally() %>%
   filter(!is.na(`Programming-Usage-Before-Workshop`)) %>%
   mutate(`Programming-Usage-Before-Workshop` =     factor(`Programming-Usage-Before-Workshop`, levels = programming)) %>%
   ggplot(aes(x = `Programming-Usage-Before-Workshop`, y = 100 * (n/sum(n)))) +
     geom_bar(stat = "identity", position = "dodge", fill = "orange") +
     geom_text(aes(label=n), size= 4) +
     scale_x_discrete(labels = function(x) lapply(strwrap(x,
                                                          width = 10,
                                                          simplify = FALSE),
                                                  paste,
                                                  collapse = "\n")) +
     theme_classic() +
     xlab("Programming Usage") +
     ylab("% respondents") +
     ggtitle("Programming Usage Pre-Workshop Varies") +
     theme(plot.title = element_text(hjust = 0.5)) +
    theme_classic(base_size = 14)
```    

```{r include=FALSE}
# The code below is for post-workshop programming usage (count)
# Programming Usage Post-Carpentry Workshop [Absolute Plot]
  programming = c("I have not been using tools like these.", "Less than once per year.", "Several times per year.", "Monthly.", "Weekly.", "Daily.")
  programming = factor(programming)
 
 data$`Programming-Usage-Since-Workshop` = factor(data$`Programming-Usage-Since-Workshop`, levels = programming)
 
  data_usage_tally <- 
   data %>% 
   group_by(`Programming-Usage-Since-Workshop`) %>% 
   tally() %>% 
   filter(!is.na(`Programming-Usage-Since-Workshop`)) 
 
 kable(data_usage_tally)
 
  ggplot(data_usage_tally, 
        aes(`Programming-Usage-Since-Workshop`, n)) +
   geom_bar(stat = "identity", fill="orange") +
   geom_text(aes(label=n), size= 4) +
   scale_x_discrete(labels = function(x) lapply(strwrap(x, width = 10, simplify = FALSE), paste, collapse="\n")) +
   theme_classic() +
   xlab("Programming Usage") +
   ylab("# Respondents") +
   ggtitle("Respondents' Programming Usage Increased") +
   theme(plot.title = element_text(hjust = 0.5)) +
   theme_classic(base_size = 14)
```

```{r include=FALSE}
# The code below is for post-workshop programming usage as a percent
#Programming Usage Post-Carpentry Workshop [Percentage Plot]
data %>%
select(`Programming-Usage-Since-Workshop`) %>%
group_by(`Programming-Usage-Since-Workshop`) %>%
tally() %>%
filter(!is.na(`Programming-Usage-Since-Workshop`)) %>%
mutate(`Programming-Usage-Since-Workshop` = factor(`Programming-Usage-Since-Workshop`, levels = programming)) %>%
ggplot(aes(x = `Programming-Usage-Since-Workshop`, y = 100 * (n/sum(n)))) +
geom_bar(stat = "identity", position = "dodge", fill = "orange", na.rm = TRUE ) +
geom_text(aes(label=n), size= 4) + # Adds count to top of bar
scale_x_discrete(labels = function(x) lapply(strwrap(x,
                                                         width = 10,
                                                         simplify = FALSE),
                                                 paste,
                                                 collapse = "\n")) +
    theme_classic() +
    xlab("") +
    ylab("% respondents") +
    ggtitle("Respondents' Programming Usage Increased") +
    theme(plot.title = element_text(hjust = 0.5)) +
    theme_classic(base_size = 14)
```

```{r fig.cap=paste("Synopsis: Respondents were asked how often they use programing languages (R, Python, etc.), databases (Access, SQL, etc.), version control software and/or the shell before completing a Carpentries workshop, and since completing a Carpentries workshop.")}
# The code below is for Pre/Post plots of programming usage
# It includes tips from Ben and Naupaka 
# Make the unique values the same
data$`Programming-Usage-Before-Workshop` <- 
  gsub("I had not been using tools like these.",
       "I have not been using tools like these.", 
       data$`Programming-Usage-Before-Workshop`)

data$`Programming-Usage-Before-Workshop` <- 
  factor(data$`Programming-Usage-Before-Workshop`, 
         levels = programming)

pre_and_post_usage <- 
data %>%
  select(`Programming-Usage-Before-Workshop`, 
          `Programming-Usage-Since-Workshop`) %>%
  gather() %>%
  group_by(key, value) %>%
  tally() %>%
  mutate( perc = 100 * (n/sum(n))) %>%
  filter(!is.na(key),
         !is.na(value)) 

  ggplot(pre_and_post_usage, 
         aes(x = factor(value, 
                        levels = programming), 
             y = perc, 
             fill = key)) +
    geom_bar(stat = "identity", 
             position = "dodge") +
    geom_text(aes(label=n), size= 4, vjust=-0.25, position = position_dodge(width = 1)) +
    scale_x_discrete(labels = function(x) lapply(strwrap(x,
                                                         width = 10,
                                                         simplify = FALSE),
                                                 paste,
                                                 collapse = "\n")) +
    theme_classic() +
    xlab("Programming Usage") +
    ylab("% Respondents") +
    scale_fill_discrete(name = "",
                        labels = c("Before Workshop", "After Workshop")) +
    ggtitle("Respondents' Programming Usage Increased") +
    theme(plot.title = element_text(hjust = 0.5)) +
    theme_classic(base_size = 14)
  ggsave("figures/change_in_programming_usage.png") 
  
# Get some % change values to use in the text
delta_pre_have_not_been_using <- 
  pre_and_post_usage %>% 
  filter(grepl("have", value)) %>% 
  spread(key, value) 

delta_pre_have_not_been_using <- 
round(delta_pre_have_not_been_using$perc[2] - 
  delta_pre_have_not_been_using$perc[1], 1)

delta_pre_daily_using <- 
  pre_and_post_usage %>% 
  filter(grepl("Daily", value)) %>% 
  spread(key, value) 

delta_pre_daily_using <- 
round(delta_pre_daily_using$perc[2] - 
  delta_pre_daily_using$perc[1], 1)
```

The most compelling (and pleasing) change in responses was a decline in the percentage of respondents who 'have not been using these tools' (`r -delta_pre_have_not_been_using`%), and an increase in the percentage of those who now use the tools on a daily basis (`r delta_pre_daily_using`%) at least six months after they attended a Carpentry workshop.

```{r}
# The code below from Ben provides a chi-square plot of the pre/post residuals
pre_and_post_test <- 
pre_and_post_usage %>% 
  select(-perc) %>% 
  spread(value, n) %>% 
  ungroup()  

# chi-sq test
pre_and_post_test_result <- 
  chisq.test(pre_and_post_test[ , !names(pre_and_post_test) == 'key'])

# standardised residuals
stdres <- data.frame(t(pre_and_post_test_result$stdres))
names(stdres) <- pre_and_post_test$key
stdres$freq <- row.names(stdres)

# just show post-workshop
stdres <- stdres[, c(2,3)]

names(stdres) <- rev(c("Frequency", "Residual"))

# large positive residuals means there were more xxx than the hypothesis of independence predicts. Where are our large +ve residuals?

# Contribution in percentage (%)
# The contribution (in %) of a given cell to the total Chi-square score is calculated as follows:
contrib <- 100 * pre_and_post_test_result$residuals^2 / pre_and_post_test_result$statistic
# scale 0 to 1 to use as alpha
range0to1 <- function(x){(x-min(x))/(max(x)-min(x))}
# reorder to match order of programming factor
contrib_0_to_1 <- as.vector(range0to1(contrib))[c(3,5,9,7,11,1)]

# colour +ve and -ve values
# http://stackoverflow.com/a/12910865/1036500
stdres$sign = ifelse(stdres$Residual >= 0, 
                          "positive", 
                          "negative")

# get the categories in a sensible order
ggplot(stdres,
       aes(factor(Frequency, 
                  levels = programming),
           Residual,
           fill = sign)) +
  geom_col(position = "dodge", 
           aes(alpha = contrib_0_to_1)) +
  xlab("") +
  ylab("Chi-square standardized residuals of\npost-workshop frequencies") +
  scale_x_discrete(labels = function(x) lapply(strwrap(x, width = 10, simplify = FALSE), paste, collapse = "\n")) +
  scale_fill_manual(name = "",
                      values = c("negative" = "red", 
                                 "positive" = "blue"),
                      labels = c("Fewer respondents than \nexpected assuming no effect", 
                                 "More respondents than \nexpected assuming no effect")) +
  ggtitle("Workshops result in respondents \nprogramming significantly more often") +
  theme(plot.title = element_text(hjust = 0.5)) +
  theme_classic(base_size = 14) +
  scale_alpha_continuous("Contribution to \nchi-square value") +
   guides(fill=guide_legend(
                 keywidth = 0.1,
                 keyheight = 0.4,
                 default.unit = "inch")
      )
ggsave("figures/programming_post_workshop.png")
```

A chi-square test indicates that the use of programming significantly increases post-workshop. The chi-squared standardized residuals for the post-workshop values show that significantly more respondents program daily six months (or more) after the workshop than would have been expected had the workshop had no effect on their practice. Similarly, significantly fewer respondents program less than once per year six months (or more) after the workshop.

# Workshop Impact
```{r echo=FALSE}
# Jonah's fix to include the correct levels. Variable was renamed to 'levels' 
# because 'order' is the name of a function in R.
# I am still not sure if this plot is 100% correct -KLJ

cols_with_Agree <- map_lgl(data, ~`%in%`("Agree", .x))
data_agree <-  data[ , cols_with_Agree]

levels = c("Strongly disagree", "Disagree", "Neutral", "Agree", "Strongly agree")

# Beth's tip to both order the factors based on levels and unify the factors
 factorfunction <- function(mydata, factlevel){
  factor(mydata, 
         levels=factlevel, 
         ordered = TRUE)
    fct_unify(mydata, 
              levels=factlevel)}
 # End tip 
 
 # Adjusting names of y axis labels
 names(data_agree) <- 
  c("Coding",
    "Confidence",
    "Career",
    "Motivation",
    "Reproducible",
    "Productivity",
    "Recognition")

data_agree_likert <- likert(data.frame(lapply(data_agree, factor, levels, ordered=TRUE)))

agree_or_strongly_agree_improved_3_things <- 
  data_agree %>% 
  select(Coding, Reproducible, Productivity) %>% 
  filter(Coding       %in% c("Agree", "Strongly agree") | # or
         Reproducible %in% c("Agree", "Strongly agree") |
         Productivity %in% c("Agree", "Strongly agree")) 

perc_agree_or_strongly_agree_improved_3_things <- 
  round(nrow(agree_or_strongly_agree_improved_3_things) / nrow(data_agree) * 100, 0)

received_professional_recognition <- 
  data_agree_likert$results %>% 
  filter(Item == "Recognition") %>% 
  select(Agree, `Strongly agree`) %>% 
  sum() %>% 
  round(0)
```
The figure below shows respondents' perception of workshop impact on several factors, including career, confidence, and continuous learning. Respondents were asked to rate their level of agreement (1-Strongly disagree to 5-Strongly agree) with the statements below. The x-axis labels for the figure are in bold, and correspond to the statement following.

+ *Reproducible*: I have made my analyses more reproducible as a result of completing the workshop. 
+ *Recognition*: I have received professional recognition for my work as a result of using the tools I learned at the workshop. 
+ *Productivity*: My research productivity has improved as a result of completing the workshop. 
+ *Motivation*: I have been motivated to seek more knowledge about the tools I learned at the workshop. 
+ *Confidence*: I have gained confidence in working with data as a result of completing the workshop. 
+ *Coding*: I have improved my coding practices as a result of completing the workshop. 
+ *Career*: I have used skills I learned at the workshop to advance my career. 

```{r}
title <- "Perception of Workshop Impact"		
plot(data_agree_likert, type =c("heat"), panel.arrange = NULL, panel.strip.color = "red", legend.position = "bottom") + ggtitle(title)
ggsave("figures/workshop_impact_heatmap.png")
```

We see an overwhelmingly positive indication that, after taking the workshop, respondents feel motivated to seek more knowledge and have gained confidence in working with data. Additionally, respondents are making their analyses more reproducible and improving their coding practices (i.e. keeping raw data raw, sharing source code openly, and using scripts). As this survey was administered to learners who had taken a workshop at least six months ago, this feeling of confidence has obviously persisted. This is quite impactful, as both the [Data Carpentry](https://carpentries.github.io/assessment-projects//data-carpentry-projects/postworkshop_analysis.html) and [Software Carpentry](https://carpentries.github.io/assessment-projects/software-carpentry-projects/analysis-postworkshop.html) post workshop survey results show learners having self-reported improved understanding of how to import and work with data. 

<!-- BM: in the paragraph below, where did the original figure of 80% come from? I cannot reproduce it. Inline R code is better than hard-coded numbers -->

Additionally, `r perc_agree_or_strongly_agree_improved_3_things`% of respondents agree or strongly agree that they have improved their coding practices, made their analyses reproducible, or improved their research productivity. They also believe the skills they learned helped them advance their career: `r received_professional_recognition`% of our respondents have received professional recognition as a result of using the tools they learned in a Carpentries workshop.

```{r include=FALSE}

title <- "Perception of Workshop Impact"
 theme_update(plot.title = element_text(hjust = 0.5))
plot(data_agree_likert) + ggtitle(title) 

#print(class(data_agree))
 data_agree <- map_if(data_agree,
                     is.character,
                     as.factor)
 
 # The plot works, but the labels are not in the correct order.
```

### Behaviors Respondents Adopted

```{r include = FALSE}
# Code for behaviors adopted
# Responses are in columns 'Behaviors-Adopted' through 'Column32'

# How many responses do we have to this question?

responses_to_behaviours <- 
  n_responses_to_the_question(data, 
                              from_colname = `Behaviors-Adopted`, 
                              to_colname = Column32)


# Use 'gather' to go from wide to long format
Behaviors <- 
data %>%
  select(`Behaviors-Adopted`:Column32) %>% 
  gather(col, Behaviors) %>% 
  group_by(Behaviors) %>% 
  tally_and_perc(Behaviors, 
                 na.rm = TRUE, 
                 question_n = responses_to_behaviours) %>%
  filter(!is.na(Behaviors)) %>% 
  arrange(desc(n))

#  how many rate either of these three?
#- data management and project organization practices : Behaviors-Adopted
#- used programming languages for automation : Column28
#- used version control to manage code : Column30

relevant_cols <- c("Behaviors-Adopted", "Column28", "Column30")

rowsums <- 
  data %>%
  select(one_of(relevant_cols)) %>% 
  # check that each row has a value for at least one col
  # convert to numeric, if NA, then put a zero, otherwise 1
  map_df(., ~ifelse(is.na(.x), 0, 1)) %>% 
  # sum to see if any rows have no reponses
  rowSums() 

# how many of these rows don't sum to zero?
idx <- ifelse(rowsums == 0, FALSE, TRUE)
number_that_adopted_any_of_those_three <- sum(idx)
```

We asked respondents to identify the behaviors they adopted as a result of completing a Carpentries workshop. We are happy to report that more than half the respondents who answered this question have improved their data management and project organization practices, have used programming languages for automation, and have used version control to manage code. Additionally, respondents are __more confident__ now in using the tools than before they completed the Carpentries workshop.
```{r}
kable(Behaviors, 
      format = "markdown", 
      digits = getOption("digits"), 
      row.names = NA, 
      col.names = NA, 
      caption = NULL, 
      format.args = list(), 
      escape = TRUE)
```
The matrix below shows a count of the highest combinations of behaviors adopted. Note that the 102 respondents who reported improving their data management and project organization also now use version control to manage code.

```{r}
# Code for matrix of behaviors (thanks Ben for the tip!)
# Combinations of behaviors adopted for individuals
behaviors_cols <- 
data %>%
  select(`Behaviors-Adopted`:Column32)

# Matrix 
m <- as.matrix(behaviors_cols) 
# the unique values in the matrix
vals <- sort(unique(as.vector(m)))

# rearrange the data so that each value is a column
bigm <- t(apply(m, 1, function(row) match(vals, row, nomatch=0)))
colnames(bigm) <- vals

# count the co-occurences of each value (diagonal is total number of rows with that value)
behaviors_co_occurences  <- as.data.frame(crossprod(bigm>0))
kable(behaviors_co_occurences, row.names = TRUE, caption = "Matrix of Common Behaviors Adopted Post-Workshop")
```

```{r heatmap, fig.width=8, fig.height = 8}
library(tidyverse)
library(viridis)
library(stringr)
wrap_width <-  30
text_size <-  8
behaviors_co_occurences  %>% 
  as.matrix() %>% 
  reshape2::melt() %>% 
  as_tibble() %>% 
  ggplot(aes(x = Var1, 
             y = Var2, 
           fill = value)) +  
  geom_tile() +
  scale_fill_viridis() +
  coord_equal() +
  xlab("") +
  ylab("") +
  scale_x_discrete(labels = function(x) str_wrap(x, width = wrap_width)) +
  scale_y_discrete(labels = function(x) str_wrap(x, width = wrap_width)) +
  theme(axis.text.x = element_text(angle = 90, size = text_size, vjust = 0.5, hjust = 1),
        axis.text.y = element_text(size = text_size))  +
  ggtitle("Heatmap of behaviour co-occurances")

ggsave("figures/behaviors_heatmap.png")

```

### Change in Confidence 

```{r}
# Code for change in confidence
confidence = c("I'm less confident now.", "I'm equally confident now.", "I'm more confident now.")
confidence = factor(confidence)

data$`Change-In-Confidence` = factor(data$`Change-In-Confidence`, levels = confidence)

data_change.in.confidence_tally <- 
  data %>% 
  group_by(`Change-In-Confidence`) %>% 
  tally() %>% 
  filter(!is.na(`Change-In-Confidence`)) %>% 
  mutate(perc = round(n / sum(n) * 100, 0 ))

# for in-text perc
perc_more_confident <- 
  data_change.in.confidence_tally %>% 
  filter(`Change-In-Confidence` == "I'm more confident now.") %>% 
  pull(perc)

# so we can say 'more than', round to nearest 5
mround <- function(x, base){ 
        base * round(x/base) 
} 

more_than_perc_more_confident <- mround(perc_more_confident,5) 
  
```

Our goal is for learners to leave a workshop with increased confidence about using the tools they were taught. More than `r more_than_perc_more_confident`% of the respondents are now more confident in using the tools they learned than before attending the workshop.


```{r}

# Use the code below for a table of the data.
# kable(data_change.in.confidence_tally, format = "markdown", row.names = FALSE, col.names = c("Change in Confidence", "%"))

ggplot(data_change.in.confidence_tally, 
       aes(`Change-In-Confidence`, y = 100 * (n/sum(n)),
           n)) +
   geom_bar(stat = "identity", fill = "orange") +
   geom_text(aes(label=n), size= 4, vjust=-0.25) +
   scale_x_discrete(labels = function(x) lapply(strwrap(x, width = 10, simplify = FALSE), paste, collapse="\n")) +
   theme_classic() +
   xlab("Change in Confidence") +
   ylab("% Respondents") +
   ggtitle("Respondents are More Confident Post-Workshop") +
   theme(plot.title = element_text(hjust = 0.5)) +
   theme_bw(base_size = 14)
ggsave("figures/change_in_confidence.png")
```

### Usage of Tools for Research and/or Work

We identified specific outcomes directly related to research and/or work, and asked learners if they had achieved these outcomes post-workshop. Respondents reported that the tools they learned improved their overall efficiency, as well as their ability to manage and analyze data.  

```{r include=FALSE}
# Data are in columns 'How-Tools-Learned-Help' through 'Column37'

# how many responses do we have to this question?

n_reponses_tools_learned_help <- 
    n_responses_to_the_question(data, 
                              from_colname = `How-Tools-Learned-Help`, 
                              to_colname = Column37)

# How many responded to either of these?
# They are improving my overall efficienct : How.Tools.Learned.Helped
# They are improving my ability to analyze data. : Column34
# They are improving my ability to manage data. : Column35

relevant_cols <- c("How-Tools-Learned-Help", "Column34", "Column35")

rowsums <- 
  data %>%
  select(one_of(relevant_cols)) %>% 
  # check that each row has a value for at least one col
  # convert to numeric, if NA, then put a zero, otherwise 1
  map_df(., ~ifelse(is.na(.x), 0, 1)) %>% 
  # sum to see if any rows have no reponses
  rowSums() 

idx <- ifelse(rowsums == 0, FALSE, TRUE)
number_that_tools_helped <- sum(idx)

how_help <- 
data %>%
  select(`How-Tools-Learned-Help`:Column37) %>% 
  gather(col, how_help) %>% 
  group_by(how_help) %>% 
  tally() %>%
  filter(!is.na(how_help)) %>% 
  arrange(desc(n)) %>% 
  rename(`How-Tools-Learned-Help` = how_help)

# Data to use in-line
no_help <- table(data$`Column36`)
no_use <- table(data$`Column37`)

tools_helped <- 
data %>%
  select(`How-Tools-Learned-Help`:Column37) %>% 
  gather(col, tools_helped) %>% 
  group_by(tools_helped) %>% 
  tally_and_perc(tools_helped, na.rm = TRUE, question_n = n_reponses_tools_learned_help) %>%  
  filter(!is.na(tools_helped)) %>% 
  arrange(desc(n)) %>%
  rename(`How Tools Covered Have Helped` = tools_helped)

```

```{r}
kable(tools_helped, 
      format = "markdown", 
      row.names = FALSE, 
      caption = "Self-Reported Perception of How Tools Help Respondents")
```

Only `r no_help` respondents said the tools they learned have not helped them, and `r no_use` respondents have not been using the tools that were covered in their workshop.

### Contributions to Academic Writing

Another possible outcome of attending a Carpentries workshop is in the use of tools learned to contribute to academic writing (i.e. a grant proposal, journal article). 
```{r}
# Code chunk for contributions to academic writing

# how many responded to this question?
rowsums <- 
data %>%
  select(`Contributed-To-Writing`) %>% 
  # check that each row has a value for at least one col
  # convert to numeric, if NA, then put a zero, otherwise 1
  map_df(., ~ifelse(is.na(.x), 0, 1)) %>% 
  # sum to see if any rows have no reponses
  rowSums() 

# for all writing responses
idx <- ifelse(rowsums == 0, FALSE, TRUE)
n_responsed_to_writing <-  sum(idx)

# table about writing
data %>% 
  tally_and_perc(`Contributed-To-Writing`, 
                 na.rm = TRUE, 
                 question_n = n_responsed_to_writing) %>% 
  kable()


writing = c("No.", "Not sure.", "Yes.")
writing = factor(writing)

data$`Contributed-To-Writing` = factor(data$`Contributed-To-Writing`, levels = writing)
Contributed_Writing <- round(prop.table(table(data$`Contributed-To-Writing`)) * 100)
```

Only `r Contributed_Writing[3]`% of our respondents said that the tools they learned contributed to their academic writing. This is an opportunity for us to explore resources we can offer the community to help use the tools for academic writing purposes.

### Continuous Learning
```{r}
# Code chunk for continuous learning 

Learning_Activities <- 
data %>%
  select(`Activities`:Column57) %>% 
  gather(col, Learning_Activities) %>% 
  group_by(Learning_Activities) %>% 
  tally() %>% 
  filter(!is.na(Learning_Activities)) %>% 
  mutate(`%` =round(n / sum(n) * 100, 0) ) %>% 
  arrange(desc(n)) %>% 
  rename(`Continuous Learning Post-Workshop` = Learning_Activities)

# To use code for Carpentry self-guided material in-line
carpentry_self_guided <- table(data$Column56)
```
A key finding is that learners continue their learning after completing a workshop. This can take many forms, including participating in short courses (in-person and online) and using self-guided material. We asked respondents to tell us which learning activities (for data management and analysis) they have participated in since completing a Carpentries workshop. The majority of respondents have used non-Carpentries, self-guided material, though `r carpentry_self_guided` responded having used Carpentries' self-guided material. Additionally, greater participation in meetups and in-person short courses has been reported by respondents.
```{r}
# Code chunk for table of continuous learning activities
kable(Learning_Activities, 
      format = "markdown", 
      row.names = FALSE, 
      col.names = c("Continuous Learning", "n", "%"), 
      caption = "Respondents Self-Reported Continuous Learning Activities")
```

The matrix below provides a breakdown of the combination of continuous learning activities respondents participated in. For example, 35 respondents have used both Carpentry and non-Carpentry self-guided material since attending a workshop.

```{r}
# Code for matrix of continuous learning (thanks Ben for the tip!)
# Combinations of continuous learning for individual
learning_cols <- 
data %>%
  select(`Activities`:Column57)

# Matrix 
m <- as.matrix(learning_cols) 
# the unique values in the matrix
vals <- sort(unique(as.vector(m)))

# rearrange the data so that each value is a column
bigm <- t(apply(m, 1, function(row) match(vals, row, nomatch=0)))
colnames(bigm) <- vals

# count the co-occurences of each value (diagonal is total number of rows with that value)
learning_co_occurences  <- as.data.frame(crossprod(bigm>0))
kable(learning_co_occurences, row.names = TRUE, caption = "Matrix of Common Continuous Learning Activities")
```

## Involvement in the Carpentries 

Learners often become actively involved with Software and/or Data Carpentry after completing a workshop. This involvement can take many forms - joining a mentoring group, becoming a workshop helper, or even becoming an instructor. The table provided below shows how respondents have involved themselves with the Carpentries since completing a workshop. Respondents were asked to check all that apply.

```{r}
# Code chunk for involvement with the Carpentries
# Data in columns 'Carpentry.Involvement through Column51
Carpentry_Involvement <- 
data %>%
  select(`Carpentry-Involvement`:Column51) %>% 
  gather(col, Carpentry_Involvement) %>% 
  group_by(Carpentry_Involvement) %>% 
  tally() %>% 
  filter(!is.na(Carpentry_Involvement)) %>% 
  arrange(desc(n)) %>% 
  rename(`Involvement Post-Workshop` = Carpentry_Involvement)

kable(Carpentry_Involvement, 
      row.names = FALSE, 
      caption = "Respondents Self-Reported Inolvement in the Carpentries Post-Workshop")
```

The matrix below displays frequent combinations of post-workshop involvement. For example, 16 of the respondents who became Carpentry instructors have attended at least one community call.

```{r}
# Code for matrix of involvement (thanks Ben for the tip!)
# Combinations of tools for individual
involvement_cols <- 
data %>%
  select(`Carpentry-Involvement`:Column51)

# Matrix tool-by-tool

m <- as.matrix(involvement_cols) 
# the unique values in the matrix
vals <- sort(unique(as.vector(m)))

# rearrange the data so that each value is a column
bigm <- t(apply(m, 1, function(row) match(vals, row, nomatch=0)))
colnames(bigm) <- vals

# count the co-occurences of each value (diagonal is total number of rows with that value)
involvement_co_occurences  <- as.data.frame(crossprod(bigm>0))
kable(involvement_co_occurences, 
      row.names = TRUE, 
      caption = "Matrix of Common Involvement")
```

```{r include=FALSE}
# Code chunk for whether respondents recommended a workshop
data_recommended_tally <-
  data %>% 
  group_by(Recommended) %>% 
  tally() %>% 
  filter(!is.na(Recommended)) %>% 
  arrange(desc(n))

# Code chunk for in-line text
said_yes <- data_recommended_tally[data_recommended_tally$Recommended == "Yes.", ]$n

# Table for responses of recommendations
# kable(data_recommended_tally)
```

```{r include=FALSE}
# Code chunk for likelihood of recommmending a Carpentry workshop
data_likelyrecommend_tally <-
  data %>% 
  group_by(`Likely-To-Recommend`) %>% 
  tally() %>% 
  filter(!is.na(`Likely-To-Recommend`)) %>% 
  arrange(desc(n))

# Table for responses
# kable(data_likelyrecommend_tally)
```

# Growth Opportunities

We are very excited to know that our workshops are having an impact on learners six months to a year after their attendance. Though the results of this survey are compelling, we do recognize issues for improvement. For example, `r USA_perc`% of respondents completed a workshop in the United States. We are continually discussing ways to broaden participation of our workshops in communities outside of the U.S.

We also realize there are some lessons that are not being taught as frequently as others, namely, the [Shell lesson](http://swcarpentry.github.io/shell-novice/) and the [SQL lesson](http://www.datacarpentry.org/sql-ecology-lesson/). We would like to understand why, and what we can do as a community to see an increase in this lesson being taught.

```{r}
# extract some values from the likert data to use in the text below
feeling_neutral_about_receiving_professional_recognition <- 
  data_agree_likert$results %>% 
  filter(Item == "Recognition") %>% 
  pull(Neutral) %>% 
  round(0)
  
either_disagreed_or_strongly_disagreed_received_any_professional_recognition <-   data_agree_likert$results %>% 
  filter(Item == "Recognition") %>% 
  mutate(dis_or_strongly_dis = sum(c(`Strongly disagree`, Disagree))) %>% 
  pull(dis_or_strongly_dis) %>% 
  round(0)
    
```


Lastly, `r feeling_neutral_about_receiving_professional_recognition`% of the respondents indicated feeling **neutral** about receiving professional recognition as a result of participating in a Carpentries workshop, and `r either_disagreed_or_strongly_disagreed_received_any_professional_recognition`% either disagreed or strongly disagreed that they had received any professional recognition. We would like to explore community development opportunities that will benefit learners’ personal as well as professional endeavours, so that time spent acquiring important skills for research is adequately recognized and possibly even rewarded.

# Summary

When our learners have successful experiences in our workshops, they are quick to share this positive experience with others. We asked respondents if they had already recommended our workshop, and `r Recommended[3]`% said yes!

This initial analysis of how Carpentries workshops have impacted learners long-term has been extremely insightful. In general, our workshops are helping learners improve their efficiency with managing and analyzing data. Learners are taking advantage of online resources to improve their skills, and many see value in becoming involved longer term with our community.

We will revisit this data when we compare the responses of learners who took a workshop more than a year ago with those who have taken a workshop less than six months ago and from six months to one year ago. Additionally, we will continue to use this survey every six months to collect data from new learners so we can monitor their progress, and add to our growing evidence base on assessment.


