Introduction to tidyeval

Tidyverse grammars such as dplyr were originally designed for interactive use and analysis scripts. Part of it is that these grammars attempt to model how data analysts think about data problems. Another reason is that they make your data first class and let you refer to data frame columns as if they were distinct objects. While it contributes to make dplyr code look concise and to the point, this last aspect unfortunately has a dark side. It makes it harder to reduce duplication by wrapping tidyverse code in functions and loops. Writing functions around dplyr pipelines and other tidyeval APIs requires a bit of special knowledge because these APIs use a special type of functions called quoting functions in order to make data first class.

If one-off code is often reasonable for common data analysis tasks, it is good practice to write reusable functions to reduce code duplication. In this vignette you will learn about quoting functions, what challenges they pose for programming, and the solutions that tidy evaluation provides to solve those problems.

Introduction

Writing functions with tidy eval

Writing functions is essential for the clarity and robustness of your code. Functions have several advantages:

  1. They prevents inconsistencies because it enforces multiple computations to follow a single recipe.

  2. They emphasise what varies (the arguments) and what is constant (every other component of the computation).

  3. They make change easier because you only need to modify one place.

  4. They make your code clearer if you give the function and its arguments informative names.

The process for creating a function is straightforward. First recognise duplication in your code. A good rule of thumb is to create a function when you have copy-pasted a piece of code three times. Can you spot the copy-paste mistake in this code duplication?

(df$a - min(df$a)) / (max(df$a) - min(df$a))
(df$b - min(df$b)) / (max(df$b) - min(df$b))
(df$c - min(df$c)) / (max(df$c) - min(df$c))
(df$d - min(df$d)) / (max(df$d) - min(df$c))

Now identify the variying parts of the expression and give them a name. x is an easy choice but it is often a good idea to reflect the type of argument expected in the name. In our case we expect a numeric vector:

(num - min(num)) / (max(num) - min(num))
(num - min(num)) / (max(num) - min(num))
(num - min(num)) / (max(num) - min(num))
(num - min(num)) / (max(num) - min(num))

We can now create a function with a relevant name:

rescale01 <- function(num) {

}

Fill it with our deduplicated code:

rescale01 <- function(num) {
  (num - min(num)) / (max(num) - min(num))
}

And refactor a little to reduce duplication further and handle more cases:

rescale01 <- function(num) {
  rng <- range(num, na.rm = TRUE, finite = TRUE)
  (num - rng[[1]]) / (rng[[2]] - rng[[1]])
}

Now you can reuse your function any place you need it:

rescale01(df$a)
rescale01(df$b)
rescale01(df$c)
rescale01(df$d)

Reducing code duplication is as much needed with tidyverse grammars as with ordinary computations. Unfortunately, the straighforward process to create functions breaks down with grammars like dplyr. To see why, let’s try to reproduce the process with a duplicated pipeline:

df1 %>% group_by(x1) %>% summarise(mean = mean(y1))
df2 %>% group_by(x2) %>% summarise(mean = mean(y2))
df3 %>% group_by(x3) %>% summarise(mean = mean(y3))
df4 %>% group_by(x4) %>% summarise(mean = mean(y4))

We first abtract out the varying parts by giving them informative names:

data %>% group_by(group_var) %>% summarise(mean = mean(summary_var))

And wrap the pipeline with a function taking these argument names:

grouped_mean <- function(data, group_var, summary_var) {
  data %>%
    group_by(group_var) %>%
    summarise(mean = mean(summary_var))
}

Unfortunately this function doesn’t actually work. When you call it dplyr complains that the variable group_var is unknown:

grouped_mean(mtcars, cyl, mpg)
#> Error in grouped_df_impl(data, unname(vars), drop): Column `group_var` is unknown

Here is the proper way of defining this function:

grouped_mean <- function(data, group_var, summary_var) {
  group_var <- enquo(group_var)
  summary_var <- enquo(summary_var)

  data %>%
    group_by(!!group_var) %>%
    summarise(mean = mean(!!summary_var))
}
grouped_mean(mtcars, cyl, mpg)
#> # A tibble: 3 x 2
#>     cyl  mean
#>   <dbl> <dbl>
#> 1     4  26.7
#> 2     6  19.7
#> 3     8  15.1

To understand how that works, we need to learn about quoting functions and what special steps are needed to be effective at programming with them. Really we only need two new concepts forming together a single pattern: quoting and unquoting. This vignette will get you up to speed with this pattern.

Evaluating functions versus quoting functions

R functions can be categorised in two broad categories: evaluating functions and quoting functions 1. These functions differ in the way they get their arguments. Evaluating functions take arguments as values. It does not matter what the expression supplied as argument is or which objects it contains. R computes the argument value following the standard rules of evaluation which the function receives passively 2.

The simplest regular function is identity(). It evaluates its single argument and returns the value. Because only the final value of the argument matters, all of these statements are completely equivalent:

identity(6)
#> [1] 6

identity(2 * 3)
#> [1] 6

a <- 2
b <- 3
identity(a * b)
#> [1] 6

On the other hand, a quoting function is not passed the value of an expression, it is passed the expression itself. We say the argument has been automatically quoted. The quoted expression might be evaluated a bit later or might not be evaluated at all. The simplest quoting function is quote(). It automatically quotes its argument and returns the quoted expression without any evaluation. Because only the expression passed as argument matters, none of these statements are equivalent:

quote(6)
#> [1] 6

quote(2 * 3)
#> 2 * 3

quote(a * b)
#> a * b

Other familiar quoting operators are "" and ~. The "" operator quotes a piece of text at parsing time and returns a string. This prevents the text from being interpreted as some R code to evaluate. The tilde operator is similar to the quote() function in that it prevents R code from being automatically evaluated and returns a quoted expression in the form of a formula. The expression is then used to define a statistical model in modelling functions. The three following expressions are doing something similar, they are quoting their input:

"a * b"
#> [1] "a * b"

~a * b
#> ~a * b

quote(a * b)
#> a * b

The first statement returns a quoted string and the other two return quoted code in a formula or as a bare expression.

Quoting and evaluating in mundane R code

As an R programmer, you are probably already familiar with the distinction between quoting and evaluating functions. Take the case of subsetting a data frame column by name. The [[ and $ operators are both standard for this task but they are used in very different situations. The former supports indirect references like variables or expressions that represent a column name while the latter takes a column name directly:

df <- data.frame(
  y = 1,
  var = 2
)

df$y
#> [1] 1

var <- "y"
df[[var]]
#> [1] 1

Technically, [[ is an evaluating function while $ is a quoting function. You can indirectly refer to columns with [[ because the subsetting index is evaluated, allowing indirect references. The following expressions are completely equivalent:

df[[var]] # Indirect
#> [1] 1

df[["y"]] # Direct
#> [1] 1

But these are not:

df$var    # Direct
#> [1] 2

df$y      # Direct
#> [1] 1

The following table summarises the fundamental asymmetry between the two subsetting methods:

Quoted Evaluated
Direct df$y df[["y"]]
Indirect ??? df[[var]]

Detecting quoting functions

Because they work so differently to standard R code, it is important to recognise auto-quoted arguments. The documentation of the quoting function should normally tell you if an argument is quoted and evaluated in a special way. You can also detect quoted arguments by yourself with some experimentation. Let’s take the following expressions involving a mix of quoting and evaluating functions:

library(MASS)
#> 
#> Attaching package: 'MASS'
#> The following object is masked from 'package:dplyr':
#> 
#>     select

mtcars2 <- subset(mtcars, cyl == 4)

sum(mtcars2$am)
#> [1] 8

rm(mtcars2)

A good indication that an argument is auto-quoted and evaluated in a special way is that the argument will not work correctly outside of its original context. Let’s try to break down each of these expressions in two steps by storing the arguments in an intermediary variable:

  1. temp <- MASS
    #> Error in eval(expr, envir, enclos): object 'MASS' not found
    
    temp <- "MASS"
    library(temp)
    #> Error in library(temp): there is no package called 'temp'

    We get these errors because there is no MASS object for R to find, and temp is interpreted by library() directly as a package name rather than as an indirect reference. Let’s try to break down the subset() expression:

  2. temp <- cyl == 4
    #> Error in eval(expr, envir, enclos): object 'cyl' not found

    R cannot find cyl because we haven’t specified where to find it. This object exists only inside the mtcars data frame.

  3. temp <- mtcars$am
    sum(temp)
    #> [1] 13

    It worked! sum() is an evaluating function and the indirect reference was resolved in the ordinary way.

  4. mtcars2 <- mtcars
    temp <- "mtcars2"
    rm(temp)
    
    exists("mtcars2")
    #> [1] TRUE
    exists("temp")
    #> [1] FALSE

    This time there was no error, but we have accidentally removed the variable temp instead of the variable it was referring to. This is because rm() auto-quotes its arguments.

Unquoting

In practice, functions that evaluate their arguments are easier to program with because they support both direct and indirect references. For quoting functions, a piece of syntax is missing. We need the ability of unquoting arguments.

Unquoting in base R

Base R provides three different ways of allowing direct references:

  • An extra function that evaluates its arguments. For instance the evaluating variant of the $ operator is [[.

  • An extra parameter that switches off auto-quoting. For instance library() evaluates its first argument if you set character.only to TRUE:

    temp <- "MASS"
    library(temp, character.only = TRUE)
  • An extra parameter that evaluates its argument. If you have a list of object names to pass to rm(), use the list argument:

    temp <- "mtcars2"
    rm(list = temp)
    
    exists("mtcars2")
    #> [1] FALSE

There is no general unquoting convention in base R so you have to read the documentation to figure out how to unquote an argument. Many functions like subset() or transform() do not provide any unquoting option at all.

Unquoting in the tidyverse!!

All quoting functions in the tidyverse support a single unquotation mechanism, the !! operator (pronounced bang-bang). You can use !! to cancel the automatic quotation and supply indirect references everywhere an argument is automatically quoted.

First let’s create a couple of variables that hold references to columns from the mtcars data frame. A simple way of creating these references is to use the fundamental quoting function quote():

# Variables referring to columns `cyl` and `mpg`
x_var <- quote(cyl)
y_var <- quote(mpg)

x_var
#> cyl

y_var
#> mpg

Here are a few examples of how !! is used throughout the tidyverse to unquote such references.

  • In dplyr most verbs quote their arguments:

    library("dplyr")
    
    by_cyl <- mtcars %>%
      group_by(!!x_var) %>%            # Refer to x_var
      summarise(mean = mean(!!y_var))  # Refer to y_var
  • In ggplot2 aes() is the main quoting function:

    library("ggplot2")
    
    ggplot(mtcars, aes(!!x_var, !!y_var)) +  # Refer to x_var and y_var
      geom_point()

    ggplot2 also features vars() which is useful for facetting:

    ggplot(mtcars, aes(disp, drat)) +
      geom_point() +
      facet_grid(vars(!!x_var))  # Refer to x_var

Indirect references in quoting functions are rarely useful in scripts but they are invaluable for writing functions. With !! we can now easily fix our wrapper function. as we’ll see in the following section.

Understanding !! with qq_show()

At this point it is normal if the concept of unquoting still feels nebulous. A good way of practicing this operation is to see for yourself what it is really doing. To that end the qq_show() function from the rlang package performs unquoting and prints the result at the screen. Here is what !! is really doing in the dplyr example (I’ve broken up the pipeline in two steps for readability):

rlang::qq_show(mtcars %>% group_by(!!x_var))
#> mtcars %>% group_by(cyl)

rlang::qq_show(data %>% summarise(mean = mean(!!y_var)))
#> data %>% summarise(mean = mean(mpg))

Similarly for the ggplot2 pipeline:

rlang::qq_show(ggplot(mtcars, aes(!!x_var, !!y_var)))
#> ggplot(mtcars, aes(cyl, mpg))

rlang::qq_show(facet_grid(vars(!!x_var)))
#> facet_grid(vars(cyl))

As you can see, unquoting a variable that contains a reference to the column cyl is equivalent to directly supplying cyl to the dplyr function.

Quoting and unquoting arguments

The basic process for creating tidyeval functions requires thinking a bit differently but is straightforward: quote and unquote.

  1. Use enquo() to make a function automatically quote its argument.
  2. Use !! to unquote the argument.

Apart from these additional two steps, the process is the same.

The abstraction step

We start as usual by identifying the varying parts of a computation and giving them informative names. These names become the arguments to the function.

grouped_mean <- function(data, group_var, summary_var) {
  data %>%
    group_by(group_var) %>%
    summarise(mean = mean(summary_var))
}

As we have seen earlier this function does not quite work yet so let’s fix it by applying the two new steps.

The quoting step

The quoting step is about making our ordinary function a quoting function. Not all parameters should be automatically quoted though. For instance the data argument refers to a real data frame that is passed around in the ordinary way. It is crucial to identify which parameters of your function should be automatically quoted: the parameters for which it is allowed to refer to columns in the data frames. In the example, group_var and summary_var are the parameters that refer to the data.

We know that the fundamental quoting function is quote() but how do we go about creating other quoting functions? This is the job of enquo(). While quote() quotes what you typed, enquo() quotes what your user typed. In other words it makes an argument automatically quote its input. This is exactly how dplyr verbs are created! Here is how to apply enquo() to the group_var and summary_var arguments:

group_var <- enquo(group_var)
summary_var <- enquo(summary_var)

The unquoting step

Finally we identify any place where these variables are passed to other quoting functions. That’s where we need to unquote with !!. In this case we pass group_var to group_by() and summary_var to summarise():

data %>%
  group_by(!!group_var) %>%
  summarise(mean = mean(!!summary_var))

Result

The finalised function looks like this:

grouped_mean <- function(data, group_var, summary_var) {
  group_var <- enquo(group_var)
  summary_var <- enquo(summary_var)

  data %>%
    group_by(!!group_var) %>%
    summarise(mean = mean(!!summary_var))
}

And voilà!

grouped_mean(mtcars, cyl, mpg)
#> # A tibble: 3 x 2
#>     cyl  mean
#>   <dbl> <dbl>
#> 1     4  26.7
#> 2     6  19.7
#> 3     8  15.1

grouped_mean(mtcars, cyl, disp)
#> # A tibble: 3 x 2
#>     cyl  mean
#>   <dbl> <dbl>
#> 1     4  105.
#> 2     6  183.
#> 3     8  353.

grouped_mean(mtcars, am, disp)
#> # A tibble: 2 x 2
#>      am  mean
#>   <dbl> <dbl>
#> 1     0  290.
#> 2     1  144.

This simple quote-and-unquote pattern will get you a long way. It makes it possible to abstract complex combinations of quoting functions into a new quoting function. However this gets us in a sort of loop: quoting functions unquote inside other quoting functions and so on. At the start of the loop is the user typing expressions that are automatically quoted. But what if we can’t or don’t want to start with expressions typed by the user? What if we’d like to start with a character vector of column names?

Calling quoting functions with strings

So far we have created a quoting function that wraps around other quoting functions. How can we break this chain of quoting? How can we go from the evaluating world to the quoting universe? The most common way this transition occurs is when you start with a character vector of column names and somehow need to pass the corresponding columns to quoting functions like dplyr::mutate(), dplyr::select(), or ggplot2::aes(). We need a way of bridging evaluating and quoting functions.

First let’s see why simply unquoting strings does not work:

var <- "height"
mutate(starwars, rescaled = !!var * 100)
#> Error in mutate_impl(.data, dots): Evaluation error: non-numeric argument to binary operator.

We get a type error. Observing the result of unquoting with qq_show() will shed some light on this:

rlang::qq_show(mutate(starwars, rescaled = !!var * 100))
#> mutate(starwars, rescaled = "height" * 100)

We have unquoted a string, and now dplyr tried to multiply that string by 100! There is a fundamental difference between these two objects:

"height"
#> [1] "height"

quote(height)
#> height

"heiht" is a string and quote(height) is a symbol, or variable name. A symbol is much more than a string, it is a reference to an R object. That’s why you have to use symbols to refer to data frame columns. Fortunately transforming strings to symbols is straightforward with the tidy eval sym() function:

sym("height")
#> height

If you use sym() instead of enquo(), you end up with an evaluating function that transforms its inputs into symbols that can suitably be unquoted:

grouped_mean2 <- function(data, group_var, summary_var) {
  group_var <- sym(group_var)
  summary_var <- sym(summary_var)

  data %>%
    group_by(!!group_var) %>%
    summarise(mean = mean(!!summary_var))
}

With this simple change we now have an evaluating wrapper which can be used in the same way as [[. You can call grouped_mean2() with direct references:

grouped_mean2(starwars, "gender", "mass")
#> # A tibble: 5 x 2
#>   gender         mean
#>   <chr>         <dbl>
#> 1 female           NA
#> 2 hermaphrodite  1358
#> 3 male             NA
#> 4 none             NA
#> # ... with 1 more row

Or indirect references:

grp_var <- "gender"
sum_var <- "mass"
grouped_mean2(starwars, grp_var, sum_var)
#> # A tibble: 5 x 2
#>   gender         mean
#>   <chr>         <dbl>
#> 1 female           NA
#> 2 hermaphrodite  1358
#> 3 male             NA
#> 4 none             NA
#> # ... with 1 more row

  1. In practice this is a bit more complex because most quoting functions evaluate at least one argument, usually the data argument.

  2. This is why regular functions are said to use standard evaluation unlike quoting functions which use non-standard evaluation (NSE). Note that the function is not entirely passive. Because arguments are lazily evaluated, the function gets to decide when an argument is evaluated, if at all.