NA values are handled in unnest_tokens so they no longer cause other columns to become NA (#82).data.table) consistently (#88).unnest_tokens, bind_tf_idf, all sparse casters) (#67, #74).stm package (#51).get_sentiments now works regardless of whether tidytext has been loaded or not (#50).unnest_tokens now supports data.table objects (#37).to_lower parameter in unnest_tokens to work properly for all tokenizing options.tidy.corpus, glance.corpus, tests, and vignette for changes to quanteda APIpair_count function, which is now in the in-development widyr packagemallet packageunnest_tokens preserves custom attributes of data frames and data.tablescast_sparse, cast_dtm, and other sparse casters to ignore groups in the input (#19)unnest_tokens so that it no longer uses tidyr’s unnest, but rather a custom version that removes some overhead. In some experiments, this sped up unnest_tokens on large inputs by about 40%. This also moves tidyr from Imports to Suggests for now.unnest_tokens now checks that there are no list columns in the input, and raises an error if present (since those cannot be unnested).format argument to unnest_tokens so that it can process html, xml, latex or man pages using the hunspell package, though only when token = "words".get_sentiments function that takes the name of a lexicon (“nrc”, “bing”, or “sentiment”) and returns just that sentiment data frame (#25)cast_sparse to work with dplyr 0.5.0pair_count function, which has been moved to pairwise_count in the widyr package. This will be removed entirely in a future version.