quanteda/quanteda: CRAN v1.4.0

Bug fixes and stability enhancements

• Fixed bug in dfm_compress() and dfm_group() that changed or deleted docvars attributes of dfm objects (#1506).
• Fixed a bug in textplot_xray() that caused incorrect facet labels when a pattern contained multiple list elements or values (#1514).
• kwic() now correctly returns the pattern associated with each match as the "keywords" attribute, for all pattern types (#1515)
• Implemented some improvements in efficiency and computation of unusual edge cases for textstat_simil() and textstat_dist().

• textstat_lexdiv() now works on tokens objects, not just dfm objects. New methods of lexical diversity now include MATTR (the Moving-Average Type-Token Ratio, Covington & McFall 2010) and MSTTR (Mean Segmental Type-Token Ratio).
• New function tokens_split() allows splitting single into multiple tokens based on a pattern match. (#1500)
• New function tokens_chunk() allows splitting tokens into new documents of equally-sized "chunks". (#1520)
• New function textstat_entropy() now computes entropy for a dfm across feature or document margins.
• The documentation for textstat_readability() is vastly improved, now providing detailing all formulas and providing full references.
• New function dfm_match() allows a user to specify the features in a dfm according to a fixed vector of feature names, including those of another dfm. Replaces dfm_select(x, pattern) where pattern was a dfm.
• A new argument vertex_labelsize added to textplot_network() to allow more precise control of label sizes, either globally or individually.

• tokens.tokens(x, remove_hyphens = TRUE) where x was generated with remove_hyphens = FALSE now behaves similarly to how the same tokens would be handled had this option been called on character input as tokens.character(x, remove_hyphens = TRUE). (#1498)