Published March 21, 2018
| Version v0.2.0
Software
Open
ropensci/tokenizers: tokenizers 0.2.0
- 1. George Mason University
- 2. University of Washington
- 3. London School of Economics and Political Science
- 4. UC Berkeley
Description
Features
- Add the
tokenize_ptb()
function for Penn Treebank tokenizations (@jrnold) (#12). - Add a function
chunk_text()
to split long documents into pieces (#30). - New functions to count words, characters, and sentences without tokenization (#36).
- New function
tokenize_tweets()
preserves usernames, hashtags, and URLS (@kbenoit) (#44). - The
stopwords()
function has been removed in favor of using the stopwords package (#46). - The package now complies with the basic recommendations of the Text Interchange Format. All tokenization functions are now methods. This enables them to take corpus inputs as either TIF-compliant named character vectors, named lists, or data frames. All outputs are still named lists of tokens, but these can be easily coerced to data frames of tokens using the
tif
package. (#49) - Add a new vignette "The Text Interchange Formats and the tokenizers Package" (#49).
tokenize_skip_ngrams
has been improved to generate unigrams and bigrams, according to the skip definition (#24).- C++98 has replaced the C++11 code used for n-gram generation, widening the range of compilers
tokenizers
supports (@ironholds) (#26). tokenize_skip_ngrams
now supports stopwords (#31).- If tokenisers fail to generate tokens for a particular entry, they return
NA
consistently (#33). - Keyboard interrupt checks have been added to Rcpp-backed functions to enable users to terminate them before completion (#37).
tokenize_words()
gains arguments to preserve or strip punctuation and numbers (#48).tokenize_skip_ngrams()
andtokenize_ngrams()
to return properly marked UTF8 strings on Windows (@patperry) (#58).
Files
ropensci/tokenizers-v0.2.0.zip
Files
(978.9 kB)
Name | Size | Download all |
---|---|---|
md5:32aa98f5d41d70a0207fdd95d2ecd9f2
|
978.9 kB | Preview Download |
Additional details
Related works
- Is supplement to
- https://github.com/ropensci/tokenizers/tree/v0.2.0 (URL)