Published February 6, 2019 | Version v1.1
Software Open

dspinellis/tokenizer: Version 1.1

Creators

  • 1. Athens University of Economics and Business

Description

Tokenize source code into integer vectors, symbols, or discrete tokens.

The following languages are currently supported.

  • C
  • C#
  • C++
  • Java
  • PHP
  • Python

 

Notes

https://github.com/dspinellis/tokenizer/

Files

dspinellis/tokenizer-v1.1.zip

Files (57.5 kB)

Name Size Download all
md5:3232d5b8133d0b692ca23e196b016d95
57.5 kB Preview Download

Additional details

Related works

Is supplement to
https://github.com/dspinellis/tokenizer/tree/v1.1 (URL)