Tokenizer

The code in this example implements a simple tokenizer that turns a linear stream of characters into a linear stream of tokens (for identifer names and (unsigned) integers).

Happy programming in ATS!!!