More structured description of token regexes

[elegios]

The current description of token regexes uses PCRE regexes, but these are very hard to read when they get large.

This may also be a good opportunity to move to actual regular expressions. That would lose some non-regular features (capture groups, back-references, look-around, etc.) but gain some other ones (intersection, difference, guaranteed linear time parsing).