Make token matching semantics user-definable

[tjvr]

As per our conversation @Hardmath123.

This makes the custom tokens feature a little more useful; you can still define your own tokens with the %-syntax just as before, but now you have complete control over how symbols get matched with tokens inside the parser.

I couldn't detect any performance difference using our benchmarks.

[kach]

I'm wondering whether it should be named "matcher" or "match". I think "match" is better.

[tjvr]

I'll call it whatever you want :-)

[kach]

Well, I was asking you! :P I feel functions tend to be named verbs — "match" — but of course the key in the object is kind of a noun, since it is telling you what the matcher is. Naming this right is important because it's user-facing so we can't change it later.

[tjvr]

Yes, it should be a verb. I'd be more concerned about whether "match" is the right word in the first place :-)

[kach]

test is both a noun and a verb… :P

[tjvr]

Closing in favour of #220.