Compare the CharacterProposal to TokenProposal(K=None)

[timvieira]

The CharacterProposal is designed to be fast while still hopefully being a good proxy for TokenProposal(K=None). How good is it in practice?

[benlebrun]

Below is a minimal example where the distribution over tokens defined by the CharacterProposal differs from the TokenProposal with K=None (i.e., the local product of experts). In this example, the character proposal places too much probability on a since the frequency of paths with a in them is too high.

Our weights will correct for this issue, but this nonetheless means that we are obtaining sub-optimal token samples from the character proposal.

[timvieira]

Excellent work, @benlebrun

[timvieira]

The example in tests/test_system.py if we vary the proposal has an interesting qualitative difference that we should dig into:

Token:

Character: