Set-up comparisons with outlines

[timvieira]

https://github.com/outlines-dev/outlines

[timvieira]

@benlipkin any thoughts on this?

[benlipkin]

I can add this in whenever we have a clear eval we want to run. It's only an import and a few lines of code and they already accept inputs in lark format. In addition, I took a look at this recently, and there have actually been a few new bugs added since I implemented this originally, so I know where the edge cases are and what types of grammars the samples will be especially poor on (as well as some new completeness edge cases that have now been introduced).

[timvieira]

One option would be to integrate it into the library as a drop-in replacement to our guided generation strategies. That doesn't require a specific benchmark; it just requires making them have a common interface. It might have the side benefit of enabling outlines-SMC.