Prerelease 0.2.0rc1 no longer returns log probs with llama.cpp

[oj-sec]

The bug
Updating from guidance==0.1.16 to prerelease guidance==0.2.0rc1 causes model.log_prob() to return 0 rather than the true log probs for a generation when using the llama.cpp backend. I have tested GGUF quants of models based on Llama, Mistral and Gemma and observed this behaviour to be model agnostic.

To Reproduce
Reproduction Colab notebook here - involves uninstalling and reinstalling Guidance, but the change in output between installs is:

# With guidance==0.1.16
from guidance import models, gen, select
import math

llm = models.LlamaCpp(f"./models/{model}", n_gpu_layers=40, n_ctx=2000, compute_log_probs=True)
output = llm + "You flip a coin. The result is: " + gen(name="coinflip", regex="(heads|tails)")
logprobs = output.log_prob("coinflip")
prob = round(math.exp(logprobs), 5)
print(f"Output:{output['coinflip']}\nLP: {logprobs}\nP: {prob}")

You flip a coin. The result is: heads
Output:heads
LP: -1.1534752799652015
P: 0.31554

# With guidance==0.2.0rc1
from guidance import models, gen, select
import math

llm = models.LlamaCpp(f"./models/{model}", n_gpu_layers=40, n_ctx=2000, compute_log_probs=True)
output = llm + "You flip a coin. The result is: " + gen(name="coinflip", regex="(heads|tails)")
logprobs = output.log_prob("coinflip")
prob = round(math.exp(logprobs), 5)
print(f"Output:{output['coinflip']}\nLP: {logprobs}\nP: {prob}")

You flip a coin. The result is: heads
Output:heads
LP: 0.0
P: 1.0

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Google Colab, Ubuntu 22.04
• Guidance Version (guidance.__version__): 0.2.0rc1 (both from pypi and from https://github.com/microsoft/guidance.git@77fc3999e1545c10f17e6da1b6cbd1feeaa1ca1a)

If I can provide any further info please let me know. Huge thanks for this amazing library.

[hudson-ai]

Hi @oj-sec thanks for bringing this up, and our apologies if it's impacting your workflow. The new parser that we're using in 0.2.0rc1 is considerably faster than what was running in previous versions, but it currently has a few limitations that we need to continue working on (probability outputs is probably the primary one). So, it's on our radar.

Thank you for submitting the issue!