[WIP][Bug] Issue 934

[riedgar-ms]

Digging into #934

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 91.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 62.20%. Comparing base (b66f2a0) to head (362c7ec).
Report is 1 commits behind head on main.

Files	Patch %	Lines
guidance/models/llama_cpp/_llama_cpp.py	91.66%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #962      +/-   ##
==========================================
+ Coverage   58.00%   62.20%   +4.19%     
==========================================
  Files          63       63              
  Lines        4848     4860      +12     
==========================================
+ Hits         2812     3023     +211     
+ Misses       2036     1837     -199     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[riedgar-ms]

I'm not very optimistic about this. This 'fix' almost certainly breaks recode() when said fix kicks in. And fundamentally, these bytes came from the LLM, so why on earth is its tokeniser refusing to have anything to do with them?

[Harsha-Nori]

Thanks so much for investigating this Richard. I agree, this feels like potentially buggy behavior directly in llama-cpp. Do we have a repro for 934 that directly uses the llama-cpp (or maybe llama-cpp-python if we need to) bindings? Instead of attempting to fix through this route, a lower level repo might mean we could/should raise an issue on llama-cpp?

[riedgar-ms]

@knilink had found:
printf '\xe6\xad' | ./llama-tokenize -m ./Meta-Llama-3-8B-Instruct.Q8_0.gguf --stdin

I have filed a llama-cpp-python-based issue (rewritten from @knilink 's investigation) directly on the HF repo whence I'm grabbing the GGUF

[riedgar-ms]

And filed the LlamaCpp bug:
ggerganov/llama.cpp#8691

[riedgar-ms]

For the record:

>>> a
'打'
>>> b
'门'
>>> a.encode()
b'\xe6\x89\x93'
>>> b.encode()
b'\xe9\x97\xa8'

So the two leading characters in the select() do not share any common bytes here.

[riedgar-ms]

I have filed a bug on the HF Hub model:
https://huggingface.co/bartowski/Meta-Llama-3-8B-Instruct-GGUF/discussions/9
But I think this is really a problem in the LlamaCpp layer