Correctly escaping stopping strings?

[andysalerno]

Hi there,

I noticed in the readme, and in example notebooks, a newline is escaped when used as a stopping string. Here's the example in the readme:

UPDATED
Where there is no guidance{{gen 'rewrite' stop="\\n-"}}
- GPT {{gen 'chapter'}}:{{gen 'verse'}}""")

Is it always the case that newlines should be represented as "\n" when used as a stopping string? Is there any difference between single quotes stop='\\n' and double quotes stop="\\n" in terms of escaping?

One other question. Does token healing apply to stopping strings? Because I've seen unusual behavior where the model outputs my stopping string, but keeps generating. It only happens sometimes. So it made me think, maybe for short stopping strings (like a single character; in one case I have stop='.') perhaps this appears in the middle of an encoded token, and doesn't trigger the model to stop.

[slundberg]

Good questions.

First, you escape stopping strings the same way you escape strings in Python (double escaping when you are writing the guidance code inside another string). The literals in Guidance code are parsed with Python's ast.literal_eval function, so anything that function can handle you can use:
https://github.com/microsoft/guidance/blob/d2d37949c2a552995e2be5f8c3cdd0fe7f05ea79/guidance/_program_executor.py#L184-L188

As for token healing, no. Stop strings are tested against the growing string as decoded by the decoder. For OpenAI the endpoint handles this or Guidance depending on if you are using a regex or not. For Transformers you can see the regex stopping code here:

https://github.com/microsoft/guidance/blob/d2d37949c2a552995e2be5f8c3cdd0fe7f05ea79/guidance/llms/_transformers.py#L580-L589

If you find a scenario where something does not stop like you expect, please share as an issue. Thanks!

[andysalerno]

Thanks!

Say I'm using a Transformer model locally, and not OpenAI. Does it always interpret the stop as regex?

In one of my templated prompts, I have something like this:
SEARCH_WEB({{gen 'search_query'}}), and I obviously want the model to stop if it emits a ). So I may try something like this:

SEARCH_WEB({{gen 'search_query' stop=')'}})

But if it's going to interpret that as regex, I would need to escape it, correct?

SEARCH_WEB({{gen 'search_query' stop='\)'}})

I could find my answer by just trying this out, or reading the code, but a confirmation would be great :D

[andysalerno]

Answering my own question from the code was easier than I thought (if I'm understanding the documentation correctly)

here: https://github.com/microsoft/guidance/blob/d2d37949c2a552995e2be5f8c3cdd0fe7f05ea79/guidance/library/_gen.py#LL20C1-L25C108

there's a distinction between stop and stop_regex.