fix: invalid identifiers

[j4k0xb]

Closes #117
Also related to #111

[0xdevalias]

Closes #117

I would say this is a partial fix/workaround to #117, though I think there is probably a better higher level fix (that can be implemented in another PR) that I would sort of suggest should be done before we consider #117 closed entirely:

The proper fix would be to use toIdentifier from @babel/types

@j4k0xb @jehna From a 2sec search I couldn't find rendered docs, but here's the relevant source for each:

• isValidIdentifier: https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/validators/isValidIdentifier.ts#L7-L25
• toIdentifier: https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/converters/toIdentifier.ts#L4-L26

toIdentifier definitely seems like a more robust approach than the current 'prefix with _' approach for sure.

Though I wonder if a 'proper fix' should also involve tweaking how we prompt for/filter the suggestions coming back from the LLM itself as well. Like instead of just forcing an invalid suggestion to be valid (with toIdentifier), we could detect that it's invalid (with isValidIdentifier) and then provide that feedback to the LLM, asking it to give a new suggestion; probably with some max retry limit; after which we could fall back to using the invalid suggestion run through toIdentifier, or log a warning and leave it un-renamed or similar.

Originally posted by @0xdevalias in #117 (comment)

---

Also related to #111

For context/continuity, while it's related to the 2nd part of the bugs identified in #111, your other fix PR is related to the original error on it:

• fix: empty code error #134

[jehna]

@0xdevalias good point! I opened another issue for that implementation at #147

[j4k0xb]

humanify/src/plugins/local-llm-rename/unminify-variable-name.ts

Line 25 in 73f0424

gbnf`A good name would be '${/[a-zA-Z] [a-zA-Z0-9]{2,12}/}'`

maybe because the regex has a space
and would ^ $ make a difference?

[jehna]

This regex is actually a representation of gbnf, which is a superset of regex that allows strings edit: spaces in between. If it were a normal regex, all variable names should have a space in them, right?

To me this seems like an issue with the llama.cpp gbnf handling

[jehna]

Another thing I'll need to check from the previous issues is that whether the users were using local or openai mode. Local mode is much more robust, openai does not have any actual guarantees about returning a valid identifier name.

[0xdevalias]

@jehna From memory, for at least some of the ones I was looking at/debugging, I believe they were using openai, not local.