gpt-3.5-turbo behaving differently

[fjrdomingues]

Openai must have changed something on the model and now the model "refuses" to answer in JSON.
Also tested on chatgpt, with the same result.
Before it was working almost 100% of the time.

We may be able to solve this by using the "gpt-3.5-turbo-0301" version of the model. Haven't tested it yet.

[fjrdomingues]

This doesn't seem to be a problem using the new implementation with langchain output format, which is great 👍 Still a warning that model can change, causing unpredictable results. This is why openai launches versions of the models like gpt-3.5-turbo-0301 https://platform.openai.com/docs/models/gpt-3-5

[hatkyinc2]

I've just run a couple of tests on my end I have no issues generating JSON in either chatGPT or playground gpt-3.5-turbo nor the gpt-3.5-turbo-0301

I can see how this could be a problem, generally, any 3rd party agent can change anything we count on at any time, which is one reason I like the local models direction, it's a frozen state.

🤷
Sounds like gpt-3.5-turbo-0301 will be short-lived, but it might be a reminder to generalize the model support more, so it's easy to test. Currently, the biggest hardcoded part is the pricing I think.

[hatkyinc2]

Please add a unit test to cover this issue if it's still relevant @fjrdomingues
or should we just close this for now?