o1-preview integration / testing

[ratkinsoncinz]

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [X] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

This is certainly more of a discussion than an issue, however: Has anyone had a chance to test this project with the o1 preview? I am currently waiting for access in AOAI Studio, however the o1 deployment in OpenAI's own platform seems to give significantly improved results over 4o, and I would be very interested to see how the preview performs in AOAI.

Also - given the stated limitations, i.e.
"o1-preview model is currently in preview and do not include some features available in other models, such as image understanding and structured outputs found in the GPT-4o and GPT-4o-mini models. For many tasks, the generally available GPT-4o models may still be more suitable."

How do we expect it to perform vs 4o in this use-case?

[pamelafox]

I have not yet experimented with o1-preview, as I don't think I have access either.

I did recently run an evaluation (using https://github.com/Azure-Samples/ai-rag-chat-evaluator) on using gpt-4o-mini and it did quite well, getting high scores for groundedness, relevance, and citation match. So I'm not sure if it's actually necessary to use o1 if gpt-4o-mini does so well, but it's always worth an evaluation for your domain data. If you haven't set up an evaluation process yet, then I recommend doing so, so that you can compare the results for your data.

[chetan2309]

@pamelafox - if I have access to o1-preview then what changes I will have to make, to make it work?

[pamelafox]

From what I understand, o1 is fairly different in that it doesnt accept many parameters (like temperature) and it doesnt support function calling, so you would need to strip out the parameters and tools in the relevant files, like chatreadretrieveread.py.

More specifics from a colleague--

"Other things to remember with the new models is they don't support the typical set of chat completions parameters yet. So no setting temperature, top_p etc. The messages array also does not support the system message role, and max_tokens is now max_completions_tokens . max_tokens is now deprecated with the new models. streaming responses is also not supported with these new models.

They may also need to run pip install openai --upgrade to pull in the change that added max_completions_tokens depending on the last time they updated their API library."

See doc: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=python-secure#usage