Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable the use of self-hosted LLMs #36

Merged
merged 12 commits into from
Aug 8, 2024

Conversation

artivis
Copy link
Contributor

@artivis artivis commented Aug 1, 2024

This PR introduces several fixes and features that enable the use of other, open-ai api compatible, LLMs such as those served by the Ollama framework.

Most notably:

  • Make use of the api endpoint
  • Add a --dry-run option to exec that prints the answer without executing it
  • Enable setting up the system prompts through env var & cli arg

@fujitatomoya
Copy link
Owner

@artivis thanks 👍 you are the 1st contributor 🥇 i will definitely look at the fix.

fujitatomoya
fujitatomoya previously approved these changes Aug 8, 2024
Copy link
Owner

@fujitatomoya fujitatomoya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this actually is a part of #21.

what this is trying to do is to be agnostic from some OpenAI default and required setting so that https://ollama.com/ can be configured and used as backend AI system.

for doing this, IMO we can take this fix to support but there are things need to be done.

  • Documentation update. all of the docs are currently related to OpenAI, documentation needs to be more generic and supported AI backend system.
  • Refactor code base to be agnostic from OpenAI only. https://ollama.com/ can be a good example for this.

ros2ai/api/config.py Show resolved Hide resolved
ros2ai/api/config.py Show resolved Hide resolved
ros2ai/verb/status.py Outdated Show resolved Hide resolved
@fujitatomoya
Copy link
Owner

@artivis i added the minor patch to fix the problem with verification.sh script. after building, verification for all container images, i will take this in.

@fujitatomoya fujitatomoya merged commit ad04c11 into fujitatomoya:rolling Aug 8, 2024
4 checks passed
@artivis artivis deleted the feat/ollama branch August 9, 2024 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants