Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hardcoded host localhost and port 9090 for a rate monitor #3531

Open
surak opened this issue Sep 20, 2024 · 1 comment
Open

Hardcoded host localhost and port 9090 for a rate monitor #3531

surak opened this issue Sep 20, 2024 · 1 comment

Comments

@surak
Copy link
Collaborator

surak commented Sep 20, 2024

In gradio_web_server.py, there is a hardcoded host and port for a supposed monitor daemon, but no daemon is around:

monitor_url = "http://localhost:9090"

This, in turn, breaks the usage of openAI endpoints with the --register to a json file.

So, for example, this openai_compatible_server.json can't run:

{
  "Llama 405": {
    "model_name": "llama3.1:405b",
    "api_type": "openai",
    "api_base": "http://localhost:11434/v1",
    "api_key": "",
    "anony_only": false,
    "recommended_config": {
      "temperature": 0.7,
      "top_p": 1.0
    }
  }
}

Because it will always fail with a CONNECTION REFUSED since we have no such monitor daemon:

2024-09-19 21:31:47 | INFO | gradio_web_server | bot_response. ip: 127.0.0.1
2024-09-19 21:31:47 | INFO | gradio_web_server | monitor error: HTTPConnectionPool(host='localhost', port=9090): Max retries exceeded with url: /is_limit_reached?model=Llama%20405%20on%20WestAI&user_id=127.0.0.1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x119061610>: Failed to establish a new connection: [Errno 61] Connection refused'))
@infwinston
Copy link
Member

Sorry this is not well documented, and we should make it optional. Current solution would be launching this monitor server.
https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/call_monitor.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants