How to custom <|SYS|> contents when using API? (after deployed)

[maxchiron]

lmdeploy provides a powerful API called /v1/chat/interactive which let request carry:

• prompt
• temperature
• ...
  But not the <|SYS|> part contents.
  I want to know is there any way to modify the <|SYS|> part contents? Just like what --system does in xtuner chat.
  

so I can make sure it is the same as in my fine-tuning datas.
Thans very much.

[lvhan028]

@AllentDan please do the favor

[AllentDan]

Just pass the system arg when starting the api_server. Besides, if you pass a prompt in the openai messages format, you can directly pass a system role with its content.

#870

[maxchiron]

Thanks for the detailed reply. Just want to make sure and HOPE my understanding is right:

1. When you say pass the system arg when starting api_server, my understanding is:
   lmdeploy serve api_server ./workspace 0.0.0.0 --server_port ${server_port} --instance_num 64 --tp 1 --system "You are a ... bot."
2. When you say using openai message format, my understanding is:

• for /v1/chat/completions:

from lmdeploy.serve.openai.api_client import APIClient
api_client = APIClient('http://{server_ip}:{server_port}')
model_name = api_client.available_models[0]

messages = [
  
  {"role": "system", 
   "content": "You are a ... bot."}

  {"role": "user", 
   "content": "Say this is a test!"}
]

for item in api_client.chat_completions_v1(model=model_name, messages=messages):
    print(item)

• but for /v1/completions:
  
  Where should the "system" contents put? Afterall only a "prompt" string is there.

---

---

and BTW, if I set --system and request with 'system' simultaneously, which one will on the top? Thanks again for your kind distribution. 👍

[AllentDan]

/v1/completions is a text completion interface which means, for conversation, you have to contact the messages to a string and then pass it in.
request is on the top.

[maxchiron]

Thanks for the reply~
The /v1/chat/completions method works fine for sending 'system' contents,
while --system method seems not working. (To avoid confussion, I must emphasis that the screenshot above is the xtuner chat help, not lmdeploy.)
Thanks again~

[AllentDan]

What arg depends on model.py. If it is InternLM, it should be --meta_instruction

[maxchiron]

What arg depends on model.py. If it is InternLM, it should be --meta_instruction

@AllentDan
I set --meta_instruction as lmdeploy serve api_server ...... --meta_instruction "You are a bot.", and the result seems the --meta_instruction contents not working.
Hence, the only way to set SYS contents drops to using /v1/chat/completions, but this api seems not supporting setting seed. Is there any way to control SYS contents and seed at the same time? Thanks.

[AllentDan]

In my testing, the --meta_instruction works fine.

As for seed, you may seed everything for the server.
Some codes like this:

def seed_everything(seed: int):
    import random, os
    import numpy as np
    import torch
    
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = True