text generation webui_en

Inference with text-generation-webui

A gradio web UI for running Large Language Models like LLaMA, llama.cpp and so on.

Step 1: install text-generation-webui

Text-generation-webui currently supports Windows/Linux/macOS/WSL systems. Refer to webui installation for more details.

Step 2: Preparing model weights

Copy the full weights of chinese-alpaca-2 to the text-generation-webui models folder, as shown below:

>>> ls models/chinese-alpaca-2-7b
config.json
generation_config.json
pytorch_model-00001-of-00002.bin
pytorch_model-00002-of-00002.bin
pytorch_model.bin.index.json
special_tokens_map.json
tokenizer_config.json
tokenizer.json
tokenizer.model

Step 3: Load the model and start the webui

Run the following command to talk to chinese-alpaca-2.

python server.py --model chinese-alpaca-2-7b --chat

In the open web page, select Chat setting -> Instruction template in order, then in Instruction template, choose Llama-v2 from the drop-down menu. Replace the prompt Answer the questions. in the Context input box with You are a helpful assistant. 你是一个乐于助人的助手。 Finally, go back to the Text generation interface, type your instruction in the input box, and you can start a conversation with chinese-alpaca-2.

Please refer to webui using docs for more detailed official instructions. If you encounter installation or running problems, please go to the original repo to ask questions, the process has been run under commit-id 32a2bbe.