text generation webui_en

Inference with text-generation-webui

A gradio web UI for running Large Language Models like LLaMA, llama.cpp and so on.

Step 1: install text-generation-webui

text-generation-webui currently supports Windows/Linux/macOS/WSLsystems. Refer to webui installation for more details.

Step 2: Preparing model weights

Copy the full weights of chinese-alpaca-2 to the text-generation-webui models folder, as shown below

>>> ls models/chinese-alpaca-2-7b
config.json
generation_config.json
pytorch_model-00001-of-00002.bin
pytorch_model-00002-of-00002.bin
pytorch_model.bin.index.json
special_tokens_map.json
tokenizer_config.json
tokenizer.json
tokenizer.model

Step 3: Load the model and start the webui

Run the following command to talk to chinese-alpaca-2.

python server.py --model chinese-alpaca-2-7b --chat

In the open web page, select Chat setting-Instruction -> template successively, and select Llama-v2 in Instruction template. And replace "Answer the questions." prompt in the Context box with "You are a helpful assistant. You are a helpful assistant." Finally, return to the Text generation interface, enter your instruction in the input box, you can talk with chinese-alpaca-2.

Please refer to webui using docs for more detailed official instructions. If you encounter installation or running problems, please go to the original repo to ask questions, the process has been run under commit-id 32a2bbe