This repository is the official implementation of ICLR 2024 paper: Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment.
You should clone the TextBox repository and follow its instructions.
git clone https://github.com/RUCAIBox/TextBox.git && cd TextBox
bash install.sh
You can download SPA datasets in: https://huggingface.co/datasets/RUCAIBox/Erya-dataset. You should download datasets such as xint in it and place them in the dataset folder.
After setting up the environment, you can either use FIGA model in the zero-shot scenario, or train it on your own dataset from scratch.
We have released FIGA model in: https://huggingface.co/RUCAIBox/Erya, which you can use directly as below.
from transformers import BertTokenizer, CPTForConditionalGeneration
tokenizer = BertTokenizer.from_pretrained("RUCAIBox/Erya")
model = CPTForConditionalGeneration.from_pretrained("RUCAIBox/Erya")
input_ids = tokenizer("安世字子孺，少以父任为郎。", return_tensors='pt')
input_ids.pop("token_type_ids")
pred_ids = model.generate(max_new_tokens=256, **input_ids)
print(tokenizer.batch_decode(pred_ids, skip_special_tokens=True))
To align your own model on another dataset from scratch, you can go as below.
python run_textbox.py --model=CPT --dataset=[dataset] --model_path=RUCAIBox/Erya4FT --epochs=[epoch_nums]
Please cite the following paper if you find our code or data helpful.
@article{guo2023beyond,
title={Beyond imitation: Leveraging fine-grained quality signals for alignment},
author={Guo, Geyang and Zhao, Ranchi and Tang, Tianyi and Zhao, Wayne Xin and Wen, Ji-Rong},
journal={arXiv preprint arXiv:2311.04072},
year={2023}
}