Fine tune vit5-base model for text summarization

[MinhDang685]

Hello VietAI team,

Thanks for sharing the pretrained models in your research paper. I am interested on fine tuning the VietAI/vit5-base language model for the abstractive summarization task. I have some questions:

1. When I run your example here, unlike @r1ckC139 in Model Checkpoint viT5-base #1 that he has random sequences, I always got a fixed-length (=max_length) unchanged array, I have try to modify the input (with "vi: " / "vietnews: " prefix, and without prefix) but the result is not changed. Could you take a look?
   
2. In the fine tuning phase, do I need to preprocess the data by adding a prefix?

Thanks a lot

[justinphan3110]

Hi @MinhDang685 , for MLM pretraining we used mesh-tensorflow. The models on HuggingFace are ready for finetuning only.

You don't need to add prefix in finetunning.

[MinhDang685]

Hi @justinphan3110, thanks for your quick reply.

• Since the model is trained with mesh-tensorflow, can I directly use it to finetune in PyTorch without any adaptation?
• Could you give this issue (that the model always generates an unchanged sequence) a look when you have time.

Thanks

[justinphan3110]

@MinhDang685

• We have just published an example code for finetunning with huggingface

• Can you double-check again if there is still a generated unchanged sequence issue?

[MinhDang685]

Hi @justinphan3110, thanks for your help, I try to generate with the model again and it works now, the output sequences now changes base on the input

I notice that you have updated the model config.json file by removing task specific prefixes, is it the cause of the issue (that I miss the "summarization" prefix before the input to indicate I want the model to perform summarization task)?

[justinphan3110]

@MinhDang685 ,
You need prefix vietnews: for VietAI/vit5-large-vietnews-summarization .
For VietAI/vit5-base-vietnews-summarization you don't need any prefix.

You can have a look over the eval scripts with HuggingFace

[MinhDang685]

thank you @justinphan3110 for pointing that out