Skip to content

Latest commit

 

History

History
39 lines (31 loc) · 2.18 KB

README.md

File metadata and controls

39 lines (31 loc) · 2.18 KB

T5 Encoder-only extension for Transformers

This repository contains the implementation of T5ForSequenceClassification and T5ForTokenClassification fully compatible with Transformers library. While this could be a feature from the library itself is not implemented yet, so this repository contains the code for preliminary experiments before being actually included to the library.

This implementation is inspired by EncT5: A Framework for Fine-tuning T5 as Non-autoregressive Models and A Universal Discriminator for Zero-Shot Generalization that made use of T5 encoder only.

Installation and use

You can simply install this library by running the following command:

python -m pip install git+https://github.com/osainz59/t5-encoder

To use the implemented classes you have to simply import t5_encoder along with transformers. Example:

import t5_encoder
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("google/t5-v1_1-base")
model = AutoModelForSequenceClassification.from_pretrained("google/t5-v1_1-base")

outputs = model(**tokenizer("This is a sentence to classify.", return_tensors="pt"))
print(outputs.logits)
>>> tensor([[ 0.0512, -0.0594]], grad_fn=<AddmmBackward0>)

GLUE results

Model CoLA SST2 MRPC STSb QQP MNLI QNLI RTE WNLI
RoBERTalarge 68.0 96.4 90.9 92.4 92.2 90.2/90.2 94.7 86.6 91.3
T5large 61.2 96.3 92.4 89.9 89.9 89.9/89.6 94.8 87.2 85.6
T5-Enclarge 55.0 96.1 93.3 92.7 91.4 90.5/90.4 94.7 88.8 47.9

NER results

Model CoNLL-2003 (F1)
RoBERTalarge 96.57
T5 -
T5-Enclarge 95.49

Important: Those results are obtained by a single run, for those datasets with very few examples the performance might change drastically.