Skip to content

A extension of Transformers library to include T5ForSequenceClassification class.

License

Notifications You must be signed in to change notification settings

osainz59/t5-encoder

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

T5 Encoder-only extension for Transformers

This repository contains the implementation of T5ForSequenceClassification and T5ForTokenClassification fully compatible with Transformers library. While this could be a feature from the library itself is not implemented yet, so this repository contains the code for preliminary experiments before being actually included to the library.

This implementation is inspired by EncT5: A Framework for Fine-tuning T5 as Non-autoregressive Models and A Universal Discriminator for Zero-Shot Generalization that made use of T5 encoder only.

Installation and use

You can simply install this library by running the following command:

python -m pip install git+https://github.com/osainz59/t5-encoder

To use the implemented classes you have to simply import t5_encoder along with transformers. Example:

import t5_encoder
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("google/t5-v1_1-base")
model = AutoModelForSequenceClassification.from_pretrained("google/t5-v1_1-base")

outputs = model(**tokenizer("This is a sentence to classify.", return_tensors="pt"))
print(outputs.logits)
>>> tensor([[ 0.0512, -0.0594]], grad_fn=<AddmmBackward0>)

GLUE results

Model CoLA SST2 MRPC STSb QQP MNLI QNLI RTE WNLI
RoBERTalarge 68.0 96.4 90.9 92.4 92.2 90.2/90.2 94.7 86.6 91.3
T5large 61.2 96.3 92.4 89.9 89.9 89.9/89.6 94.8 87.2 85.6
T5-Enclarge 55.0 96.1 93.3 92.7 91.4 90.5/90.4 94.7 88.8 47.9

NER results

Model CoNLL-2003 (F1)
RoBERTalarge 96.57
T5 -
T5-Enclarge 95.49

Important: Those results are obtained by a single run, for those datasets with very few examples the performance might change drastically.

About

A extension of Transformers library to include T5ForSequenceClassification class.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages