Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the example script for data pretokenization #101

Merged
merged 4 commits into from
Apr 30, 2024
Merged

Conversation

mryab
Copy link
Member

@mryab mryab commented Apr 18, 2024

This PR provides a minimal example of how to tokenize data locally before submitting it to the Finetuning API. The script supports loss masking (for padding tokens in this case) by adding the labels field.

Running to get the dataset with just labels and the attention mask:

python examples/tokenize_data.py

Running with loss labels as well (padding tokens will have an index of -100 and will be ignored in the loss):

python examples/tokenize_data.py --add-labels --out-filename data_with_labels.parquet

I checked that both together files check and together files upload work with the outputs, the filetype is reported to be Parquet.

TODO:

  • Adding a reference in README/website docs
  • Checking everything through the API

@mryab mryab requested a review from orangetin April 18, 2024 11:39
@mryab mryab changed the title Add example of data pretokenization Add the example script for data pretokenization Apr 19, 2024
@mryab mryab marked this pull request as ready for review April 19, 2024 15:12
@mryab mryab requested a review from azahed98 April 19, 2024 15:12
@mryab mryab requested review from Nutlope and removed request for orangetin April 24, 2024 10:37
Copy link
Contributor

@Nutlope Nutlope left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Contributor

@azahed98 azahed98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@orangetin orangetin merged commit 236edca into main Apr 30, 2024
6 of 11 checks passed
@orangetin orangetin deleted the add-parquet-examples branch April 30, 2024 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants