Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for dataset name in PretokenizeRunner #438

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

keltin13
Copy link

@keltin13 keltin13 commented Mar 1, 2025

Description

PretokenizerRunner doesn't support datasets name argument. Usually you can get around this by specifying the data_dir, but supporting this argument brings the interface more in line with the datasets interface. Additionally, the column name was previously not being tracked in the metadata, and now is. Finally, a pretokenizer test was using pretokenize_runner instead of PretokenizerRunner, which was deprecated, and has now been upgraded.

Summary of updates:

  • Added optional dataset name parameter
  • Updated metadata to track dataset name and column name
  • Added test for loading dataset with name argument and checking metadata
  • Updated tests to use PretokenizeRunner instead of depricated pretokenizer_runner

Dependencies:

  • Loads new (but small) dataset from HF for tests

Fixes # (issue)

Type of change

  • New feature (non-breaking change which adds functionality)

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

You have tested formatting, typing and tests

  • I have run make check-ci to check format and linting. (you can run make format to format code if needed.)

Summary of updates:
- Added optional dataset name parameter
- Updated metadata to track dataset name and column name
- Added test for loading dataset with name argument and checking metadata
- Updated tests to use PretokenizeRunner instead of depricated pretokenizer_runner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants