Description

Fixes bug with PretokenizerRunner when streaming=True due to differences in interface with Dataset and IterableDataset. Specifically, num_proc is an argument to the map method for the former but not the latter. Similarly, Dataset uses set_format and IterableDataset uses with_format. Added test.

Also replaced a usage of deprecated pretokenize_runner with PretokenizeRunner in a separate test.

Fixes #441

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing tests pass locally with my changes
• I have not rewritten tests relating to key interfaces which would affect backward compatibility

You have tested formatting, typing and tests

• I have run make check-ci to check format and linting. (you can run make format to format code if needed.)