New nvtext::wordpiece_tokenizer APIs

[davidwendt]

Description

Creates a new word-piece-tokenizer which replaces the existing subword-tokenizer in nvtext.
The subword-tokenizer logic is to split out and specialized to perform basic tokenizing with the word-piece logic only.
The normalizing part is already a separate API. The output will be a lists column of tokens only.

The first change is that the new API uses wordpiece instead of subword. Here are the 2 C++ API declarations:

std::unique_ptr<wordpiece_vocabulary> load_wordpiece_vocabulary(
  cudf::strings_column_view const& input,
  rmm::cuda_stream_view stream,
  rmm::device_async_resource_ref mr);

The vocabulary is loaded as a strings column and the returned object can be used on multiple calls to the next API:

std::unique_ptr<cudf::column> wordpiece_tokenize(
  cudf::strings_column_view const& input,
  wordpiece_vocabulary const& vocabulary,
  cudf::size_type max_words_per_row,
  rmm::cuda_stream_view stream,
  rmm::device_async_resource_ref mr);

This will return a lists column of integers which represent the tokens for each row. The max_words_per_row will stop the tokenizing process for each row once the number of input words (characters delimited by space) has been reached. This means you may get more tokens than max_words_per_row for a row if a single word produces multiple tokens.

Note, that this API expects the input string to already be normalized -- processed by the nvtext::normalize_characters API which is also being reworked in #17818

The Python interface has the following pattern:

from cudf.core.wordpiece_tokenize import WordPieceVocabulary

input_string = .... # output of the normalizer
vocab_file = os.path.join(datadir, "bert_base_cased_sampled/vocab.txt")
vc = cudf.read_text(vocab_file, delimiter="\n", strip_delimiters=True)
wpt = WordPieceVocabulary(vc)
wpr = wpt.tokenize(input_string)

The output is a lists column of the tokens and no longer the tensor-data and metadata format.
If this format is needed, then we can consider a 3rd API that converts the output here to that format.

Closes #17507

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[davidwendt]

/ok to test

[davidwendt]

/ok to test

[davidwendt]

/ok to test

[davidwendt]

Adding reference here to issue #12403 as well.
This may be closed in a future PR when the current subword-tokenizer API is deprecated.