Skip to content

Commit c04b97a

Browse files
authored
Update documentation of Rust feature (#1711)
* Update documentation of Rust feature * Synchronize README.md and src/lib.rs
1 parent bdfc38b commit c04b97a

File tree

2 files changed

+4
-0
lines changed

2 files changed

+4
-0
lines changed

tokenizers/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,3 +137,5 @@ fn main() -> Result<()> {
137137
**progressbar**: The progress bar visualization is enabled by default. It might be disabled if
138138
compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios)
139139
dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar.
140+
**http**: This feature enables downloading the tokenizer via HTTP. It is disabled by default.
141+
With this feature enabled, `Tokenizer::from_pretrained` becomes accessible.

tokenizers/src/lib.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,8 @@
125125
//! **progressbar**: The progress bar visualization is enabled by default. It might be disabled if
126126
//! compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios)
127127
//! dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar.
128+
//! **http**: This feature enables downloading the tokenizer via HTTP. It is disabled by default.
129+
//! With this feature enabled, `Tokenizer::from_pretrained` becomes accessible.
128130
129131
#[macro_use]
130132
extern crate log;

0 commit comments

Comments
 (0)