[Bug]: issue with tokenizers

[projecthorizon993]

Checklist

• The issue exists after disabling all extensions
• The issue exists on a clean installation of webui
• The issue is caused by an extension, but I believe it is caused by a bug in the webui
• The issue exists in the current version of the webui
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

can't open because of the tokenizers can't build

Steps to reproduce the problem

idk
it just pop up

What should have happened?

it run as usual but now it bonked

What browsers do you use to access the UI ?

No response

Sysinfo

from my console can't open the webUI

Console logs

Traceback (most recent call last):
  File "/workspace/file/stable-diffusion-webui/launch.py", line 48, in <module>
    main()
  File "/workspace/file/stable-diffusion-webui/launch.py", line 39, in main
    prepare_environment()
  File "/workspace/file/stable-diffusion-webui/modules/launch_utils.py", line 423, in prepare_environment
    run_pip(f"install -r \"{requirements_file}\"", "requirements")
  File "/workspace/file/stable-diffusion-webui/modules/launch_utils.py", line 144, in run_pip
    return run(f'"{python}" -m pip {command} --prefer-binary{index_url_line}', desc=f"Installing {desc}", errdesc=f"Couldn't install {desc}", live=live)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/file/stable-diffusion-webui/modules/launch_utils.py", line 116, in run
    raise RuntimeError("\n".join(error_bits))
RuntimeError: Couldn't install requirements.
Command: "/home/gitpod/.pyenv/versions/3.12.4/bin/python" -m pip install -r "requirements_versions.txt" --prefer-binary
Error code: 1

Additional information

No response

[Swinec]

+1, Gentoo linux, analog issue, installation step fail

Compiling tokenizers v0.13.3 (/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/tokenizers-lib)
           Running `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' -C metadata=89b09084cb326b58 -C extra-filename=-89b09084cb326b58 --out-dir /tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps -C strip=debuginfo -L dependency=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps --extern aho_corasick=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libaho_corasick-8ba363174647299e.rmeta --extern cached_path=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libcached_path-6bfc0561b4dbd446.rmeta --extern clap=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libclap-540591df4bacc00c.rmeta --extern derive_builder=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libderive_builder-24b0e15fc888a13f.rmeta --extern dirs=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libdirs-9b2b450d525477fc.rmeta --extern esaxx_rs=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libesaxx_rs-e2b589202c958bcf.rmeta --extern getrandom=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libgetrandom-0840ff858fccc57a.rmeta --extern indicatif=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libindicatif-387c5d1912c4bf6a.rmeta --extern itertools=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libitertools-244b09eccdfc1a09.rmeta --extern lazy_static=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/liblazy_static-70f2c43a9ded1614.rmeta --extern log=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/liblog-868b1533bc35336a.rmeta --extern macro_rules_attribute=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libmacro_rules_attribute-7457d1e82e5afc40.rmeta --extern monostate=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libmonostate-02a71e617c2c010f.rmeta --extern onig=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libonig-32f51c4c1e73388c.rmeta --extern paste=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libpaste-e36699270d1a400e.so --extern rand=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librand-c5a2a4aaba2fadc7.rmeta --extern rayon=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librayon-d407a8180c10eeb6.rmeta --extern rayon_cond=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librayon_cond-095b0ed7b7312de4.rmeta --extern regex=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libregex-079ed91e94dc338a.rmeta --extern regex_syntax=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libregex_syntax-c3ba2266c864a422.rmeta --extern reqwest=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libreqwest-7e6229659f846647.rmeta --extern serde=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libserde-7be8408fc50a07da.rmeta --extern serde_json=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libserde_json-af21dccbb02e514e.rmeta --extern spm_precompiled=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libspm_precompiled-3dde457deecc9f1c.rmeta --extern thiserror=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libthiserror-b1959683ffc698fa.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_normalization_alignments-f1efa90d7fe69706.rmeta --extern unicode_segmentation=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_segmentation-ad70de3121ea8944.rmeta --extern unicode_categories=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_categories-436bd1ea0d9dd645.rmeta -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/bzip2-sys-9f0cc99bf07a3f85/out/lib -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/zstd-sys-99695ae48306d6f1/out -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/esaxx-rs-4237ceb012f7dd82/out -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/onig_sys-43450e8faa843f01/out`
      warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:265:21
          |
      265 |                 let mut target_node = &mut best_path_ends_at[key_pos];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`
          |
          = note: `#[warn(unused_mut)]` on by default
      
      warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:282:21
          |
      282 |                 let mut target_node = &mut best_path_ends_at[starts_at + mblen];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`
      
      warning: variable does not need to be mutable
         --> tokenizers-lib/src/pre_tokenizers/byte_level.rs:200:59
          |
      200 |     encoding.process_tokens_with_offsets_mut(|(i, (token, mut offsets))| {
          |                                                           ----^^^^^^^
          |                                                           |
          |                                                           help: remove this `mut`
      
      error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
         --> tokenizers-lib/src/models/bpe/trainer.rs:526:47
          |
      522 |                     let w = &words[*i] as *const _ as *mut _;
          |                             -------------------------------- casting happend here
      ...
      526 |                         let word: &mut Word = &mut (*w);
          |                                               ^^^^^^^^^
          |
          = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
          = note: `#[deny(invalid_reference_casting)]` on by default
      
      warning: `tokenizers` (lib) generated 3 warnings
      error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted
      
      Caused by:
        process didn't exit successfully: `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' -C metadata=89b09084cb326b58 -C extra-filename=-89b09084cb326b58 --out-dir /tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps -C strip=debuginfo -L dependency=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps --extern aho_corasick=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libaho_corasick-8ba363174647299e.rmeta --extern cached_path=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libcached_path-6bfc0561b4dbd446.rmeta --extern clap=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libclap-540591df4bacc00c.rmeta --extern derive_builder=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libderive_builder-24b0e15fc888a13f.rmeta --extern dirs=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libdirs-9b2b450d525477fc.rmeta --extern esaxx_rs=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libesaxx_rs-e2b589202c958bcf.rmeta --extern getrandom=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libgetrandom-0840ff858fccc57a.rmeta --extern indicatif=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libindicatif-387c5d1912c4bf6a.rmeta --extern itertools=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libitertools-244b09eccdfc1a09.rmeta --extern lazy_static=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/liblazy_static-70f2c43a9ded1614.rmeta --extern log=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/liblog-868b1533bc35336a.rmeta --extern macro_rules_attribute=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libmacro_rules_attribute-7457d1e82e5afc40.rmeta --extern monostate=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libmonostate-02a71e617c2c010f.rmeta --extern onig=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libonig-32f51c4c1e73388c.rmeta --extern paste=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libpaste-e36699270d1a400e.so --extern rand=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librand-c5a2a4aaba2fadc7.rmeta --extern rayon=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librayon-d407a8180c10eeb6.rmeta --extern rayon_cond=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librayon_cond-095b0ed7b7312de4.rmeta --extern regex=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libregex-079ed91e94dc338a.rmeta --extern regex_syntax=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libregex_syntax-c3ba2266c864a422.rmeta --extern reqwest=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libreqwest-7e6229659f846647.rmeta --extern serde=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libserde-7be8408fc50a07da.rmeta --extern serde_json=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libserde_json-af21dccbb02e514e.rmeta --extern spm_precompiled=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libspm_precompiled-3dde457deecc9f1c.rmeta --extern thiserror=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libthiserror-b1959683ffc698fa.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_normalization_alignments-f1efa90d7fe69706.rmeta --extern unicode_segmentation=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_segmentation-ad70de3121ea8944.rmeta --extern unicode_categories=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_categories-436bd1ea0d9dd645.rmeta -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/bzip2-sys-9f0cc99bf07a3f85/out/lib -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/zstd-sys-99695ae48306d6f1/out -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/esaxx-rs-4237ceb012f7dd82/out -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/onig_sys-43450e8faa843f01/out` (exit status: 1)
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

[FournyP]

I face the same issue while installing requirements :

error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

[xangelix]

This issue "shouldn't" happen if you are using python 3.10, the highest supported version for this project. As much as I wish we had a more up to date version of tokenizers and python compatible-- we don't.

Perhaps this is caused because pypi does not have prebuilt binaries for that old of a version of tokenizers for that new of a version of python?

Repost here if python 3.10.x doesn't fix it!

[projecthorizon993]

nope
still 3.10.12 still have that bug idk what todo still tokenizer issue

[mmmpipi]

same problem on ArchLinux

[projecthorizon993]

same problem on ArchLinux

fixed the tokenizer

this one mental abuse me for over a week

Commit hash: 82a973c
Launching Web UI with arguments: --listen --theme dark --skip-torch-cuda-test --no-half --use-cpu all --share
Traceback (most recent call last):
File "/workspace/file/stable-diffusion-webui/launch.py", line 48, in
main()
File "/workspace/file/stable-diffusion-webui/launch.py", line 44, in main
start()
File "/workspace/file/stable-diffusion-webui/modules/launch_utils.py", line 465, in start
import webui
File "/workspace/file/stable-diffusion-webui/webui.py", line 13, in
initialize.imports()
File "/workspace/file/stable-diffusion-webui/modules/initialize.py", line 17, in imports
import pytorch_lightning # noqa: F401
File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/init.py", line 35, in
from pytorch_lightning.callbacks import Callback # noqa: E402
File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/callbacks/init.py", line 14, in
from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in
from pytorch_lightning.callbacks.callback import Callback
File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 25, in
from pytorch_lightning.utilities.types import STEP_OUTPUT
File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 27, in
from torchmetrics import Metric
...
rcParams['backend'] = os.environ.get('MPLBACKEND')
File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/matplotlib/init.py", line 738, in setitem
raise ValueError(f"Key {key}: {ve}") from None
ValueError: Key backend: 'module://matplotlib_inline.backend_inline' is not a valid value for backend; supported values are ['gtk3agg', 'gtk3cairo', 'gtk4agg', 'gtk4cairo', 'macosx', 'nbagg', 'notebook', 'qtagg', 'qtcairo', 'qt5agg', 'qt5cairo', 'tkagg', 'tkcairo', 'webagg', 'wx', 'wxagg', 'wxcairo', 'agg', 'cairo', 'pdf', 'pgf', 'ps', 'svg', 'template']