Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem whith **tokenizers-0.13.4rc3** compiling Windows 10 Environment 22H2 #819

Open
amadoui opened this issue Sep 22, 2024 · 2 comments
Open

Problem whith **tokenizers-0.13.4rc3** compiling Windows 10 Environment 22H2 #819

amadoui opened this issue Sep 22, 2024 · 2 comments

Comments

@amadoui
Copy link

amadoui commented Sep 22, 2024

I got a problem whith tokenizers-0.13.4rc3 compiling
I am trying on compile following the guide with miniconda on Windows 10 Environment 22H2 (OS Build 19045.4894)
I have the following version installed in miniconda venv:

  • conda 24.7.1
  • Python 3.10.14 (tried with Python 3.12 / tried with Python 3.11)
  • rustc 1.74.1 (a28077b28 2023-12-04) (tried with rustc 1.76 and failed / tried with rustc 1.72.1 and cannot compile because rustc 1.74 is required)
   Compiling tokenizers v0.13.4 (C:\Users\user\AppData\Local\Temp\easy_install-urg_k0xs\tokenizers-0.13.4rc3\tokenizers-lib)
error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
   --> tokenizers-lib\src\models\bpe\trainer.rs:541:47
    |
537 |                     let w = &words[*i] as *const _ as *mut _;
    |                             -------------------------------- casting happend here
...
541 |                         let word: &mut Word = &mut (*w);
    |                                               ^^^^^^^^^
    |
    = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
    = note: `#[deny(invalid_reference_casting)]` on by default

error: could not compile `tokenizers` (lib) due to previous error
warning: build failed, waiting for other jobs to finish...
Traceback (most recent call last):
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 252, in build_extension
    cargo_messages = subprocess.check_output(
  File "C:\Users\user\.conda\envs\tortoise\lib\subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Users\user\.conda\envs\tortoise\lib\subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'rustc', '--lib', '--message-format=json-render-diagnostics', '--manifest-path', 'Cargo.toml', '--release', '--features', 'pyo3/extension-module', '--crate-type', 'cdylib', '--']' returned non-zero exit status 101.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 165, in save_modules
    yield saved
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 207, in setup_context
    yield
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 268, in run_setup
    _execfile(setup_script, ns)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 50, in _execfile
    exec(code, globals, locals)
  File "C:\Users\user\AppData\Local\Temp\easy_install-urg_k0xs\tokenizers-0.13.4rc3\setup.py", line 13, in <module>
    long_description=long_description,
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 183, in setup
    return run_commands(dist)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 199, in run_commands
    dist.run_commands()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\bdist_egg.py", line 168, in run
    cmd = self.call_command('install_lib', warn_dir=False)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\bdist_egg.py", line 154, in call_command
    self.run_command(cmdname)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\install_lib.py", line 19, in run
    self.build()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\command\install_lib.py", line 110, in build
    self.run_command('build_ext')
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\setuptools_ext.py", line 171, in run
    build_rust.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\command.py", line 100, in run
    self.run_for_extension(ext)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 138, in run_for_extension
    dylib_paths = self.build_extension(ext, self.target)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 261, in build_extension
    raise CompileError(format_called_process_error(e, include_stdout=False))
distutils.errors.CompileError: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release --features pyo3/extension-module --crate-type cdylib --` failed with code 101

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Windows\System32\tortoise-tts\setup.py", line 6, in <module>
    setuptools.setup(
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 183, in setup
    return run_commands(dist)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 199, in run_commands
    dist.run_commands()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\install.py", line 97, in run
    self.do_egg_install()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\install.py", line 158, in do_egg_install
    cmd.run(show_deprecation=False)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 439, in run
    self.easy_install(spec, not self.no_deps)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 689, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 739, in install_item
    self.process_distribution(spec, dist, deps)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 788, in process_distribution
    distros = WorkingSet([]).resolve(
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\pkg_resources\__init__.py", line 897, in resolve
    dist = self._resolve_dist(
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\pkg_resources\__init__.py", line 933, in _resolve_dist
    dist = best[req.key] = env.best_match(
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\pkg_resources\__init__.py", line 1271, in best_match
    return self.obtain(req, installer)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\pkg_resources\__init__.py", line 1307, in obtain
    return installer(requirement) if installer else None
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 712, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 737, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 934, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 1206, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 1192, in run_setup
    run_setup(setup_script, args)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 258, in run_setup
    with setup_context(setup_dir):
  File "C:\Users\user\.conda\envs\tortoise\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 199, in setup_context
    with save_modules():
  File "C:\Users\user\.conda\envs\tortoise\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 178, in save_modules
    saved_exc.resume()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 152, in resume
    raise exc.with_traceback(self._tb)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 165, in save_modules
    yield saved
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 207, in setup_context
    yield
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 268, in run_setup
    _execfile(setup_script, ns)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 50, in _execfile
    exec(code, globals, locals)
  File "C:\Users\user\AppData\Local\Temp\easy_install-urg_k0xs\tokenizers-0.13.4rc3\setup.py", line 13, in <module>
    long_description=long_description,
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 183, in setup
    return run_commands(dist)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 199, in run_commands
    dist.run_commands()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\bdist_egg.py", line 168, in run
    cmd = self.call_command('install_lib', warn_dir=False)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\bdist_egg.py", line 154, in call_command
    self.run_command(cmdname)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\install_lib.py", line 19, in run
    self.build()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\command\install_lib.py", line 110, in build
    self.run_command('build_ext')
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\setuptools_ext.py", line 171, in run
    build_rust.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\command.py", line 100, in run
    self.run_for_extension(ext)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 138, in run_for_extension
    dylib_paths = self.build_extension(ext, self.target)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 261, in build_extension
    raise CompileError(format_called_process_error(e, include_stdout=False))
setuptools.sandbox.UnpickleableException: CompileError('`cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release --features pyo3/extension-module --crate-type cdylib --` failed with code 101')

(tortoise) C:\Windows\System32\tortoise-tts>

Any ideas about how to compile tokenizers-0.13.4rc3 succefully on Windows 10?
@kleethesama
Copy link

This #816 (comment) fixed it for me.

@amadoui
Copy link
Author

amadoui commented Sep 23, 2024

@kleethesama tried your suggestion but i got the same issue when compiling "tokenizers-0.13.4rc3", i suspect the rustc compiler is not suitable for this task but cannot figure out how to deal with it

Compiling tokenizers v0.13.4 (C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\tokenizers-lib)
           Running `rustc --crate-name tokenizers --edition=2018 tokenizers-lib\src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"cached-path\"" --cfg "feature=\"clap\"" --cfg "feature=\"cli\"" --cfg "feature=\"default\"" --cfg "feature=\"dirs\"" --cfg "feature=\"esaxx_fast\"" --cfg "feature=\"http\"" --cfg "feature=\"indicatif\"" --cfg "feature=\"onig\"" --cfg "feature=\"progressbar\"" --cfg "feature=\"reqwest\"" -C metadata=571339cac8db871d -C extra-filename=-571339cac8db871d --out-dir C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps -L dependency=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps --extern aho_corasick=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libaho_corasick-6e1ceb2730930a08.rmeta --extern cached_path=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libcached_path-b7c63fc96b7b8a96.rmeta --extern clap=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libclap-84048ff9c95fb4cc.rmeta --extern derive_builder=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libderive_builder-2b0b7bf261866a03.rmeta --extern dirs=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libdirs-49d8e5d94e87fa5e.rmeta --extern esaxx_rs=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libesaxx_rs-48bffd32ccaeb3f5.rmeta --extern getrandom=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libgetrandom-0d5f94c7c56cebb1.rmeta --extern indicatif=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libindicatif-90913102d1e1fe0f.rmeta --extern itertools=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libitertools-a824d246a521d621.rmeta --extern lazy_static=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\liblazy_static-90ac5ec4ca579963.rmeta --extern log=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\liblog-e72785d52d978e26.rmeta --extern macro_rules_attribute=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmacro_rules_attribute-c5ccec30f814a496.rmeta --extern monostate=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmonostate-f82384f188902be6.rmeta --extern onig=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libonig-0a91cd76d2a7fb72.rmeta --extern paste=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\paste-0469fbd59fc2e04c.dll --extern rand=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librand-bbef81b04dff903c.rmeta --extern rayon=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librayon-0692a0d06ad1db6b.rmeta --extern rayon_cond=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librayon_cond-7ec684bc4c5c804d.rmeta --extern regex=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libregex-5553ea54d73d96cc.rmeta --extern regex_syntax=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libregex_syntax-7f4550e60d4205c7.rmeta --extern reqwest=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libreqwest-721a7e7c36e12898.rmeta --extern serde=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libserde-21a5e9db8ed00d2f.rmeta --extern serde_json=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libserde_json-8457fa567cc6f19d.rmeta --extern spm_precompiled=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libspm_precompiled-2071f0bb46ce086f.rmeta --extern thiserror=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libthiserror-1ea9a51dd83679bd.rmeta --extern unicode_normalization_alignments=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_normalization_alignments-e02e49ff2fe58290.rmeta --extern unicode_segmentation=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_segmentation-10977e716c0679c6.rmeta --extern unicode_categories=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_categories-a7fb754aa5a4a1a4.rmeta -L native=C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.52.6\lib -L native=C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.48.5\lib -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\bzip2-sys-d909296b23c64ab5\out\lib -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\zstd-sys-ca9050e4fbf40534\out -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\esaxx-rs-d8450fba8101abff\out -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\onig_sys-84a28246dad559a1\out`
           Running `rustc --crate-name ndarray --edition=2018 C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\ndarray-0.13.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=0a691bf027808050 -C extra-filename=-0a691bf027808050 --out-dir C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps -L dependency=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps --extern matrixmultiply=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmatrixmultiply-05b30d9aee34739d.rmeta --extern num_complex=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libnum_complex-5aefd29d72e65d50.rmeta --extern num_integer=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libnum_integer-da28bd3984fda6f1.rmeta --extern num_traits=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libnum_traits-d946bd1460732915.rmeta --extern rawpointer=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librawpointer-5e716fe4b452057e.rmeta --cap-lints allow`
      error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
         --> tokenizers-lib\src\models\bpe\trainer.rs:541:47
          |
      537 |                     let w = &words[*i] as *const _ as *mut _;
          |                             -------------------------------- casting happend here
      ...
      541 |                         let word: &mut Word = &mut (*w);
          |                                               ^^^^^^^^^
          |
          = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
          = note: `#[deny(invalid_reference_casting)]` on by default

      error: could not compile `tokenizers` (lib) due to previous error

      Caused by:
        process didn't exit successfully: `rustc --crate-name tokenizers --edition=2018 tokenizers-lib\src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"cached-path\"" --cfg "feature=\"clap\"" --cfg "feature=\"cli\"" --cfg "feature=\"default\"" --cfg "feature=\"dirs\"" --cfg "feature=\"esaxx_fast\"" --cfg "feature=\"http\"" --cfg "feature=\"indicatif\"" --cfg "feature=\"onig\"" --cfg "feature=\"progressbar\"" --cfg "feature=\"reqwest\"" -C metadata=571339cac8db871d -C extra-filename=-571339cac8db871d --out-dir C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps -L dependency=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps --extern aho_corasick=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libaho_corasick-6e1ceb2730930a08.rmeta --extern cached_path=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libcached_path-b7c63fc96b7b8a96.rmeta --extern clap=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libclap-84048ff9c95fb4cc.rmeta --extern derive_builder=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libderive_builder-2b0b7bf261866a03.rmeta --extern dirs=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libdirs-49d8e5d94e87fa5e.rmeta --extern esaxx_rs=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libesaxx_rs-48bffd32ccaeb3f5.rmeta --extern getrandom=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libgetrandom-0d5f94c7c56cebb1.rmeta --extern indicatif=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libindicatif-90913102d1e1fe0f.rmeta --extern itertools=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libitertools-a824d246a521d621.rmeta --extern lazy_static=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\liblazy_static-90ac5ec4ca579963.rmeta --extern log=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\liblog-e72785d52d978e26.rmeta --extern macro_rules_attribute=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmacro_rules_attribute-c5ccec30f814a496.rmeta --extern monostate=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmonostate-f82384f188902be6.rmeta --extern onig=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libonig-0a91cd76d2a7fb72.rmeta --extern paste=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\paste-0469fbd59fc2e04c.dll --extern rand=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librand-bbef81b04dff903c.rmeta --extern rayon=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librayon-0692a0d06ad1db6b.rmeta --extern rayon_cond=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librayon_cond-7ec684bc4c5c804d.rmeta --extern regex=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libregex-5553ea54d73d96cc.rmeta --extern regex_syntax=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libregex_syntax-7f4550e60d4205c7.rmeta --extern reqwest=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libreqwest-721a7e7c36e12898.rmeta --extern serde=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libserde-21a5e9db8ed00d2f.rmeta --extern serde_json=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libserde_json-8457fa567cc6f19d.rmeta --extern spm_precompiled=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libspm_precompiled-2071f0bb46ce086f.rmeta --extern thiserror=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libthiserror-1ea9a51dd83679bd.rmeta --extern unicode_normalization_alignments=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_normalization_alignments-e02e49ff2fe58290.rmeta --extern unicode_segmentation=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_segmentation-10977e716c0679c6.rmeta --extern unicode_categories=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_categories-a7fb754aa5a4a1a4.rmeta -L native=C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.52.6\lib -L native=C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.48.5\lib -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\bzip2-sys-d909296b23c64ab5\out\lib -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\zstd-sys-ca9050e4fbf40534\out -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\esaxx-rs-d8450fba8101abff\out -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\onig_sys-84a28246dad559a1\out` (exit code: 1)
      warning: build failed, waiting for other jobs to finish...
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Successfully built tortoise-tts
Failed to build tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@amadoui @kleethesama