pyproject.toml not in the main directory

[szalpal]

Hi,

When trying to build tokenizers for Python 3.13t, I think I noticed a problem inside xet-core.

tokenizers depends on huggingface_hub, which with recent release introduced xet-core dependency. When trying to build tokenizers from source (pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python), build fails since 2nd-order dependency (xet-core) can't be build. I believe it's because the build system expects pyproject.toml to be inside main directory. I believe the problem occurs in Python 3.13t only, since it's a version that does not support Limited API (abi3), therefore the build system can't download hf-xet package and tries to build it from scratch.

Issue that captures Python 3.13t support in tokenizers: huggingface/tokenizers#1767

May I ask for confirmation, is this a real problem and shall it be tackled?

Steps to reproduce:

1. Install Python 3.13t (e.g. using pyenv)

$ curl https://pyenv.run | bash
$ pyenv install 3.13t
$ pyenv global 3.13t

2. Build tokenizers

pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python

3. Error:

Collecting hf-xet<2.0.0,>=1.1.0 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.2.dev0)
  Downloading hf_xet-1.1.0.tar.gz (263 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [343 lines of output]
[...]
      💥 maturin failed
        Caused by: Failed to normalize python source path `python`
        Caused by: No such file or directory (os error 2)
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/pip-modern-metadata-u7qvyddu', '--interpreter', '/usr/bin/python']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-u7qvyddu --interpreter /usr/bin/python`

[julien-c]

interesting 👀

[rajatarya]

Hi @szalpal Thanks for filing this issue! The underlying issue is also captured in #255, which is that xet-core repo is not laid out for building the Python package from the root of the repo. Instead, the hf-xet Python package is built out of the hf_xet subdirectory.

Do you know how to specify a subdirectory for building when specifying a dependency for a Python package?

[rajatarya]

As a workaround, I tried this:

pip install git+https://github.com/huggingface/[email protected]#subdirectory=hf_xet
pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python

But the cargo.toml is still at 1.0.3 and so hf-xet gets built using maturin to have that package version, which then has pip try to get & build the package again, which fails.

I will work on moving the pyproject.toml to the root of the repo as a fix, and I will update our release process so the cargo.toml gets committed with the version being built.

 ✘ rajat@cantrust  ~  pip install git+https://github.com/huggingface/[email protected]#subdirectory=hf_xet
pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
Collecting git+https://github.com/huggingface/[email protected]#subdirectory=hf_xet
  Cloning https://github.com/huggingface/xet-core.git (to revision v1.1.0) to /private/var/folders/c_/wqqh5ytn6vs2wfzwhbxpckrr0000gn/T/pip-req-build-vbrbk360
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/xet-core.git /private/var/folders/c_/wqqh5ytn6vs2wfzwhbxpckrr0000gn/T/pip-req-build-vbrbk360
  Running command git checkout -q e4dca78b196d0765c3cac97c85075643077f83a8
  Resolved https://github.com/huggingface/xet-core.git to commit e4dca78b196d0765c3cac97c85075643077f83a8
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: hf-xet
  Building wheel for hf-xet (pyproject.toml) ... done
  Created wheel for hf-xet: filename=hf_xet-1.0.3-cp313-cp313t-macosx_11_0_arm64.whl size=5050523 sha256=7f01eb814131422f0c09617308dfec0c6827f23c8219a1271c75daef85552cfb
  Stored in directory: /private/var/folders/c_/wqqh5ytn6vs2wfzwhbxpckrr0000gn/T/pip-ephem-wheel-cache-7ed5ewqm/wheels/c9/14/a3/ecfda4622633d91a82d4bb39ae0135a3031a9edea211a64201
Successfully built hf-xet
Installing collected packages: hf-xet
Successfully installed hf-xet-1.0.3

[notice] A new release of pip is available: 25.0.1 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
Collecting git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
  Cloning https://github.com/huggingface/tokenizers.git to /private/var/folders/c_/wqqh5ytn6vs2wfzwhbxpckrr0000gn/T/pip-req-build-yj7t2evl
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/tokenizers.git /private/var/folders/c_/wqqh5ytn6vs2wfzwhbxpckrr0000gn/T/pip-req-build-yj7t2evl
  Resolved https://github.com/huggingface/tokenizers.git to commit 759d7aa77a8cbec14837e8d389d54b18f155eae6
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting huggingface-hub<1.0,>=0.16.4 (from tokenizers==0.21.2.dev0)
  Using cached huggingface_hub-0.31.1-py3-none-any.whl.metadata (13 kB)
Collecting filelock (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.2.dev0)
  Using cached filelock-3.18.0-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec>=2023.5.0 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.2.dev0)
  Using cached fsspec-2025.3.2-py3-none-any.whl.metadata (11 kB)
Collecting packaging>=20.9 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.2.dev0)
  Using cached packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting pyyaml>=5.1 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.2.dev0)
  Using cached pyyaml-6.0.2.tar.gz (130 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting requests (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.2.dev0)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting tqdm>=4.42.1 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.2.dev0)
  Using cached tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
Collecting typing-extensions>=3.7.4.3 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.2.dev0)
  Using cached typing_extensions-4.13.2-py3-none-any.whl.metadata (3.0 kB)
Collecting hf-xet<2.0.0,>=1.1.0 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.2.dev0)
  Using cached hf_xet-1.1.0.tar.gz (263 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      💥 maturin failed
        Caused by: python source path `python` does not exist or is invalid
        Caused by: No such file or directory (os error 2)
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/private/var/folders/c_/wqqh5ytn6vs2wfzwhbxpckrr0000gn/T/pip-modern-metadata-vmg1vd8i', '--interpreter', '/Users/rajat/.pyenv/versions/3.13.3t/bin/python3.13']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /private/var/folders/c_/wqqh5ytn6vs2wfzwhbxpckrr0000gn/T/pip-modern-metadata-vmg1vd8i --interpreter /Users/rajat/.pyenv/versions/3.13.3t/bin/python3.13`
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 25.0.1 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
 ✘ rajat@cantrust  ~ 

[rajatarya]

@szalpal

This is fixed in the hf-xet 1.1.2 release. Please verify working on your end and close, thanks!

pip install "huggingface_hub[hf_xet]"