I am not able to install transformers python module in my Chaquopy android studio.

[baponkar]

Chaquopy 14.0.2

Python 3.8.10

Gradle 7.5

Android Studio 2022.1.1

Relevant parts of your build.gradle file

Inside of build.gradle

android {
    namespace 'com.baponkar.objectdetector'
    compileSdk 33

    defaultConfig {
        applicationId "com.example.chaquopy_130application"
        minSdk 24
        targetSdk 33
        versionCode 1
        versionName "1.0"

        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"

        ndk {
            abiFilters "armeabi-v7a", "arm64-v8a", "x86", "x86_64"
        }

        python {
            version "3.8"
            //buildPython
            //"C:\\Users\\Bapon\\AppData\\Local\\Programs\\Python\\Python38\\python.exe"
        }
python {
            pip {
                install "opencv-contrib-python-headless"
                install "Keras==2.3.1"
                install "tensorflow"
                install "numpy"
                install "pillow"
                install "transformers"
            }
        }

I also have tried with include

python {
            pip {
                install "opencv-contrib-python-headless"
                install "Keras==2.3.1"
                install "tensorflow"
                install "numpy"
                install "pillow"
                install "safetensors"
                install "setuptools_rust"
                install "transformers"
            }
        }

Describe your issue

When I try to install transformers then build failed with this error:

  Building wheel for tokenizers (PEP 517): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: 'C:\Users\Bapon\AppData\Local\Programs\Python\Python38\python.exe' 'B:\AndroidStudioProjects\ObjectDetector\app\build\generated\python\bp\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\Bapon\AppData\Local\Temp\tmpxbw8q5pg'
       cwd: C:\Users\Bapon\AppData\Local\Temp\pip-install-gggr06au\tokenizers
  Complete output (51 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-cpython-38
  creating build\lib.win-amd64-cpython-38\tokenizers
  copying py_src\tokenizers\__init__.py -> build\lib.win-amd64-cpython-38\tokenizers
  creating build\lib.win-amd64-cpython-38\tokenizers\models
  copying py_src\tokenizers\models\__init__.py -> build\lib.win-amd64-cpython-38\tokenizers\models
  creating build\lib.win-amd64-cpython-38\tokenizers\decoders
  copying py_src\tokenizers\decoders\__init__.py -> build\lib.win-amd64-cpython-38\tokenizers\decoders
  creating build\lib.win-amd64-cpython-38\tokenizers\normalizers
  copying py_src\tokenizers\normalizers\__init__.py -> build\lib.win-amd64-cpython-38\tokenizers\normalizers
  creating build\lib.win-amd64-cpython-38\tokenizers\pre_tokenizers
  copying py_src\tokenizers\pre_tokenizers\__init__.py -> build\lib.win-amd64-cpython-38\tokenizers\pre_tokenizers
  creating build\lib.win-amd64-cpython-38\tokenizers\processors
  copying py_src\tokenizers\processors\__init__.py -> build\lib.win-amd64-cpython-38\tokenizers\processors
  creating build\lib.win-amd64-cpython-38\tokenizers\trainers
  copying py_src\tokenizers\trainers\__init__.py -> build\lib.win-amd64-cpython-38\tokenizers\trainers
  creating build\lib.win-amd64-cpython-38\tokenizers\implementations
  copying py_src\tokenizers\implementations\base_tokenizer.py -> build\lib.win-amd64-cpython-38\tokenizers\implementations
  ERROR: Command errored out with exit status 1:

  copying py_src\tokenizers\implementations\bert_wordpiece.py -> build\lib.win-amd64-cpython-38\tokenizers\implementations
  copying py_src\tokenizers\implementations\byte_level_bpe.py -> build\lib.win-amd64-cpython-38\tokenizers\implementations
  copying py_src\tokenizers\implementations\char_level_bpe.py -> build\lib.win-amd64-cpython-38\tokenizers\implementations
  copying py_src\tokenizers\implementations\sentencepiece_bpe.py -> build\lib.win-amd64-cpython-38\tokenizers\implementations
  copying py_src\tokenizers\implementations\sentencepiece_unigram.py -> build\lib.win-amd64-cpython-38\tokenizers\implementations
  copying py_src\tokenizers\implementations\__init__.py -> build\lib.win-amd64-cpython-38\tokenizers\implementations
  creating build\lib.win-amd64-cpython-38\tokenizers\tools
  copying py_src\tokenizers\tools\visualizer.py -> build\lib.win-amd64-cpython-38\tokenizers\tools
  copying py_src\tokenizers\tools\__init__.py -> build\lib.win-amd64-cpython-38\tokenizers\tools
  copying py_src\tokenizers\__init__.pyi -> build\lib.win-amd64-cpython-38\tokenizers
  copying py_src\tokenizers\models\__init__.pyi -> build\lib.win-amd64-cpython-38\tokenizers\models
  copying py_src\tokenizers\decoders\__init__.pyi -> build\lib.win-amd64-cpython-38\tokenizers\decoders
  copying py_src\tokenizers\normalizers\__init__.pyi -> build\lib.win-amd64-cpython-38\tokenizers\normalizers
  copying py_src\tokenizers\pre_tokenizers\__init__.pyi -> build\lib.win-amd64-cpython-38\tokenizers\pre_tokenizers
  copying py_src\tokenizers\processors\__init__.pyi -> build\lib.win-amd64-cpython-38\tokenizers\processors
  copying py_src\tokenizers\trainers\__init__.pyi -> build\lib.win-amd64-cpython-38\tokenizers\trainers
  copying py_src\tokenizers\tools\visualizer-styles.css -> build\lib.win-amd64-cpython-38\tokenizers\tools
  running build_ext
  running build_rust
  error: can't find Rust compiler
  
  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
  
  To update pip, run:
  
      pip install --upgrade pip
  
  and then retry package installation.
  
  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
  Building wheel for safetensors (PEP 517): started
  error: can't find Rust compiler

  ERROR: Failed building wheel for tokenizers

  Building wheel for safetensors (PEP 517): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: 'C:\Users\Bapon\AppData\Local\Programs\Python\Python38\python.exe' 'B:\AndroidStudioProjects\ObjectDetector\app\build\generated\python\bp\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\Bapon\AppData\Local\Temp\tmps5ed2dfv'
       cwd: C:\Users\Bapon\AppData\Local\Temp\pip-install-gggr06au\safetensors
  ERROR: Command errored out with exit status 1:

  Complete output (25 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-cpython-38
  creating build\lib.win-amd64-cpython-38\safetensors
  copying py_src\safetensors\flax.py -> build\lib.win-amd64-cpython-38\safetensors
  copying py_src\safetensors\numpy.py -> build\lib.win-amd64-cpython-38\safetensors
  copying py_src\safetensors\paddle.py -> build\lib.win-amd64-cpython-38\safetensors
  copying py_src\safetensors\tensorflow.py -> build\lib.win-amd64-cpython-38\safetensors
  copying py_src\safetensors\torch.py -> build\lib.win-amd64-cpython-38\safetensors
  copying py_src\safetensors\__init__.py -> build\lib.win-amd64-cpython-38\safetensors
  running build_ext
  running build_rust
  error: can't find Rust compiler
  
  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
  
  To update pip, run:
  error: can't find Rust compiler

  
      pip install --upgrade pip
  
  and then retry package installation.
  
  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
  ----------------------------------------
  ERROR: Failed building wheel for safetensors
Failed to build tokenizers safetensors
ERROR: Could not build wheels for tokenizers, safetensors which use PEP 517 and cannot be installed directly
  ERROR: Failed building wheel for safetensors

Could not build wheels for tokenizers, safetensors which use PEP 517 and cannot be installed directly

Chaquopy: Exit status 1

> Task :app:generateDebugPythonRequirements FAILED

It will be very helpful for me for any help.
Thankyou

[vmitro]

Huggingface transformers is a Python package that uses native components, sort of libraries, written in Rust programming language, that speed up some data (pre)processing (mainly tokenization) utilizing the fact that Rust is natively compiled a pretty fast.

The transformers' setup script searches for a (the?) Rust compiler which you don't have on your system.

In order to use this package on Chaquopy, you are going to have to build a so called Python Wheel (a .whl file), which is a sort of zip archive that gets extracted on the Python's (or virtual environment's) Path and let's you use the package by simply importing it. You can get the all the info how to build your own packages in the server/pypi directory of this repository.

Keep in mind that you'll still have to actually setup a Rust build environment. I was able to compile and build a wheel for tiktoken (sadly haven't got time to test it at all) using rustup (rustup.rs) and installing additional compile targets for Arm/Arm64 architecture.

You can check the build files for tiktoken on my repo (https://github.com/vmitro/chaquopy-experimental/tree/master/server/pypi/packages/tiktoken). Please not it's a clumsy solution, ideally one would edit the setup.py script and do the necessary changes and preparations in Python but meh.

[baponkar]

Thankyou brother! for pointing me the right direction.
I have been solved this issue by installing rust in my windows machine.

[mhsmith]

I have been solved this issue by installing rust in my windows machine.

I doubt that simply installing Rust will be enough on its own. Even if the build completes, it'll probably be a build for Windows which won't run correctly on Android.

I suggest you first try using one of the versions of transformers for which we already have a working tokenizers version (#608 (comment)). If those versions are too old for you, please post a comment on #607 explaining why.

[baponkar]

Sorry! It did not work, I mistakenly removed install "transformers" then try build and I thought it worksbut now it is not working i.e. gradle building failed.

[baponkar]

I have been solved this issue by installing rust in my windows machine.

I doubt that simply installing Rust will be enough on its own. Even if the build completes, it'll probably be a build for Windows which won't run correctly on Android.

I suggest you first try using one of the versions of transformers for which we already have a working tokenizers version (#608 (comment)). If those versions are too old for you, please post a comment on #607 explaining why.

install "transformers==4.15.0" I think this worked for me as it is showing no error in gradle build.
Thankyou so much @mhsmith

[baponkar]

.