[Bug] Compiling android model of Llama-2-7b-chat-hf on Windows

[ashmon]

🐛 Bug

Error encountered in latest when compiling android model of Llama-2-7b-chat-hf on Windows.

python -m mlc_llm.build --target android --max-seq-len 768 --model dist/models/Llama-2-7b-chat-hf --quantization q4f16_1

Target configured: opencl -keys=opencl,gpu -max_function_args=128 -max_num_threads=256 -max_shared_memory_per_block=16384 -max_threads_per_block=256 -texture_spatial_limit=16384 -thread_warp_size=1
[23:26:03] D:\a\package\package\tvm\src\node\reflection.cc:109: AttributeError: relax.expr.Var object has no attributed shard_dim
Stack trace not available when DMLC_LOG_STACK_TRACE is disabled at compile time.

[23:26:03] D:\a\package\package\tvm\src\node\reflection.cc:109: AttributeError: relax.expr.Var object has no attributed shard_strategy
Stack trace not available when DMLC_LOG_STACK_TRACE is disabled at compile time.

Those two attrib errors repeat roughly 100 times then finally the index out of bounds hits. 

[23:26:05] D:\a\package\package\tvm\src\relax\ir\expr.cc:174: Check failed: index < tuple_info->fields.size() (197 vs. 197) : Index out of bounds: Tuple params is of size 197, and cannot be accessed with index 197
Stack trace not available when DMLC_LOG_STACK_TRACE is disabled at compile time.

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "c:\Users\UserName\project\mlc-llm\mlc_llm\build.py", line 46, in <module>
    main()
  File "c:\Users\UserName\project\mlc-llm\mlc_llm\build.py", line 42, in main
    core.build_model_from_args(parsed_args)
  File "c:\Users\UserName\project\mlc-llm\mlc_llm\core.py", line 645, in build_model_from_args
    new_params = utils.convert_weights(param_manager, params, args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\UserName\project\mlc-llm\mlc_llm\utils.py", line 229, in convert_weights
    mod_transform = relax.transform.LazyTransformParams()(mod_transform)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\UserName\anaconda3\envs\project\Lib\site-packages\tvm\ir\transform.py", line 238, in __call__
    return _ffi_transform_api.RunPass(self, mod)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\UserName\anaconda3\envs\project\Lib\site-packages\tvm\_ffi\_ctypes\packed_func.py", line 239, in __call__
    raise_last_ffi_error()
  File "C:\Users\UserName\anaconda3\envs\project\Lib\site-packages\tvm\_ffi\base.py", line 415, in raise_last_ffi_error
    _LIB.TVMGetLastPythonError.restype = ctypes.c_void_p
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\UserName\anaconda3\envs\project\Lib\ctypes\__init__.py", line 389, in __getattr__
    func = self.__getitem__(name)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\UserName\anaconda3\envs\project\Lib\ctypes\__init__.py", line 394, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: function 'TVMGetLastPythonError' not found. Did you mean: 'TVMAPISetLastPythonError'?

Steps to reproduce the behavior:

The is following the outlined Android App instructions via https://llm.mlc.ai/docs/deploy/android.html

Environment

Windows 10 10.0.19045, building for Android
Rust installed and exposed to PATH.
Android studio installed and configured.
OpenJDK installed
All env vars configured and independently verified.

The test basis now tested on two different computers with similar but different spec.
fresh conda env py3.11
mlc-llm installed via pip with --recursive.
tvm installed via pip
python -m pip install --pre -U -f https://mlc.ai/wheels mlc-ai-nightly
zstd.dll downloaded and coped to conda env/site-packages/tvm/
Llama-2-7b-chat-hf pulled via git. into correct dist/models/Llama-2-7b-chat-hf subfolder.

To verify TVM here is the following support info.
python -c "import tvm; print(tvm.file)"
C:\Users\UserName\anaconda3\envs\project\Lib\site-packages\tvm_init_.py

python -c "import tvm; print(tvm._ffi.base._LIB)"
<CDLL 'C:\Users\UserName\anaconda3\envs\project\Lib\site-packages\tvm\tvm.dll', handle 7ffd3a1b0000 at 0x2275b9db7d0>

(project) c:\Users\UserName\project\mlc-llm>python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))"

USE_NVTX: OFF
USE_GTEST: AUTO
SUMMARIZE: OFF
USE_IOS_RPC: OFF
USE_MSC: OFF
USE_ETHOSU:
CUDA_VERSION: NOT-FOUND
USE_LIBBACKTRACE: AUTO
DLPACK_PATH: 3rdparty/dlpack/include
USE_TENSORRT_CODEGEN: OFF
USE_THRUST: OFF
USE_TARGET_ONNX: OFF
USE_AOT_EXECUTOR: ON
BUILD_DUMMY_LIBTVM: OFF
USE_CUDNN: OFF
USE_TENSORRT_RUNTIME: OFF
USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR: OFF
USE_CCACHE: AUTO
USE_ARM_COMPUTE_LIB: OFF
USE_CPP_RTVM:
USE_OPENCL_GTEST: /path/to/opencl/gtest
USE_MKL: OFF
USE_PT_TVMDSOOP: OFF
MLIR_VERSION: NOT-FOUND
USE_CLML: OFF
USE_STACKVM_RUNTIME: OFF
USE_GRAPH_EXECUTOR_CUDA_GRAPH: OFF
ROCM_PATH: /opt/rocm
USE_DNNL: OFF
USE_VITIS_AI: OFF
USE_MLIR: OFF
USE_RCCL: OFF
USE_LLVM: llvm-config --link-static
USE_VERILATOR: OFF
USE_TF_TVMDSOOP: OFF
USE_THREADS: ON
USE_MSVC_MT: OFF
BACKTRACE_ON_SEGFAULT: OFF
USE_GRAPH_EXECUTOR: ON
USE_NCCL: OFF
USE_ROCBLAS: OFF
GIT_COMMIT_HASH: 62c05266986ea6639a9fd16fb87ba75a9ec056a8
USE_VULKAN: ON
USE_RUST_EXT: OFF
USE_CUTLASS: OFF
USE_CPP_RPC: OFF
USE_HEXAGON: OFF
USE_CUSTOM_LOGGING: OFF
USE_UMA: OFF
USE_FALLBACK_STL_MAP: OFF
USE_SORT: ON
USE_RTTI: ON
GIT_COMMIT_TIME: 2023-10-07 16:42:11 -0700
USE_HEXAGON_SDK: /path/to/sdk
USE_BLAS: none
USE_ETHOSN: OFF
USE_LIBTORCH: OFF
USE_RANDOM: ON
USE_CUDA: OFF
USE_COREML: OFF
USE_AMX: OFF
BUILD_STATIC_RUNTIME: OFF
USE_CMSISNN: OFF
USE_KHRONOS_SPIRV: OFF
USE_CLML_GRAPH_EXECUTOR: OFF
USE_TFLITE: OFF
USE_HEXAGON_GTEST: /path/to/hexagon/gtest
PICOJSON_PATH: 3rdparty/picojson
USE_OPENCL_ENABLE_HOST_PTR: OFF
INSTALL_DEV: OFF
USE_PROFILER: ON
USE_NNPACK: OFF
LLVM_VERSION: 17.0.2
USE_OPENCL: OFF
COMPILER_RT_PATH: 3rdparty/compiler-rt
RANG_PATH: 3rdparty/rang/include
USE_SPIRV_KHR_INTEGER_DOT_PRODUCT: OFF
USE_OPENMP: OFF
USE_BNNS: OFF
USE_CUBLAS: OFF
USE_METAL: OFF
USE_MICRO_STANDALONE_RUNTIME: OFF
USE_HEXAGON_EXTERNAL_LIBS: OFF
USE_ALTERNATIVE_LINKER: AUTO
USE_BYODT_POSIT: OFF
USE_HEXAGON_RPC: OFF
USE_MICRO: OFF
DMLC_PATH: 3rdparty/dmlc-core/include
INDEX_DEFAULT_I64: ON
USE_RELAY_DEBUG: OFF
USE_RPC: ON
USE_TENSORFLOW_PATH: none
TVM_CLML_VERSION:
USE_MIOPEN: OFF
USE_ROCM: OFF
USE_PAPI: OFF
USE_CURAND: OFF
TVM_CXX_COMPILER_PATH: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.35.32215/bin/HostX64/x64/cl.exe
HIDE_PRIVATE_SYMBOLS: OFF

Let me know if there is any other info I can gather for you.
-Cort

[junrushao]

I noticed three issues with your report:

One is, which seems related to sharding:

[23:26:03] D:\a\package\package\tvm\src\node\reflection.cc:109: AttributeError: relax.expr.Var object has no attributed shard_dim
[23:26:03] D:\a\package\package\tvm\src\node\reflection.cc:109: AttributeError: relax.expr.Var object has no attributed shard_strategy

And the second one is:

[23:26:05] D:\a\package\package\tvm\src\relax\ir\expr.cc:174: Check failed: index < tuple_info->fields.size() (197 vs. 197) : Index out of bounds: Tuple params is of size 197, and cannot be accessed with index 197

The last one seems to related to some latest change from upstream TVM (CC the author @Lunderberg if you'd love to take a look):

AttributeError: function 'TVMGetLastPythonError' not found. Did you mean: 'TVMAPISetLastPythonError'?

Would you like to provide the full python stack trace of the first two issues? Particularly the first one will be helpful to me :)

[junrushao]

Related: #1112

[junrushao]

Update: #1112 (comment)

[junrushao]

Fixed