MLPerf ResNet inference workflow failing on Windows

[gfursin]

Hi @anandhu-eng and @arjunsuresh,

I just tried the CM-MLPerf workflow for ResNet benchmark with the latest CM4MLOps mlperf-inference branch on my Windows machine and it fails when downloading a model.

It's weird because we have a GitHub test for this benchmark on Windows...

Do you mind to check if it works on your side, please? If not, I am curious why our tests didn't capture that.

Here is the command I used on Windows 11 (copy/paste from our GitHub test):

cm pull repo mlcommons@cm4mlops --branch=mlperf-inference

cm rm cache -f

cm run script --tags=run-mlperf,inference,_submission,_short --submitter="cTuning" --hw_name=default --model=resnet50 --implementation=python --backend=onnxruntime --device=cpu --scenario=Offline --test_query_count=500 --target_qps=1 -v --quiet 

and here is the error:

Downloading from https://zenodo.org/record/4735647/files/resnet50_v1.onnx

File resnet50_v1.onnx already present, original checksum and computed checksum matches! Skipping Download..

Traceback (most recent call last):
  File "C:\!Progs\Python310\Scripts\cm-script.py", line 33, in <module>
    sys.exit(load_entry_point('cmind==2.3.8.1', 'console_scripts', 'cm')())
  File "C:\!Progs\Python310\lib\site-packages\cmind\cli.py", line 37, in run
    r = cm.access(argv, out='con')
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1477, in _run
    r = customize_code.preprocess(ii)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\script\run-mlperf-inference-app\customize.py", line 231, in preprocess
    r = cm.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 761, in access
    return cm.access(i)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1537, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta,  env, state, const, const_state, add_deps_recursive,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 2882, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 3033, in _run_deps
    r = self.cmind.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1367, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 2882, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 3033, in _run_deps
    r = self.cmind.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1537, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta,  env, state, const, const_state, add_deps_recursive,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 2882, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 3033, in _run_deps
    r = self.cmind.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1537, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta,  env, state, const, const_state, add_deps_recursive,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 2882, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 3033, in _run_deps
    r = self.cmind.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1477, in _run
    r = customize_code.preprocess(ii)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\script\download-file\customize.py", line 210, in preprocess
    env['CM_DOWNLOAD_CMD'] =  env['CM_DOWNLOAD_CMD'].replace('&', '^&').replace('|', '^|').replace('(', '^(').replace(')', '^)')
KeyError: 'CM_DOWNLOAD_CMD'
                                                                                                                

Thanks!

[arjunsuresh]

Hi @gfursin this PR should fix it. I think it went fine in github action because the cache was clean.