ValidationError: A lot of validation errors for RunConfig

[dangokuson]

Describe the bug
There are a lot of validation errors for RunConfig by using JSON config in the examples folder

    return cls.parse_file(file_path)
  File "pydantic/main.py", line 569, in pydantic.main.BaseModel.parse_file
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 7 validation errors for RunConfig
data_configs -> name
  DataConfig name speech_commands_v0.02 should only contain letters, numbers and underscore. (type=value_error)
evaluators -> common_evaluator
  speech_commands_v0.02 not found in {} (type=value_error)
engine
  Invalid evaluators (type=value_error)
passes -> conversion
  Invalid engine (type=value_error)
passes -> transformers_optimization
  Invalid engine (type=value_error)
passes -> quantization
  Invalid engine (type=value_error)
passes -> perf_tuning
  Invalid engine (type=value_error)

To Reproduce
I tried to optimize AST model in the examples folder by below command line, but it does not works.

olive run --config ast.json --setup

Expected behavior
A clear and concise description of what you expected to happen.

Olive config
I am using Python 3.9.19

Package             Version
------------------- -----------
alembic             1.13.2
annotated-types     0.7.0
certifi             2024.8.30
charset-normalizer  3.3.2
coloredlogs         15.0.1
colorlog            6.8.2
filelock            3.16.1
flatbuffers         24.3.25
fsspec              2024.9.0
greenlet            3.1.0
huggingface-hub     0.25.0
humanfriendly       10.0
idna                3.10
Jinja2              3.1.4
lightning-utilities 0.11.7
Mako                1.3.5
MarkupSafe          2.1.5
mpmath              1.3.0
networkx            3.2.1
numpy               1.26.4
olive-ai            0.7.0
onnx                1.16.2
onnxruntime         1.19.2
optuna              4.0.0
packaging           24.1
pandas              2.2.2
pip                 24.2
protobuf            3.20.3
pydantic            2.9.2
pydantic_core       2.23.4
python-dateutil     2.9.0.post0
pytz                2024.2
PyYAML              6.0.2
regex               2024.9.11
requests            2.32.3
safetensors         0.4.5
setuptools          75.1.0
six                 1.16.0
SQLAlchemy          2.0.35
sympy               1.13.3
tokenizers          0.19.1
torch               2.2.2
torchmetrics        1.4.2
tqdm                4.66.5
transformers        4.44.2
typing_extensions   4.12.2
tzdata              2024.1
urllib3             2.2.3
wheel               0.44.0

Olive logs

Traceback (most recent call last):
  File "/Users/ubuntu/workspace/projects/AI_Research/workbench/tests/olive_tests/AST/main.py", line 2, in <module>
    olive_run("ast.json")
  File "/Users/ubuntu/anaconda3/envs/pydev39-olive/lib/python3.9/site-packages/olive/workflows/run/run.py", line 294, in run
    run_config: RunConfig = RunConfig.parse_file_or_obj(run_config)
  File "/Users/ubuntu/anaconda3/envs/pydev39-olive/lib/python3.9/site-packages/olive/common/config_utils.py", line 126, in parse_file_or_obj
    return cls.parse_file(file_path)
  File "pydantic/main.py", line 569, in pydantic.main.BaseModel.parse_file
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 7 validation errors for RunConfig
data_configs -> name
  DataConfig name speech_commands_v0.02 should only contain letters, numbers and underscore. (type=value_error)
evaluators -> common_evaluator
  speech_commands_v0.02 not found in {} (type=value_error)
engine
  Invalid evaluators (type=value_error)
passes -> conversion
  Invalid engine (type=value_error)
passes -> transformers_optimization
  Invalid engine (type=value_error)
passes -> quantization
  Invalid engine (type=value_error)
passes -> perf_tuning
  Invalid engine (type=value_error)

Other information

• OS: [e.g. Windows, Linux]
• Olive version: [e.g. 0.4.0 or main]
• ONNXRuntime package and version: [e.g. onnxruntime-gpu: 1.16.1]
• Transformers package version: [e.g. transformers 4.44.1]

Additional context
Add any other context about the problem here.

[jambayk]

The examples are in sync with the main branch so installing olive from pypi don't always work with the latest examples. Please refer to https://github.com/microsoft/Olive/tree/main/examples#important for more details.

For the ast example, could you try removing the 0.02 in

Olive/examples/AST/ast.json

Line 35 in d3070aa

"data_config": "speech_commands_v0.02",

? We might have missed updating that part.