Dictionary as paramters in config files

[minhptx]

🐛 Describe the bug

Dictionaries are parsed as Param and thus sometimes prevent dictionary-type parameters cannot be used in config files. Example for datasets::load step as follows:

read_data: {
        type: "datasets::load",
        path: "json",
        data_files: {
            train: "train.jsonl",
            dev: "dev.jsonl"
        },
    },

data_files parameter cannot be used in this case since dictionaries in config files are Param by default. Can be fixed by using as_dict function to convert Param:

    def run(self, path: str, **kwargs) -> Union[ds.DatasetDict, ds.Dataset]:  # type: ignore
        """
        Load the HuggingFace dataset specified by ``path``.

        ``path`` is the canonical name or path to the dataset. Additional key word arguments
        are passed as-is to :func:`datasets.load_dataset()`.
        """
        kwargs = {x: y.as_dict() for x, y in kwargs.items()}
        dataset = ds.load_dataset(path, **kwargs)

Versions

Python 3.8.13
absl-py==1.2.0
accelerate==0.12.0
ai2-tango==0.14.0
aiohttp==3.8.3
aiohttp-cors==0.7.0
aiosignal==1.2.0
antlr4-python3-runtime==4.9.3
anyio==3.6.1
argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work
argon2-cffi-bindings @ file:///tmp/build/80754af9/argon2-cffi-bindings_1644569684262/work
asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work
astunparse==1.6.3
async-timeout==4.0.2
attrs==22.1.0
autoflake==1.6.1
Babel @ file:///tmp/build/80754af9/babel_1620871417480/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
base58==2.1.1
beaker-py==1.10.3
beautifulsoup4==4.11.1
black==22.10.0
bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work
blessed==1.19.1
blingfire==0.1.8
blis==0.7.8
boto3==1.24.59
botocore==1.27.59
bpemb==0.3.4
brotlipy==0.7.0
bs4==0.0.1
cached-path==1.1.6
cachetools==5.2.0
catalogue==2.0.8
certifi==2022.9.24
cffi @ file:///tmp/abs_98z5h56wf8/croots/recipe/cffi_1659598650955/work
charset-normalizer==2.1.1
chex==0.1.5
click==8.0.4
click-help-colors==0.9.1
cloudpickle==2.2.0
colorama==0.4.5
coloredlogs==15.0.1
colorful==0.5.4
commonmark==0.9.1
confection==0.0.3
conllu==4.5.2
contourpy==1.0.5
cryptography @ file:///tmp/build/80754af9/cryptography_1652083738073/work
cupy-cuda113==10.6.0
cycler==0.11.0
cymem==2.0.6
Cython==0.29.32
datasets==2.5.2
debugpy @ file:///tmp/build/80754af9/debugpy_1637091796427/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
Deprecated==1.2.13
dill==0.3.5.1
distlib==0.3.6
dm-tree==0.1.7
docker==6.0.0
docker-pycreds==0.4.0
elasticsearch==7.17.6
elasticsearch-dsl==7.4.0
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.0/en_core_web_sm-3.4.0-py3-none-any.whl
entrypoints @ file:///tmp/build/80754af9/entrypoints_1649926445639/work
etils==0.8.0
evaluate==0.2.2
executing @ file:///opt/conda/conda-bld/executing_1646925071911/work
fairscale==0.4.9
faiss==1.7.2
fastjsonschema @ file:///opt/conda/conda-bld/python-fastjsonschema_1661371079312/work
fastrlock==0.8
filelock==3.8.0
flair==0.11.3
flatbuffers==22.9.24
flax==0.5.0
fonttools==4.37.4
frozenlist==1.3.1
fsspec==2022.8.2
ftfy==6.1.1
future==0.18.2
gast==0.4.0
gdown==4.4.0
gensim==4.2.0
gevent==22.8.0
gitdb==4.0.9
GitPython==3.1.28
glob2==0.7
google-api-core==2.8.2
google-auth==2.12.0
google-auth-oauthlib==0.4.6
google-cloud-core==2.3.2
google-cloud-storage==2.5.0
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.4.0
googleapis-common-protos==1.56.4
gpustat==1.0.0
greenlet==1.1.3.post0
grpcio==1.43.0
h5py==3.7.0
huggingface-hub==0.9.1
humanfriendly==10.0
hydra-core==1.2.0
hyperopt==0.2.7
idna==3.4
importlib-metadata==4.11.3
importlib-resources==5.9.0
inflect==6.0.0
iniconfig==1.1.1
ipykernel @ file:///tmp/build/80754af9/ipykernel_1646982648551/work/dist/ipykernel-6.9.1-py3-none-any.whl
ipython @ file:///opt/conda/conda-bld/ipython_1657652213665/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
Janome==0.4.2
jaro-winkler==2.0.3
jax==0.3.20
jaxlib==0.3.20
jedi @ file:///tmp/build/80754af9/jedi_1644315233700/work
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
json5 @ file:///tmp/build/80754af9/json5_1624432770122/work
jsonlines==3.1.0
jsonnet==0.18.0
jsonschema==4.16.0
jupyter-core @ file:///opt/conda/conda-bld/jupyter_core_1651671229925/work
jupyter-server @ file:///tmp/abs_b88b31b8-83b9-476d-a46d-e563c421f38fvsnyi1ur/croots/recipe/jupyter_server_1658754481507/work
jupyter_client @ file:///opt/conda/conda-bld/jupyter_client_1661848916004/work
jupyterlab @ file:///tmp/abs_12f3h01vmy/croots/recipe/jupyterlab_1658907535764/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab_server @ file:///opt/conda/conda-bld/jupyterlab_server_1664908231281/work
keras==2.10.0
Keras-Preprocessing==1.1.2
kilt @ git+https://github.com/facebookresearch/KILT.git@2664322b9a994be686e4c3a9e8f75a0b70927f22
kiwisolver==1.4.4
konoha==4.6.5
langcodes==3.3.0
langdetect==1.0.9
libclang==14.0.6
lightgbm==3.3.2
loguru==0.6.0
lxml==4.9.1
Markdown==3.3.4
MarkupSafe==2.1.1
matplotlib==3.6.0
matplotlib-inline @ file:///opt/conda/conda-bld/matplotlib-inline_1662014470464/work
mistune==0.8.4
mkl-fft==1.3.1
mkl-random @ file:///tmp/build/80754af9/mkl_random_1626186064646/work
mkl-service==2.4.0
more-itertools==8.14.0
mpld3==0.3
mpmath==1.2.1
msgpack==1.0.4
multidict==6.0.2
multiprocess==0.70.13
murmurhash==1.0.8
mypy-extensions==0.4.3
nbclassic @ file:///opt/conda/conda-bld/nbclassic_1644943264176/work
nbclient @ file:///tmp/build/80754af9/nbclient_1650308366712/work
nbconvert @ file:///opt/conda/conda-bld/nbconvert_1649751911790/work
nbformat @ file:///opt/conda/conda-bld/nbformat_1663744952973/work
nest-asyncio @ file:///tmp/build/80754af9/nest-asyncio_1649847908682/work
networkx==2.8.7
nltk==3.7
nmslib==2.1.1
notebook @ file:///tmp/abs_abf6xa6h6f/croots/recipe/notebook_1659083654985/work
numpy==1.23.2
nvidia-ml-py==11.495.46
oauthlib==3.2.1
omegaconf==2.2.3
onnxruntime==1.12.1
opencensus==0.11.0
opencensus-context==0.1.3
opt-einsum==3.3.0
optax==0.1.3
overrides==3.1.0
packaging==21.3
pandas==1.5.0
pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work
parso @ file:///opt/conda/conda-bld/parso_1641458642106/work
pastel==0.2.1
pathspec==0.10.1
pathtools==0.1.2
pathy==0.6.2
petname==2.6
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow==9.2.0
pkgutil_resolve_name==1.3.10
platformdirs==2.5.2
pluggy==1.0.0
poethepoet==0.16.2
polars==0.13.62
portalocker==2.5.1
pptree==3.1
preshed==3.0.7
prettytable==3.4.1
prometheus-client @ file:///tmp/abs_d3zeliano1/croots/recipe/prometheus_client_1659455100375/work
promise==2.3
prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1633440160888/work
protobuf==3.19.4
psutil==5.9.2
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work
py==1.11.0
py-spy==0.3.14
py4j==0.10.9.7
pyarrow==9.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.6.1
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pydantic==1.9.2
pyDeprecate==0.3.2
pyflakes==2.5.0
Pygments==2.13.0
pyjnius==1.4.2
pymongo==4.2.0
pyOpenSSL @ file:///opt/conda/conda-bld/pyopenssl_1643788558760/work
pyparsing==3.0.9
pyrootutils==1.0.4
pyrsistent==0.18.1
pyserini==0.17.1
PySocks==1.7.1
pytest==7.1.3
python-dateutil==2.8.2
python-dotenv==0.21.0
pytorch-lightning==1.7.7
pytz==2022.4
PyYAML==6.0
pyzmq @ file:///opt/conda/conda-bld/pyzmq_1657724186960/work
ray==1.13.0
-e git+ssh://[email protected]/minhptx/read.git@983be8afad8e26ecb1c38a9c05eea88c77001802#egg=read
regex==2022.9.13
requests==2.28.1
requests-oauthlib==1.3.1
responses==0.18.0
rich==12.6.0
rouge==1.0.1
rsa==4.9
s3transfer==0.6.0
sacrebleu==2.2.1
sacremoses==0.0.53
sarge==0.1.7.post1
scikit-learn==1.1.2
scipy==1.6.1
seaborn==0.12.0
segtok==1.5.11
Send2Trash @ file:///tmp/build/80754af9/send2trash_1632406701022/work
sentence-transformers==2.2.2
sentencepiece==0.1.95
sentry-sdk==1.9.10
setproctitle==1.3.2
setuptools-scm==7.0.5
shortuuid==1.0.9
six==1.16.0
smart-open==5.2.1
smmap==5.0.0
sniffio==1.3.0
soupsieve==2.3.2.post1
spacy==3.4.1
spacy-legacy==3.0.10
spacy-loggers==1.0.3
sqlitedict==2.0.0
srsly==2.4.4
stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work
sympy==1.11.1
tabulate==0.9.0
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-cpu==2.10.0
tensorflow-estimator==2.10.0
tensorflow-io-gcs-filesystem==0.27.0
termcolor==2.0.1
terminado @ file:///tmp/build/80754af9/terminado_1644322581811/work
testpath @ file:///opt/conda/conda-bld/testpath_1655908557405/work
thinc==8.1.3
threadpoolctl==3.1.0
tokenizers==0.12.1
tomli==2.0.1
toolz==0.12.0
torch @ https://download.pytorch.org/whl/cu113/torch-1.12.1%2Bcu113-cp38-cp38-linux_x86_64.whl
torch-scatter==2.0.9
torchmetrics==0.10.0
torchvision @ https://download.pytorch.org/whl/cu113/torchvision-0.13.1%2Bcu113-cp38-cp38-linux_x86_64.whl
tornado @ file:///opt/conda/conda-bld/tornado_1662061693373/work
tqdm==4.64.1
traitlets==5.4.0
transformers==4.22.2
trectools==0.0.49
typer==0.4.2
typing_extensions==4.4.0
ujson==5.5.0
Unidecode==1.3.6
urllib3==1.26.12
virtualenv==20.16.5
wandb==0.13.4
wasabi==0.10.1
wcwidth==0.2.5
webencodings==0.5.1
websocket==0.2.1
websocket-client==0.58.0
Werkzeug==2.2.2
Wikipedia-API==0.5.4
wikitextparser==0.51.0
wrapt==1.14.1
xxhash==3.0.0
yarl==1.8.1
zipp==3.8.1
zope.event==4.5.0
zope.interface==5.5.0