How to use Hydra and DVC?

[aksg87]

Hello all,

I am very impressed with DVC and also CML. I have created a machine learning pipeline using Hydra.cc which sets up experiments from YAML files and outputs all experiment parameters, TensorFlow runs, and Model weights into clean outputs. I then store the larger artifacts (model weights) in GitLFS and the lighter-weight files (YAML) into Git.

I saw a few people use both Hydra and DVC. Are there suggestions on this integration? Lots of features of DVC seem really powerful so I would love to blend the best of both worlds. I'm just starting to learn about DVC via some of the great YouTube docs 🙂

[efiop]

Hi :) There is no native integration, but people do ask about it from time to time. See #4883 There are no plans to implement anything on our side in that direction in the near future, but please feel free to leave a comment in the issue, so we know there is interest.

[daavoo]

Hei @aksg87 , as @efiop answered, there is no native integration. Please take a look at this question and answer on the forums, where I tried to give a high level overview of some things to consider:

https://discuss.dvc.org/t/dvc-and-hydra-integration/868/2

It would be great to hear more about what features are you currently using from hydra

[dberenbaum]

Another interesting hydra feature regarding doing sweeps over different parameter configurations: https://hydra.cc/docs/tutorials/basic/running_your_app/multi-run/

[d-miketa]

I ran across the following issue combining DVC with Hydra. This is (roughly) my dvc.yaml:

stages:
  data_cleaning:
    cmd: >
      python stages/my_data_cleaning_script.py
      limit_examples=${limit_examples}
    deps:
      - stages/my_data_cleaning_script.py
      - data/raw
    params:
      - limit_examples
    outs:
      - data/cleaned

with limit_examples of type Optional[int]. my_data_cleaning_script.py, which is a Hydra script, expects for its limit_examples argument either an integer or null.

My params.yaml file looks like this:

limit_examples: null

But DVC seems to internally convert null into the Pythonic None, and then pass that onto the Python script. Hydra (IMO arguably correctly) interprets None as a string:

Running stage 'my_data_cleaning_script.py':                                                                                                                                                                                                                                                         
> python stages/my_data_cleaning_script.py limit_examples=None
Error merging override limit_examples=None
Value 'None' of type 'str' could not be converted to Integer
    full_key: limit_examples
    object_type=Config

I think converting null to None is unexpected behaviour. Can it be disabled?

[dberenbaum]

@d-miketa Hydra expects the string "null" here, so I believe you can simply use limit_examples: "null" to specify that.

[d-miketa]

Yeah that’ll work, it’s just unexpected and hacky - I’d expect YAML to be presented as YAML.

[dberenbaum]

I’d expect YAML to be presented as YAML

Good point. None is awkward here and it should be either null or some other equivalent, but that could be represented any number of ways (for example, as an empty value so you would end up with python stages/my_data_cleaning_script.py limit_examples=).

Since Hydra is using its own domain-specific language for its override syntax, you specifically need the string null and not any YAML equivalent, so I think you still want to quote "null" to ensure it is parsed as expected. The good news is that we are looking into better Hydra integration, so hopefully we can provide a better experience here.

[dberenbaum]

Added #8036 to track the bug.

[d-miketa]

Fair enough! Thanks for the support and look forward to the Hydra integration. :)

[d-miketa]

I noticed a couple more issues with Hydra and YAML parsing, one serious and one less so.

To set the stage, we realised that even if you use a Hydra config YAML, you can still use DVC to write into it using dvc exp run -S <param> and it works remarkably well. You still have to explicitly write parameters to watch into the params key of your stage, but you do not need to pass them directly to the training script. For example,

stages:
  train:
    cmd: "python train.py ${data_folder}"
    params:
      - data_folder

becomes

stages:
  train:
    cmd: "python train.py"
    params:
      - hydra_cfgs/main.yaml:
          - data_folder

and n_layers is still both tracked and passed into the training script. This becomes more helpful if you have a large number of parameters.

You can also specify parameters inside the smaller Hydra configs, e.g. with

stages:
  train:
    cmd: "python train.py"
    params:
      - hydra_cfgs/main.yaml:
          - data_folder
          - model
      - hydra_cfgs/model/random_forest.yaml
          - model.depth

you can run dvc exp run -S hydra_cfgs/main.yaml:model=random_forest -S hydra_cfgs/model/random_forest.yaml:model.depth=5 which overwrites the depth parameter value inside random_forest.yaml:

# @package _global_
---

model:
  depth: 3

turns into

model:
  depth: 5

Unfortunately you lose the comment at the top, which is required by Hydra to correctly nest the contents of random_forest.yaml inside main.yaml -- and the first issue to which I want to draw attention in this comment. The solution should be simple, just preserve every commented line before and up to the first line containing only ---. (My yamllint keeps telling me YAML files should have a --- at the top anyway.)

The second issue concerns indentation (and is again prompted by my yamllint). YAML files are processed from

mylist
  - item1
  - item2

to

mylist
- item1
- item2

i.e. without the 2 spaces of indentation under a list. I don't think this hurts decoding though, so it's probably merely an aesthetic issue.

[dberenbaum]

We have been working on a proposal for how to better integrate DVC and Hydra and would be happy to get feedback from @d-miketa and others. Instead of using DVC to inject values into the Hydra config, we think it will be more useful to do the opposite and use the Hydra output config to inject values into the DVC params. This should provide nearly full flexibility to compose complex configurations with Hydra while making it easy to track the final config in DVC.

Basic workflow

Hydra can replace or add parameters from the command line (see https://hydra.cc/docs/intro/#basic-example). DVC can use Hydra to enhance the existing dvc exp run --set-param workflow by (see below for an example):

• Replacing current parameter parsing logic, providing a well-supported backend that isn’t specific to DVC.
• Adding functionality like +train.max_depth=5, which enables adding new params from the command line (in a way that is familiar to Hydra users).

Start with the following dvc.yaml:

stages:
  train:
    cmd: python src/train.py data/features model.pkl
    deps:
    - data/features
    - src/train.py
    params:
    - train
    outs:
    - model.pkl

And the following params.yaml:

train:
  seed: 20170428
  n_est: 50
  min_split: 0.01

Run a new experiment with:

$ dvc exp run --set-param train.min_split=0.05 --set-param +train.max_depth=5

This will run an experiment with the following modified params.yaml:

train:
  seed: 20170428
  n_est: 50
  min_split: 0.05 # Modified this parameter (same as current behavior)
  max_depth: 5 # Added this parameter (new behavior)

Advanced workflow

Hydra can also be used to compose configuration from multiple files (see https://hydra.cc/docs/intro/#composition-example for an example). DVC can use Hydra to update params.yaml at runtime, providing more flexible project configuration.

Building off the previous example, assume you have multiple model types you want to train. You can specify parameters for each under conf:

$ tree
.
└── conf
    └── train
        ├── gradient_boosting.yaml
        └── random_forest.yaml

conf/train/gradient_boosting.yaml looks like:

algo: GradientBoostingClassifier # Specify model algorithm
seed: 20170428
n_est: 50
learning_rate: 0.1 # New parameter for gradient boosting

Now you can run a new gradient boosting experiment using that configuration:

$ dvc exp run --set-param +train=gradient_boosting

This will update the train section of params.yaml with the values from conf/train/gradient_boosting.yaml and run an experiment with the following params.yaml:

train:
  algo: GradientBoostingClassifier
  seed: 20170428
  n_est: 50
  learning_rate: 0.1

Using Hydra enables even more advanced templating and composition, including:

• [Interpolation](https://hydra.cc/docs/patterns/specializing_config/)
• [Extending configs](https://hydra.cc/docs/patterns/extending_configs/)

Hyperparameter sweeps

Hydra also allows for running hyperparameter sweeps. Using Hydra, the same could be supported in DVC (see https://hydra.cc/docs/intro/#multirun):

$ dvc exp run --set-param train=random_forest,gradient_boosting

Benefits for Hydra users

Note: There are no new features in this section, just more info for those who are interested.

Existing Hydra users get:

• Automatic caching and tracking of the outputs of each experiment.

  For each experiment run, Hydra will create a new output directory like:

  outputs/2019-09-25/15-16-17
  ├── .hydra
  │   ├── config.yaml
  │   ├── hydra.yaml
  │   └── overrides.yaml
  └── my_app.log

  Over time, the outputs directory becomes a mess. DVC can keep only the current outputs in the workspace and compare everything with dvc exp show.

  When using Hydra, launching an experiment with the same config as one already ran will result in recomputing an identical run. By tracking the experiment outputs with DVC, experiments already reproduced will be retrieved from the run-cache instead of being recomputed.

• Language agnostic

  You can benefit from Hydra functionality without relying on the Hydra Python API for assembling the config (no more @hydra.main() calls).

  Because DVC assembles the config into a single static params.yaml before running the stage, you only need a library that can load a single yaml file.

  You can also combine Hydra-generated config with DVC stages running arbitrary CLI commands:

  stages:
    Rscript:
      # Dict unpacking + R argparse
      cmd: bash train.R ${hydra_config}

[d-miketa]

That looks fantastic - would love to help test drive this at some point. :)

[dberenbaum]

That would be great! @daavoo can share with you once he has something to test drive.