DVCLive 3.0 (#4840)

* Drop save_dvc_exp=True * Remove local file mention * Add mention about ignoring * change dvc.yaml path for dvclive (#4846) * change dvc.yaml path for dvclive * catch other dvclive/dvc.yaml refs * Drop `model_file` * Apply suggestions from code review --------- Co-authored-by: Dave Berenbaum <[email protected]>
iterative · Sep 22, 2023 · 710d054 · 710d054
1 parent 3029fc5
commit 710d054
Show file tree

Hide file tree

Showing 25 changed files with 90 additions and 146 deletions.
diff --git a/content/docs/command-reference/metrics/index.md b/content/docs/command-reference/metrics/index.md
@@ -105,11 +105,11 @@ like:
 }
 ```
 
-It will also generate `dvclive/dvc.yaml`, which includes:
+It will also generate `dvc.yaml`, which includes:
 
 ```yaml
 metrics:
-  - metrics.json
+  - dvclive/metrics.json
 ```
 
 Now let's print metrics values that we are tracking in this

diff --git a/content/docs/dvclive/how-it-works.md b/content/docs/dvclive/how-it-works.md
@@ -16,11 +16,11 @@ The contents of the directory will depend on the methods used:
 | `Live.log_param()`                                      | `dvclive/params.yaml`                                                                |
 | `Live.log_plot()`                                       | `dvclive/plots/custom`                                                               |
 | `Live.log_sklearn_plot()`                               | `dvclive/plots/sklearn`                                                              |
-| `Live.make_dvcyaml()`                                   | `dvclive/dvc.yaml`                                                                   |
+| `Live.make_dvcyaml()`                                   | `dvc.yaml`                                                                           |
 | `Live.make_report()`                                    | `dvclive/report.{md/html}`                                                           |
 | `Live.make_summary()`                                   | `dvclive/metrics.json`                                                               |
-| `Live.next_step()`                                      | `dvclive/dvc.yaml`<br>`dvclive/metrics.json`<br>`dvclive/report.{md/html}`           |
-| `Live.end()`                                            | `dvclive/dvc.yaml`<br>`dvclive/metrics.json`<br>`dvclive/report.{md/html}`           |
+| `Live.next_step()`                                      | `dvc.yaml`<br>`dvclive/metrics.json`<br>`dvclive/report.{md/html}`                   |
+| `Live.end()`                                            | `dvc.yaml`<br>`dvclive/metrics.json`<br>`dvclive/report.{md/html}`                   |
 
 ### Example
 
@@ -35,7 +35,7 @@ from PIL import Image
 
 EPOCHS = 2
 
-with Live(save_dvc_exp=True) as live:
+with Live(report="notebook") as live:
     live.log_param("epochs", EPOCHS)
 
     for i in range(EPOCHS):
@@ -54,8 +54,8 @@ with Live(save_dvc_exp=True) as live:
 The resulting structure will be:
 
 ```
+dvc.yaml
 dvclive
-├── dvc.yaml
 ├── metrics.json
 ├── params.yaml
 ├── plots
@@ -69,7 +69,7 @@ dvclive
 │   │       └── metric.tsv
 │   └── sklearn
 │       └── confusion_matrix.json
-└── report.html
+└── report.md
 model.pt
 model.pt.dvc
 ```
@@ -86,11 +86,11 @@ experiment tracker.
 DVCLive relies on Git to track the [directory] it generates, so it will save
 each run to the same path and overwrite the results each time. DVCLive uses Git
 to manage results, code changes, and data changes
-([with DVC](#track-large-artifacts-with-dvc)). Include
-[`save_dvc_exp=True`](/doc/dvclive/live#parameters) to auto-track as a <abbr>DVC
-experiment</abbr> so you don't need to worry about manually making Git commits
-or branches for each experiment. You can recover them using `dvc exp` commands
-or using Git.
+([with DVC](#track-large-artifacts-with-dvc)).
+
+By default, DVCLive will save a <abbr>DVC experiment</abbr> so you don't need to
+worry about manually making Git commits or branches for each experiment. You can
+recover them using `dvc exp` commands or using Git.
 
 ### Track large artifacts with DVC
 
@@ -109,13 +109,11 @@ with Git, in which case you can use
 
 ## Setup to Run with DVC
 
-DVCLive by default [generates] its own `dvc.yaml` file to configure the
-experiment results, but you can create your own `dvc.yaml` file at the base of
-your repository (or elsewhere) to define a [pipeline](#setup-to-run-with-dvc) to
-run experiments with DVC or
+You can create or modify the `dvc.yaml` file at the base of your repository (or
+elsewhere) to define a [pipeline](#setup-to-run-with-dvc) to run experiments
+with DVC or
 [customize plots](/doc/user-guide/experiment-management/visualizing-plots#defining-plots).
-Do not reuse the DVCLive `dvc.yaml` file since it gets overwritten during each
-experiment run. A pipeline stage for model training might look like:
+A pipeline stage for model training might look like:
 
 ```yaml
 stages:
@@ -152,13 +150,6 @@ downstream in your pipeline. For example, to cache all DVCLive plots:
 +       - dvclive/plots
 ```
 
-<admon type="warn">
-
-Do not add the entire DVCLive [directory] since DVC does not expect the DVCLive
-`dvc.yaml` file to be inside the [outputs].
-
-</admon>
-
 [directory]: /doc/dvclive/how-it-works#directory-structure
 [cache]: /doc/start/data-management/data-versioning
 [outputs]: /doc/user-guide/pipelines/defining-pipelines#outputs

diff --git a/content/docs/dvclive/index.md b/content/docs/dvclive/index.md
@@ -19,18 +19,11 @@ can jump directly to its corresponding page.
 ### Initialize DVCLive
 
 ```python
-with Live(save_dvc_exp=True) as live:
+with Live() as live:
 ```
 
 See [`Live()`](/doc/dvclive/live) for details.
 
-<admon type="info">
-
-Including `save_dvc_exp=True` will automatically
-[track the results](/doc/dvclive/how-it-works#git-integration).
-
-</admon>
-
 ### Log data
 
 <toggle>
@@ -120,6 +113,8 @@ See `Live.next_step()`.
 Under the hood, `Live.next_step()` calls `Live.make_summary()`,
 `Live.make_dvcyaml()`, and `Live.make_report()`.
 
+When access is enabled, updates will be sent to DVC Studio.
+
 If you want to decouple the `step` update from the rest of the calls, you can
 manually modify the `Live.step` property and call `Live.make_summary()` /
 `Live.make_dvcyaml()` / `Live.make_report()`.
@@ -133,7 +128,7 @@ Joining the above snippets, you can include DVCLive in your training code:
 
 from dvclive import Live
 
-with Live(save_dvc_exp=True) as live:
+with Live() as live:
 
     live.log_param("epochs", NUM_EPOCHS)
 

diff --git a/content/docs/dvclive/live/end.md b/content/docs/dvclive/live/end.md
@@ -37,7 +37,7 @@ model.fit(
 By default, `Live.end()` will call `Live.make_summary()`, `Live.make_dvcyaml()`,
 and `Live.make_report()`.
 
-If `save_dvc_exp=True` has been passed to `Live`, it will
+If `save_dvc_exp=True`, it will
 [save a new DVC experiment](/doc/dvclive/how-it-works#git-integration) and write
 a `dvc.yaml` file configuring what DVC will show for logged plots, metrics, and
 parameters.
diff --git a/content/docs/dvclive/live/index.md b/content/docs/dvclive/live/index.md
@@ -9,8 +9,8 @@ class Live:
         self,
         dir: str = "dvclive",
         resume: bool = False,
-        report: Optional[str] = "auto",
-        save_dvc_exp: bool = False,
+        report: Optional[str] = None,
+        save_dvc_exp: bool = True,
         dvcyaml: bool = True,
         exp_message: Optional[str] = None,
     ):
@@ -49,7 +49,7 @@ You can use `Live()` as a context manager. When exiting the context manager,
 - `dir` - Location of the directory to store
   [outputs](/doc/dvclive/how-it-works).
 
-- `dvc_file` - `{Live.dir}/dvc.yaml`.
+- `dvc_file` - Path for `dvc.yaml` file.
 
 - `metrics_file` - `{Live.dir}/metrics.json`.
 
@@ -68,30 +68,27 @@ You can use `Live()` as a context manager. When exiting the context manager,
 - `resume` - If `True`, DVCLive will try to read the previous `step` from the
   `metrics_file` and start from that point. Defaults to `False`.
 
-- `report` - Any of `auto`, `html`, `notebook`, `md` or `None`. See
-  `Live.make_report()`.
+- `report` - Any of `html`, `notebook`, `md` or `None`. See
+  `Live.make_report()`. Defaults to `None`.
 
-  The `auto` mode (default) will use `md` format if the `CI` env var is present
-  and [matplotlib](https://matplotlib.org/) is installed and `html` otherwise.
-
-  If `report` is `None`, `Live.make_report()` won't generate anything.
-
-- `save_dvc_exp` - If `True`, DVCLive will create a new
+- `save_dvc_exp` - If `True` , DVCLive will create a new
   [DVC experiment](/doc/dvclive/how-it-works#git-integration) as part of
-  `Live.end()`. Defaults to `False`.
+  `Live.end()`. Defaults to `True`.
 
-- `dvcyaml` - If `True`, DVCLive will write
-  [DVC configuration](/doc/user-guide/project-structure/dvcyaml-files) for
-  metrics, plots, and parameters to
-  [`Live.dvc_file`](/doc/dvclive/live#properties) as part of `Live.next_step()`
-  and `Live.end()`. See `Live.make_dvcyaml()`. Defaults to `True`.
+  If you are using DVCLive inside a <abbr>DVC Pipeline</abbr> and running with
+  `dvc exp run`, the option will be ignored.
 
-  <admon type="tip">
+- `dvcyaml` - If `True` (default), DVCLive will write
+  [DVC configuration](/doc/user-guide/project-structure/dvcyaml-files) for
+  metrics, plots, and parameters to `dvc.yaml` (at the root of the repository)
+  as part of `Live.next_step()` and `Live.end()`. See `Live.make_dvcyaml()`.
 
-  If you are already tracking DVCLive metrics, plots, and parameters in your own
-  `dvc.yaml` file, set `dvcyaml=False` to avoid duplication.
+  If a string like `subdir/dvc.yaml`, DVCLive will write the configuration to
+  that path (file must be named `dvc.yaml`).
 
-  </admon>
+  If `False`, DVCLive will not write to `dvc.yaml` (useful if you are tracking
+  DVCLive metrics, plots, and parameters independently and want to avoid
+  duplication).
 
 - `cache_images` - If `True`, DVCLive will <abbr>cache</abbr> any images logged
   with `Live.log_image()` as part of `Live.end()`. Defaults to `False`.

diff --git a/content/docs/dvclive/live/log_image.md b/content/docs/dvclive/live/log_image.md
@@ -11,7 +11,7 @@ def log_image(name: str, val):
 ```py
 from dvclive import Live
 
-with Live(cache_images=True, save_dvc_exp=True) as live:
+with Live(cache_images=True) as live:
     # 1. Log an image from a numpy array:
     import numpy as np
     img_numpy = np.ones((500, 500), np.uint8) * 255
@@ -84,7 +84,7 @@ images using the following pattern
 import numpy as np
 from dvclive import Live
 
-with Live(save_dvc_exp=True) as live:
+with Live() as live:
     base_img = np.ones((500, 500), np.uint8)
     for i in range(10):
       live.log_image(

diff --git a/content/docs/dvclive/live/log_plot.md b/content/docs/dvclive/live/log_plot.md
@@ -44,12 +44,11 @@ The method will dump the provided `datapoints` to
 `{Live.dir}/plots/custom/{name}.json` and store the provided properties to be
 included in the `plots` section written by `Live.make_dvcyaml()`.
 
-The example snippet would produce the following `dvc.yaml` in
-`{Live.dir}/{Live.dvc_file}`:
+The example snippet would produce the following `dvc.yaml`:
 
 ```yaml
 plots:
-  - plots/custom/iris_feature_importance.json:
+  - dvclive/plots/custom/iris_feature_importance.json:
       template: bar_horizontal
       x: importance
       y: name

diff --git a/content/docs/dvclive/live/log_sklearn_plot.md b/content/docs/dvclive/live/log_sklearn_plot.md
@@ -34,19 +34,19 @@ The method will compute and dump the `kind` plot (see
 format compatible with `dvc plots`.
 
 It will also store the provided properties to be included in the `plots` section
-written by `Live.make_dvcyaml()`. The example snippet would produce the
-following `dvc.yaml` in `{Live.dir}/{Live.dvc_file}`:
+written by `Live.make_dvcyaml()`. The example snippet would add the following to
+`dvc.yaml`:
 
 ```yaml
 plots:
-  - plots/sklearn/roc.json:
+  - dvclive/plots/sklearn/roc.json:
       template: simple
       x: fpr
       y: tpr
       title: Receiver operating characteristic (ROC)
       x_label: False Positive Rate
       y_label: True Positive Rate
-  - plots/sklearn/cm.json:
+  - dvclive/plots/sklearn/cm.json:
       template: confusion
       x: actual
       y: predicted

diff --git a/content/docs/dvclive/live/make_dvcyaml.md b/content/docs/dvclive/live/make_dvcyaml.md
@@ -22,8 +22,8 @@ live.make_dvcyaml()
 
 ## Description
 
-Creates `{Live.dir}/dvc.yaml`, which describes and configures metrics, plots,
-and parameters. DVC tools use this file to show reports and experiments tables.
+Creates `dvc.yaml`, which describes and configures metrics, plots, and
+parameters. DVC tools use this file to show reports and experiments tables.
 
 <admon type="info">
 
@@ -34,12 +34,12 @@ If `Live(dvcyaml=True)`, `Live.next_step()` and `Live.end()` will call
 
 ```yaml
 params:
-  - params.yaml
+  - dvclive/params.yaml
 metrics:
-  - metrics.json
+  - dvclive/metrics.json
 plots:
-  - plots/metrics
-  - plots/sklearn/confusion_matrix.json:
+  - dvclive/plots/metrics
+  - dvclive/plots/sklearn/confusion_matrix.json:
       template: confusion
       x: actual
       y: predicted

diff --git a/content/docs/dvclive/ml-frameworks/catalyst.md b/content/docs/dvclive/ml-frameworks/catalyst.md
@@ -34,9 +34,6 @@ Where:
 
 ## Parameters
 
-- `model_file` - (`None` by default) - The name of the file where the model will
-  be saved at the end of each `step`.
-
 - `live` - (`None` by default) - Optional [`Live`] instance. If `None`, a new
   instance will be created using `**kwargs`.
 
@@ -51,7 +48,7 @@ Where:
 from dvclive import Live
 from dvclive.catalyst import DVCLiveCallback
 
-with Live("custom_dir", save_dvc_exp=True) as live:
+with Live("custom_dir") as live:
     runner.train(
         model=model,
         criterion=criterion,
@@ -77,7 +74,7 @@ runner.train(
     loaders=loaders,
     num_epochs=2,
     callbacks=[
-      DVCLiveCallback(save_dvc_exp=True, dir="custom_dir")])
+      DVCLiveCallback(dir="custom_dir")])
 ```
 
 [`live`]: /doc/dvclive/live
diff --git a/content/docs/dvclive/ml-frameworks/fastai.md b/content/docs/dvclive/ml-frameworks/fastai.md
@@ -35,9 +35,6 @@ Where:
 
 ## Parameters
 
-- `model_file` - (`None` by default) - The name of the file where the model will
-  be saved at the end of each `step`.
-
 - `live` - (`None` by default) - Optional [`Live`] instance. If `None`, a new
   instance will be created using `**kwargs`.
 
@@ -52,7 +49,7 @@ Where:
 from dvclive import Live
 from dvclive.fastai import DVCLiveCallback
 
-with Live("custom_dir", save_dvc_exp=True) as live:
+with Live("custom_dir") as live:
     learn = tabular_learner(data_loader, metrics=accuracy)
     learn.fit_one_cycle(
       n_epoch=2,
@@ -67,7 +64,7 @@ with Live("custom_dir", save_dvc_exp=True) as live:
 ```python
 learn.fit_one_cycle(
   n_epoch=2,
-  cbs=[DVCLiveCallback(save_dvc_exp=True, dir="custom_dir")])
+  cbs=[DVCLiveCallback(dir="custom_dir")])
 ```
 
 [`live`]: /doc/dvclive/live