Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: list/import/get: document --config/--remote/--remote-config #4722

Merged
merged 1 commit into from
Aug 8, 2023
Merged

docs: list/import/get: document --config/--remote/--remote-config #4722

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion content/docs/command-reference/get.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@ directory.

```usage
usage: dvc get [-h] [-q | -v] [-o <path>] [--rev <commit>]
[--show-url] [-j <number>] [-f] url path
[--show-url] [-j <number>] [-f]
[--config <path>] [--remote <name>]
[--remote-config [<name>=<value> ...]]
url path

positional arguments:
url Location of DVC or Git repository to download from
Expand Down Expand Up @@ -82,6 +85,16 @@ name.
storage location (URL) of the target data. If `path` is a Git-tracked file,
this option is ignored.

- `--config <path>` - path to a [config file](/doc/command-reference/config)
that will be merged with the config in the target repository.

- `--remote <name>` - name of the `dvc remote` to set as a default in the target
repository.

- `--remote-config [<name>=<value> ...]` - `dvc remote` config options to merge
with a remote's config (default or one specified by `--remote`) in the target
repository.

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
Expand Down Expand Up @@ -219,3 +232,15 @@ Untracked files:
[get started example repo]: https://github.com/iterative/example-get-started
[switching between versions]:
/doc/start/data-management/data-versioning#switching-between-versions

## Example: Set AWS profile for default remote

```cli
$ dvc get https://github.com/iterative/example-get-started-s3 data/prepared --remote-config profile=myprofile
```

## Example: Set default remote

```cli
$ dvc get https://github.com/iterative/example-get-started-s3 data/prepared --remote myremote
```
116 changes: 116 additions & 0 deletions content/docs/command-reference/import.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ usage: dvc import [-h] [-q | -v]
[-o <path>] [--rev <commit>]
[--no-exec | --no-download]
[-j <number>]
[--config <path>] [--remote <name>]
[--remote-config [<name>=<value> ...]]
url path

positional arguments:
Expand Down Expand Up @@ -118,6 +120,16 @@ file.
speed up the operation. Note that the default value can be set in the source
repo using the `jobs` config option of `dvc remote modify`.

- `--config <path>` - path to a [config file](/doc/command-reference/config)
that will be merged with the config in the target repository.

- `--remote <name>` - name of the `dvc remote` to set as a default in the target
repository.

- `--remote-config [<name>=<value> ...]` - `dvc remote` config options to merge
with a remote's config (default or one specified by `--remote`) in the target
repository.

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
Expand Down Expand Up @@ -386,3 +398,107 @@ C, otherwise the import chain resolution would fail.

The `dvc remote default` for all repos in the import chain must also be
accessible (repo C needs to have all the appropriate credentials).

## Example: Set default remote

```cli
$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared --remote myremote
...
$ cat prepared.dvc
deps:
- path: data/prepared
repo:
url: https://github.com/iterative/example-get-started-s3
rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b
remote: myremote
outs:
- md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir
size: 27
nfiles: 3
path: prepared
```

## Example: Set AWS profile for default remote

```cli
$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared --remote-config profile=myprofile
...
$ cat prepared.dvc
deps:
- path: data/prepared
repo:
url: https://github.com/iterative/example-get-started-s3
rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b
remote:
profile: myprofile
outs:
- md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir
size: 27
nfiles: 3
path: prepared
```

## Example: Create new AWS S3 remote and set it as default

If remote with that name already exists, its config will be merged with options
provided by `--remote-config`.

```cli
$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared \
--remote myremote \
--remote-config url=s3://mybucket/mypath profile=myprofile
...
$ cat prepared.dvc
deps:
- path: data/prepared
repo:
url: https://github.com/iterative/example-get-started-s3
rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b
config:
core:
remote: myremote
remote:
myremote:
url: s3://mybucket/mypath
profile: myprofile
outs:
- md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir
size: 27
nfiles: 3
path: prepared
```

## Example: Set AWS secret keys for particular remote

In this example, instead of using `--remote myremote` with `--remote-config` and
exposing your secrets in dvcfile, you could use `--config` to use a gitignored
config file. The format of the config file is the same as produced by
`dvc config`.

```cli
$ cat myconfig
[core]
remote = myremote

[remote "myremote"]
access_key_id = myaccesskeyid
secret_access_key = mysecretaccesskey
$ cat .gitignore # make sure you are not commiting this file to git
...
/myconfig
...
$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared --config myconfig
...
$ cat prepared.dvc
deps:
- path: data/prepared
repo:
url: https://github.com/iterative/example-get-started-s3
rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b
config: myconfig
outs:
- md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir
size: 27
nfiles: 3
path: prepared
```
24 changes: 24 additions & 0 deletions content/docs/command-reference/list.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ and by Git.
```usage
usage: dvc list [-h] [-q | -v] [-R] [--dvc-only]
[--json] [--rev [<commit>]]
[--config <path>] [--remote <name>]
[--remote-config [<name>=<value> ...]]
url [path]

positional arguments:
Expand Down Expand Up @@ -71,6 +73,16 @@ accessed with `dvc get`, `dvc import`, or `dvc.api`.
- `--json` - prints the command's output in easily parsable JSON format, instead
of a human-readable table.

- `--config <path>` - path to a [config file](/doc/command-reference/config)
that will be merged with the config in the target repository.

- `--remote <name>` - name of the `dvc remote` to set as a default in the target
repository.

- `--remote-config [<name>=<value> ...]` - `dvc remote` config options to merge
with a remote's config (default or one specified by `--remote`) in the target
repository.

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
Expand Down Expand Up @@ -161,3 +173,15 @@ installed):
```cli
$ dvc list . -R --dvc-only | xargs python -m zipfile -c data.zip
```

## Example: Set AWS profile for default remote

```cli
$ dvc list https://github.com/iterative/example-get-started-s3 data/prepared --remote-config profile=myprofile
```

## Example: Set default remote

```cli
$ dvc list https://github.com/iterative/example-get-started-s3 data/prepared --remote myremote
```
14 changes: 8 additions & 6 deletions content/docs/user-guide/project-structure/dvc-files.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,14 +73,16 @@ The following subfields may be present under `deps` entries:
| `md5`<br/>`etag`<br/>`checksum` | Only in <abbr>external dependencies</abbr> created with `dvc import-url`: Hash value of the imported file or directory. MD5 is used for local paths and SSH; [ETag] for HTTP, S3, GCS, and Azure; and a special _checksum_ for HDFS and WebHDFS. |
| `size` | Size of the file or directory (sum of all files). |
| `nfiles` | If this dependency is a directory, the number of files inside (recursive). |
| `repo` | Only in external dependencies created with `dvc import`: It can contain `url`, `rev`, and `rev_lock` (detailed below). |
| `repo` | Only in external dependencies created with `dvc import`: It can contain `url`, `rev`, `rev_lock`, `config` and `remote` (detailed below). |

### Dependency `repo` subfields:

| Field | Description |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | URL of Git repository with source DVC project |
| `rev` | Only when `dvc import --rev` is used: Specific commit hash, branch or tag name, etc. (a [Git revision]) used to import the dependency from. |
| `rev_lock` | Git commit hash of the external <abbr>DVC repository</abbr> at the time of importing or updating the dependency (with `dvc update`) |
| Field | Description |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | URL of Git repository with source DVC project |
| `rev` | Only when `dvc import --rev` is used: Specific commit hash, branch or tag name, etc. (a [Git revision]) used to import the dependency from. |
| `rev_lock` | Git commit hash of the external <abbr>DVC repository</abbr> at the time of importing or updating the dependency (with `dvc update`) |
| `config` | When `dvc import --config` is used: Path to a [config file](/doc/command-reference/config) that will be merged with the config in the target repository. When both `--remote` and `--remote-config` are used: config options that will be merged with the config in the target repository. See examples section in`dvc import`. |
| `remote` | Only when `dvc import --remote` or `--remote-config` is used: name of the `dvc remote` to set as a default or remote config options to merge with a default remote's config in the target repository. See examples section in `dvc import`. |

[git revision]: https://git-scm.com/docs/revisions