Skip to content

Commit

Permalink
fix: add some explanation on the readme for the file list option (#105)
Browse files Browse the repository at this point in the history
  • Loading branch information
renaudjester authored and uriii3 committed Jul 31, 2024
1 parent 7dcc768 commit 27ca798
Showing 1 changed file with 20 additions and 7 deletions.
27 changes: 20 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,65 +11,78 @@

## Features


The `copernicusmarine` offers capabilities through both **Command Line Interface (CLI)** and **Python API**:


- **Metadata Information**: List and retrieve metadata information on all variables, datasets, products, and their associated documentation.
- **Subset Datasets**: Subset datasets to extract only the parts of interest, in preferred format, such as Analysis-Ready Cloud-Optimized (ARCO) Zarr or NetCDF file format.
- **Advanced Filters**: Apply simple or advanced filters to get multiple files, in original formats like NetCDF/GeoTIFF, via direct Marine Data Store connections.
- **No Quotas**: Enjoy no quotas, neither on volume size nor bandwidth.

## Installation


For installation, multiple options are available depending on your setup:

### Mamba | Conda


A `conda` package is available on [Anaconda](https://anaconda.org/conda-forge/copernicusmarine).

You can install it using `mamba` (or conda) through the `conda-forge` channel with the following command:


```bash
mamba install conda-forge::copernicusmarine --yes
```

To upgrade the Toolbox with mamba (or conda):


```bash
mamba update --name copernicusmarine copernicusmarine --yes
```

### Docker


A docker image is also available here: [https://hub.docker.com/r/copernicusmarine/copernicusmarine](https://hub.docker.com/r/copernicusmarine/copernicusmarine)

First step is to pull the container image:


```bash
docker pull copernicusmarine/copernicusmarine:latest
```

Then run it:


```bash
docker run -it --rm copernicusmarine/copernicusmarine --version
```

### Pip


Otherwise, if you already have an environment (safer to clone it), the package can be installed using the `pip` command:


```bash
python -m pip install copernicusmarine
```

And to **upgrade the package** to the newest available version, run:


```bash
python -m pip install copernicusmarine --upgrade
```

## User Guide


For more comprehensive details on how to use the `copernicusmarine` Toolbox, please refer to our [Help Center](https://help.marine.copernicus.eu/en/collections/9080063-copernicus-marine-toolbox). It ensures a smooth migration for existing users of legacy services such as MOTU, OPeNDAP or FTP.

### Network configuration
Expand Down Expand Up @@ -161,6 +174,7 @@ You can use the `--skip-if-user-logged-in` option to skip the configuration file

#### Access points migration and evolution


If you still have a configuration for legacy services (e.g. `~/motuclient/motuclient-python.ini`, `~/.netrc` or `~/_netrc`) in your home directory, it will automatically be taken into account with commands `get` and `subset` without the need for running the `login` command.
If the configuration files are already available in another directory, when running commands `subset` or `get`, you can use the `--credentials-file` option to point to the files.

Expand Down Expand Up @@ -389,6 +403,7 @@ Example of `file_list.txt` with paths that would be directly downloaded without
> another_dataset/history/BO/AR_PR_BO_58JM.nc
```


Example of `file_list.txt` with absolute paths:

```txt
Expand All @@ -397,13 +412,11 @@ s3://mdl-native-10/native/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083de
s3://mdl-native-10/native/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1M-m_202012/2021/CMEMS_v5r1_IBI_PHY_MY_PdE_01mav_20210301_20210331_R20230101_RE01.nc
```

Example of `file_list.txt` with partial paths matching the absolute paths (equivalent to previous example):
Note that a path to a file can be seen in 3 parts:

```txt
CMEMS_v5r1_IBI_PHY_MY_NL_01yav_20000101_20001231_R20221101_RE01.nc
CMEMS_v5r1_IBI_PHY_MY_NL_01yav_20010101_20011231_R20221101_RE01.nc
CMEMS_v5r1_IBI_PHY_MY_NL_01yav_20020101_20021231_R20221101_RE01.nc
```
- the provenance that indicates in which bucket the data is. For example, `s3://mdl-native-10/`. It can be found in the metadata.
- the productID and datasetID. For example, `IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1M-m_202012/`. It also contains the version when the dataset has one.
- the filename which is everything that comes after the dataset id. For example, `2021/CMEMS_v5r1_IBI_PHY_MY_PdE_01mav_20210301_20210331_R20230101_RE01.nc`. It should be considered like a filename. If any components are absent, the file name is not complete and the file cannot be directly downloaded. Thus a listing of all the files is necessary in order to download the file. For example, `2021/CMEMS_v5r1_IBI_PHY_MY_PdE_01mav_20210301_20210331_R20230101_RE01.nc` is a filename and `CMEMS_v5r1_IBI_PHY_MY_PdE_01mav_20210301_20210331_R20230101_RE01.nc` is an incomplete filename.

> **_NOTE:_** This option is compatible with the file generated by the `--create-file-list` option if you generated a ".txt" file.
Expand Down Expand Up @@ -440,7 +453,7 @@ copernicusmarine get --dataset-id cmems_obs-ins_blk_phybgcwav_mynrt_na_irr --ind

Returns:

```text
```txt
INFO - 2024-04-03T12:58:40Z - Dataset version was not specified, the latest one was selected: "202311"
INFO - 2024-04-03T12:58:40Z - Dataset part was not specified, the first one was selected: "history"
INFO - 2024-04-03T12:58:40Z - You forced selection of service: original-files
Expand Down

0 comments on commit 27ca798

Please sign in to comment.