From 6ce1e80fde620444be4fc8241e653e6440d15c46 Mon Sep 17 00:00:00 2001 From: voetberg Date: Fri, 13 Dec 2024 11:12:34 -0600 Subject: [PATCH 1/2] Clients: Update bin --help, add mapping for >=36.0 --- docs/user/migrating_from_35_client.md | 118 ++++++++++++++++++ tools/run_in_docker/generate_bin_help_docs.sh | 38 ++++++ website/sidebars.json | 3 +- 3 files changed, 158 insertions(+), 1 deletion(-) create mode 100644 docs/user/migrating_from_35_client.md diff --git a/docs/user/migrating_from_35_client.md b/docs/user/migrating_from_35_client.md new file mode 100644 index 00000000000..9a91b3617cc --- /dev/null +++ b/docs/user/migrating_from_35_client.md @@ -0,0 +1,118 @@ +--- +title: Migrating from pre-36.0.0 Clients +--- + +In Rucio 36.0.0, a new rucio client was released. +This new client redos the entire structure of both `rucio` and `rucio-admin`. +`rucio-admin` is made redundant by the update, their commands are included in the `rucio >=36.0` client. + +Note: + Please report any issues you have with the new client [on this github issue](https://github.com/rucio/rucio/issues/7224) + +# Legacy Mapping + +* Using `rucio-admin` or `rucio --legacy` automatically uses the old version. Old help menus are still visible with this method. +* Old commands still work - they display a warning. +* Not all commands are included in the 36 Client Release. Not all are one-to-one mapped. +* A warning it printed (not logged) to consoles to encourage migration, along with the closest mapped command. + + +# Remapping Existing Commands + +|`rucio <36.0`| `rucio >=36.0`| +| ------------- | ------------- | +| rucio list-file-replicas | rucio replica list | +| rucio list-dataset-replicas | rucio replica dataset | +| rucio add-dataset | rucio did add --type dataset | +| rucio add-container | rucio did add --type container | +| rucio attach | rucio did content add | +| rucio detach | rucio did content remove | +| rucio ls | rucio did list | +| rucio list-dids | rucio did list | +| rucio list-parent-dids | rucio did show --parent | +| rucio list-scopes | rucio scope list | +| rucio close | rucio did update --close | +| rucio reopen | rucio did update --open | +| rucio stat | rucio did show | +| rucio erase | rucio did remove | +| rucio list-content | rucio did content list | +| rucio list-content-history | rucio did content history | +| rucio upload | rucio upload | +| rucio get | rucio download | +| rucio download | rucio download | +| rucio get-metadata | rucio did metadata list | +| rucio set-metadata | rucio did metadata add | +| rucio delete-metadata | rucio did metadata remove | +| rucio list-rse-usage | rucio rse show | +| rucio list-account-usage | rucio account usage | +| rucio list-account-limits | rucio account limit list | +| rucio add-rule | rucio rule add | +| rucio delete-rule | rucio rule remove | +| rucio rule-info | rucio rule info | +| rucio list-rules | rucio rule list | +| rucio list-rules-history | rucio rule history | +| rucio update-rule | rucio rule update | +| rucio move-rule | rucio rule update --move | +| rucio list-rses | rucio rse list | +| rucio list-suspicious-replicas | rucio replica state suspicious | +| rucio list-rse-attributes | rucio rse attribute list | +| rucio touch | rucio did update --touch | +| rucio add-lifetime-exception | rucio lifetime-exception add | + +|`rucio-admin <36.0`| `rucio >=36.0`| +| ------------- | ------------- | +| rucio data import | rucio upload | +| rucio data export | rucio download | +| rucio account -h | rucio account -h | +| rucio account add | rucio account add | +| rucio account delete | rucio account remove | +| rucio account info | rucio account show | +| rucio account update | rucio account update | +| rucio account set-limits | rucio account limit add | +| rucio account get-limits | rucio account limit list | +| rucio account delete-limits | rucio account limit remove | +| rucio account ban | rucio account update --ban True | +| rucio account unban | rucio account update --ban False | +| rucio account list-attributes | rucio account attribute list | +| rucio account add-attribute | rucio account attribute add | +| rucio account delete-attribute | rucio account attribute remove | +| rucio identity -h | rucio account identity -h | +| rucio identity add | rucio account identity add | +| rucio identity delete | rucio account identity remove | +| rucio rse -h | rucio rse -h | +| rucio rse add | rucio rse add | +| rucio rse list | rucio rse list | +| rucio rse update | rucio rse update | +| rucio rse info | rucio rse show | +| rucio rse set-attribute | rucio rse attribute add | +| rucio rse delete-attribute | rucio rse attribute remove | +| rucio rse delete-distance | rucio rse distance remove | +| rucio rse get-distance | rucio rse distance list | +| rucio rse get-attribute | rucio rse attribute list | +| rucio rse add-protocol | rucio rse protocol add | +| rucio rse delete-protocol | rucio rse protocol remove | +| rucio rse delete | rucio rse remove | +| rucio rse add-qos-policy | rucio rse qos add | +| rucio rse add-distance | rucio rse distance add | +| rucio rse delete-qos-policy | rucio rse qos remove | +| rucio rse list-qos-policies | rucio rse qos list | +| rucio rse set-limit | rucio rse limit add | +| rucio rse delete-limit | rucio rse limit remove | +| rucio scope -h | rucio scope -h | +| rucio scope add | rucio scope add | +| rucio scope list | rucio scope list | +| rucio config -h | rucio config -h | +| rucio config get | rucio config list | +| rucio config set | rucio config add | +| rucio config delete | rucio config remove | +| rucio subscription -h | rucio subscription -h | +| rucio subscription add | rucio subscription add | +| rucio subscription list | rucio subscription | +| rucio subscription update | rucio subscription update | +| rucio subscription reevaluate | rucio subscription touch | +| rucio replicas -h | rucio replica -h | +| rucio replicas quarantine | rucio replica state quarantine | +| rucio replicas declare-bad | rucio replica state bad | +| rucio replicas declare-temporary-unavailable | rucio replica state unavailable | +| rucio replicas set-tombstone | rucio replica remove | +| rucio replicas list-pfns | rucio replica list file --pfns | \ No newline at end of file diff --git a/tools/run_in_docker/generate_bin_help_docs.sh b/tools/run_in_docker/generate_bin_help_docs.sh index 557da7485e5..d3273928ed0 100755 --- a/tools/run_in_docker/generate_bin_help_docs.sh +++ b/tools/run_in_docker/generate_bin_help_docs.sh @@ -17,10 +17,48 @@ function generate_bin_help_for_file { echo "\`\`\`" } +function generate_bin_help_for_main { + echo "---" + echo "title: rucio" + echo "---" + + echo "" + echo "# client" + echo "\`\`\`" + + # shellcheck disable=SC2005 + echo "$(rucio --help)" + + echo "\`\`\`" + + for i in account config did lifetime-exception replica rse rule scope subscription; do + echo "" + echo "# "$i" " + echo "\`\`\`" + + # shellcheck disable=SC2005 + echo "$(rucio "$i" --help)" + + echo "\`\`\`" + done + + echo "" + echo "# legacy client" + echo "\`\`\`" + + # shellcheck disable=SC2005 + echo "$(rucio -h --legacy)" + + echo "\`\`\`" + +} echo "Generating the Executable Help Docs..." + mkdir -p /auto_generated/bin for f in rucio/bin/*; do generate_bin_help_for_file "$f" > /auto_generated/bin/"$(basename "$f")".md done + +generate_bin_help_for_main > /auto_generated/bin/rucio.md \ No newline at end of file diff --git a/website/sidebars.json b/website/sidebars.json index 68b33c8ad72..a89065eb0b7 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -40,7 +40,8 @@ } ] }, - "user/developing_with_rucio" + "user/developing_with_rucio", + "user/migrating_from_35_client" ], "Operator": [ "operator/setting_up_demo", From c5a1dcbc2c4b8778061b739b0195558c9a02b615 Mon Sep 17 00:00:00 2001 From: voetberg Date: Fri, 13 Dec 2024 11:48:01 -0600 Subject: [PATCH 2/2] Update the CLI user docs to address updated CLI --- docs/index.md | 1 - docs/user/using_the_admin_client.md | 137 ------------------------ docs/user/using_the_client.md | 155 +++++++++++++++++++++++----- website/sidebars.json | 9 -- 4 files changed, 130 insertions(+), 172 deletions(-) delete mode 100644 docs/user/using_the_admin_client.md diff --git a/docs/index.md b/docs/index.md index 69c6f390b6b..3c647f00842 100644 --- a/docs/index.md +++ b/docs/index.md @@ -36,7 +36,6 @@ from single files up to Petabyte sized datasets. - [Setting Up the Rucio Client](user/setting_up_the_rucio_client.md) - [Using the Client](user/using_the_client.md) -- [Using the Admin Client](user/using_the_admin_client.md) ## Administration diff --git a/docs/user/using_the_admin_client.md b/docs/user/using_the_admin_client.md deleted file mode 100644 index c3341d5e7c8..00000000000 --- a/docs/user/using_the_admin_client.md +++ /dev/null @@ -1,137 +0,0 @@ ---- -id: using_the_admin_client -title: "Using the Admin Client" ---- - -Rucio provides a CLI for administrative tasks. The get methods can be executed -by any user, but the set methods require some admin privileges. See the -[rucio-admin help page](bin/rucio-admin.md). -The command line client for administrative tasks is called ``rucio-admin``. - -## Getting help - -To get an overview of the available ``rucio-admin`` subcommands and flags, run: - -```bash -rucio-admin --help -``` - -## Enable command line autocompletion - -If you would like to automatically complete ``rucio-admin`` commands, install -the [argcomplete](https://pypi.org/project/argcomplete/) package and run: - -```bash -eval "$(register-python-argcomplete rucio-admin)" -``` - -Next, type ``rucio-admin `` (note the trailing space) and press the -Tab key to see all available options. To use the autocompletion -feature, type enough letters of a subcommand or flag to uniquely define it -and then press Tab. - -## Account and identity methods - -To create a new account: - -```bash - $ rucio-admin account add --type USER --email jdoe@blahblih.com jdoe -``` - -You can choose different types in the list USER, GROUP, SERVICE. Different -policies/permissions can be set depending on the account type. Once the account -is created, you need to create and attach an identity to this account: - -```bash - $ rucio-admin identity add --type X509 \ - --id "CN=jdoe,OU=Users,OU=Organic Units,DC=blih,DC=blah" \ - --email jdoe@blahblih.com --account jdoe -``` - -The list of possible identity types is X509, GSS, USERPASS, SSH: - -```bash - $ rucio-admin account list-identities jdoe - Identity: CN=jdoe,OU=Users,OU=Organic Units,DC=blih,DC=blah, type: X509 -``` - -You can set attributes to the users: - -```bash - $ rucio-admin account add-attribute --key country --value xyz jdoe -``` - -And list these attributes: - -```bash - $ rucio-admin account list-attributes jdoe - +---------+-------+ - | Key | Value | - |---------+-------| - | country | xyz | - +---------+-------+ -``` - -You can also list all the accounts matching a certain attribute using the filter -option: - -```bash - $ rucio-admin account list --filters "country=xyz" - jdoe -``` - -To set the quota for one account on a given RSE: - -```bash - $ rucio-admin account set-limits jdoe SITE2_SCRATCH 10000000000000 - Set account limit for account jdoe on RSE SITE2_SCRATCH: 10.000 TB - $ rucio-admin account get-limits dcameron SITE2_SCRATCH - Quota on SITE2_SCRATCH for jdoe : 10 TB -``` - -## Scope methods - -To create a new scope: - -```bash - $ rucio-admin scope add --account jdoe --scope user.jdoe -``` - -Only the owner of the scope or privileged users can write into the scope. - -To list all the scopes: - -```bash - $ rucio-admin scope list - user.janedoe - user.jdoe -``` - -## RSE methods - -To create a new RSE: - -```bash - $ rucio-admin rse add SITE2_SCRATCH -``` - -To add a RSE attribute: - -```bash - $ rucio-admin rse set-attribute --rse SITE2_SCRATCH --key country --value xyz -``` - -To check an RSE attribute: - -```bash - $ rucio-admin rse get-attribute SITE2_SCRATCH - country: xyz -``` - -## Replica methods - -To declare bad (i.e. corrupted or lost replicas): - -```bash - $ rucio-admin replicas declare-bad --reason "File corrupted" https//path/to/lost/file -``` diff --git a/docs/user/using_the_client.md b/docs/user/using_the_client.md index 8f158252769..e4165b9ef74 100644 --- a/docs/user/using_the_client.md +++ b/docs/user/using_the_client.md @@ -6,6 +6,12 @@ title: Using the Client Rucio provides several commands for the end-user. See [executables](bin/rucio.md). The command line client is called ``rucio``. +Please note that these commands are not meant to be included in python scripts. +For such applications, we encourage using the [python client](pathname:///html/site/client.html). + +If you are using a version of the client prior to ~36.0, please view the [migration guide](user/migrating_from_35_client.md). + +# Basic Commands ## Getting help @@ -257,7 +263,7 @@ auth_token_file_path = /path/to/token/file You can query the list of available RSEs: ```bash -$ rucio list-rses +$ rucio rse list SITE1_DISK SITE1_TAPE SITE2_DISK @@ -269,7 +275,7 @@ If the RSEs are tagged with attributes you can build RSE expressions and query the sites matching these expressions: ```bash -$ rucio list-rses --rses "tier=1&disk=1" +$ rucio rse list --rses "tier=1&disk=1" SITE1_DISK SITE2_DISK ``` @@ -279,7 +285,7 @@ SITE2_DISK To list all the possible scopes: ```bash -$ rucio list-scopes +$ rucio scope list mc data user.jdoe @@ -290,7 +296,7 @@ You can query the DIDs matching a certain pattern. It always requires to specify the scope in which you want to search: ```bash -$ rucio list-dids user.jdoe:* +$ rucio did list --did user.jdoe:* +-------------------------------------------+--------------+ | SCOPE:NAME | [DID TYPE] | |-------------------------------------------+--------------| @@ -307,7 +313,7 @@ $ rucio list-dids user.jdoe:* You can filter by key/value, e.g.: ```bash -$ rucio list-dids --filter type=CONTAINER +$ rucio did list --filter type=CONTAINER --did user.jdoe:* +-------------------------------------------+--------------+ | SCOPE:NAME | [DID TYPE] | |-------------------------------------------+--------------| @@ -320,7 +326,7 @@ If you want to resolve a collection (CONTAINER or DATASET) into the list of its constituents: ```bash -$ rucio list-content user.jdoe:user.jdoe.test.container.1234.1 +$ rucio did content list --did user.jdoe:user.jdoe.test.container.1234.1 +------------------------------------+--------------+ | SCOPE:NAME | [DID TYPE] | |------------------------------------+--------------| @@ -329,30 +335,15 @@ $ rucio list-content user.jdoe:user.jdoe.test.container.1234.1 +------------------------------------+--------------+ ``` -You can resolve also the collections (CONTAINER or DATASET) into the list of -files: - -```bash -$ rucio list-files user.jdoe:user.jdoe.test.container.1234.1 -+-----------------------+---------+-------------+------------+----------+ -| SCOPE:NAME | GUID | ADLER32 | FILESIZE | EVENTS | -|-----------------------+---------+-------------+------------+----------| -| user.jdoe:test.file.1 | 9DF3... | ad:56fb0723 | 39.247 kB | | -| user.jdoe:test.file.2 | 67E8... | ad:e3e573b5 | 636.075 kB | | -| user.jdoe:test.file.3 | 32CD... | ad:22849380 | 641.427 kB | | -+-----------------------+---------+-------------+------------+----------+ -Total files : 3 -Total size : 1.316 MB: -``` - ## Rules operations You can create a new rule like this: ```bash $ rucio add-rules --lifetime 1209600 \ - user.jdoe:user.jdoe.test.container.1234.1 1 \ - "tier=1&disk=1" + --did user.jdoe:user.jdoe.test.container.1234.1 \ + --copies 1 \ + --rses "tier=1&disk=1" a12e5664555a4f12b3cc6991db5accf9 ``` @@ -361,7 +352,7 @@ The command returns the rule_id of the rule. You can list the rules for a particular DID: ```bash -$ rucio list-rules user.jdoe:user.jdoe.test.container.1234.1 +$ rucio rule list --did user.jdoe:user.jdoe.test.container.1234.1 ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES ---- ------- ---------- -------------------- -------------- ------ ------- a... jdoe user.... OK[3/0/0] tier=1&disk=1 1 2018... @@ -409,3 +400,117 @@ Downloaded files : 3 Files already found locally : 0 Files that cannot be downloaded : 0 ``` + +# Operator Commands + +Rucio provides a CLI for administrative tasks. The get methods can be executed +by any user, but the set methods require some admin privileges. + + +## Account and identity methods + +To create a new account: + +```bash + $ rucio account add --type USER --email jdoe@blahblih.com jdoe +``` + +You can choose different types in the list USER, GROUP, SERVICE. Different +policies/permissions can be set depending on the account type. Once the account +is created, you need to create and attach an identity to this account: + +```bash + $ rucio account identity add --type X509 \ + --id "CN=jdoe,OU=Users,OU=Organic Units,DC=blih,DC=blah" \ + --email jdoe@blahblih.com \ + --account jdoe +``` + +The list of possible identity types is X509, GSS, USERPASS, SSH, SAML, OIDC: + +```bash + $ rucio account identity list --account jdoe + Identity: CN=jdoe,OU=Users,OU=Organic Units,DC=blih,DC=blah, type: X509 +``` + +You can set attributes to the users: + +```bash + $ rucio account attribute add --key country --value xyz --account jdoe +``` + +And list these attributes: + +```bash + $ rucio-admin account attribute list --account jdoe + +---------+-------+ + | Key | Value | + |---------+-------| + | country | xyz | + +---------+-------+ +``` + +You can also list all the accounts matching a certain attribute using the filter +option: + +```bash + $ rucio account list --filters "country=xyz" + jdoe +``` + +To add the quota for one account on a given RSE: + +```bash + $ rucio account limit add --account jdoe --rses SITE2_SCRATCH --bytes 10000000000000 + Set account limit for account jdoe on RSE SITE2_SCRATCH: 10.000 TB + + $ rucio account limit list --account jdoe --rses SITE2_SCRATCH + Quota on SITE2_SCRATCH for jdoe : 10 TB +``` + +## Scope methods + +To create a new scope: + +```bash + $ rucio scope add --account jdoe --scope user.jdoe +``` + +Only the owner of the scope or privileged users can write into the scope. + +To list all the scopes: + +```bash + $ rucio scope list + user.janedoe + user.jdoe +``` + +## RSE methods + +To create a new RSE: + +```bash + $ rucio rse add --rse SITE2_SCRATCH +``` + +To add a RSE attribute: + +```bash + $ rucio rse attribute add --rse SITE2_SCRATCH --key country --value xyz +``` + +To check an RSE attribute: + +```bash + $ rucio rse attribute list --rse SITE2_SCRATCH + country: xyz +``` + +## Replica methods + +To declare bad (i.e. corrupted or lost replicas): + +```bash + $ rucio replica state update bad --reason "File corrupted" --files https//path/to/lost/file +``` diff --git a/website/sidebars.json b/website/sidebars.json index a89065eb0b7..02160ea3fca 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -31,15 +31,6 @@ "user/setting_up_the_rucio_client", "user/configuring_the_client", "user/using_the_client", - "user/using_the_admin_client", - { - "Python Client API": [ - { - "type": "autogenerated", - "dirName": "client_api" - } - ] - }, "user/developing_with_rucio", "user/migrating_from_35_client" ],