From 7c12ca4c905ac29edc892431e0fc95c08270cb72 Mon Sep 17 00:00:00 2001 From: Callum Dickinson Date: Fri, 16 Aug 2024 19:37:07 +1200 Subject: [PATCH] Improvement documentation for using Object Storage containers * Add guidelines for naming containers and objects. * Rewrite the section on using Object Storage via the CLI, with more detail, nicer formatting, and ease of navigation via anchors. * Add documentation for using the Swift Client to manage containers. * Some other related small fixes and improvements. --- source/identity-access-management/roles.rst | 2 +- .../cli/aws-cli/container-create.rst | 23 + .../cli/aws-cli/container-delete.rst | 23 + .../cli/aws-cli/container-list.rst | 24 + .../cli/aws-cli/container-show.rst | 3 + .../cli/aws-cli/object-create.rst | 20 + .../cli/aws-cli/object-delete.rst | 15 + .../cli/aws-cli/object-list.rst | 57 +++ .../cli/aws-cli/object-save.rst | 30 ++ .../cli/aws-cli/object-show.rst | 16 + .../cli/aws-cli/prerequisites.rst | 76 ++++ .../catalystcloud-client/container-create.rst | 30 ++ .../catalystcloud-client/container-delete.rst | 14 + .../catalystcloud-client/container-list.rst | 7 + .../catalystcloud-client/container-show.rst | 15 + .../catalystcloud-client/object-create.rst | 36 ++ .../catalystcloud-client/object-delete.rst | 7 + .../cli/catalystcloud-client/object-list.rst | 12 + .../cli/catalystcloud-client/object-save.rst | 20 + .../cli/catalystcloud-client/object-show.rst | 18 + .../catalystcloud-client/prerequisites.rst | 27 ++ .../cli/curl/container-create.rst | 19 + .../cli/curl/container-delete.rst | 23 + .../cli/curl/container-list.rst | 47 ++ .../cli/curl/container-show.rst | 26 ++ .../object-storage/cli/curl/object-create.rst | 28 ++ .../object-storage/cli/curl/object-delete.rst | 19 + .../object-storage/cli/curl/object-list.rst | 38 ++ .../object-storage/cli/curl/object-save.rst | 44 ++ .../object-storage/cli/curl/object-show.rst | 28 ++ .../object-storage/cli/curl/prerequisites.rst | 23 + .../cli/swift-client/container-create.rst | 14 + .../cli/swift-client/container-delete.rst | 19 + .../cli/swift-client/container-list.rst | 7 + .../cli/swift-client/container-show.rst | 23 + .../cli/swift-client/object-create.rst | 24 + .../cli/swift-client/object-delete.rst | 8 + .../cli/swift-client/object-list.rst | 8 + .../cli/swift-client/object-save.rst | 39 ++ .../cli/swift-client/object-show.rst | 19 + .../cli/swift-client/prerequisites.rst | 28 ++ .../object-storage/tutorial-scripts/curl.rst | 104 ----- .../tutorial-scripts/endpoint-urls.rst | 32 ++ .../object-storage/tutorial-scripts/heat.rst | 15 +- .../tutorial-scripts/openstackCLI.rst | 86 ---- .../tutorial-scripts/swiftAPI.rst | 44 +- source/object-storage/using-container.rst | 410 ++++++++++++++++-- 47 files changed, 1397 insertions(+), 253 deletions(-) create mode 100644 source/object-storage/cli/aws-cli/container-create.rst create mode 100644 source/object-storage/cli/aws-cli/container-delete.rst create mode 100644 source/object-storage/cli/aws-cli/container-list.rst create mode 100644 source/object-storage/cli/aws-cli/container-show.rst create mode 100644 source/object-storage/cli/aws-cli/object-create.rst create mode 100644 source/object-storage/cli/aws-cli/object-delete.rst create mode 100644 source/object-storage/cli/aws-cli/object-list.rst create mode 100644 source/object-storage/cli/aws-cli/object-save.rst create mode 100644 source/object-storage/cli/aws-cli/object-show.rst create mode 100644 source/object-storage/cli/aws-cli/prerequisites.rst create mode 100644 source/object-storage/cli/catalystcloud-client/container-create.rst create mode 100644 source/object-storage/cli/catalystcloud-client/container-delete.rst create mode 100644 source/object-storage/cli/catalystcloud-client/container-list.rst create mode 100644 source/object-storage/cli/catalystcloud-client/container-show.rst create mode 100644 source/object-storage/cli/catalystcloud-client/object-create.rst create mode 100644 source/object-storage/cli/catalystcloud-client/object-delete.rst create mode 100644 source/object-storage/cli/catalystcloud-client/object-list.rst create mode 100644 source/object-storage/cli/catalystcloud-client/object-save.rst create mode 100644 source/object-storage/cli/catalystcloud-client/object-show.rst create mode 100644 source/object-storage/cli/catalystcloud-client/prerequisites.rst create mode 100644 source/object-storage/cli/curl/container-create.rst create mode 100644 source/object-storage/cli/curl/container-delete.rst create mode 100644 source/object-storage/cli/curl/container-list.rst create mode 100644 source/object-storage/cli/curl/container-show.rst create mode 100644 source/object-storage/cli/curl/object-create.rst create mode 100644 source/object-storage/cli/curl/object-delete.rst create mode 100644 source/object-storage/cli/curl/object-list.rst create mode 100644 source/object-storage/cli/curl/object-save.rst create mode 100644 source/object-storage/cli/curl/object-show.rst create mode 100644 source/object-storage/cli/curl/prerequisites.rst create mode 100644 source/object-storage/cli/swift-client/container-create.rst create mode 100644 source/object-storage/cli/swift-client/container-delete.rst create mode 100644 source/object-storage/cli/swift-client/container-list.rst create mode 100644 source/object-storage/cli/swift-client/container-show.rst create mode 100644 source/object-storage/cli/swift-client/object-create.rst create mode 100644 source/object-storage/cli/swift-client/object-delete.rst create mode 100644 source/object-storage/cli/swift-client/object-list.rst create mode 100644 source/object-storage/cli/swift-client/object-save.rst create mode 100644 source/object-storage/cli/swift-client/object-show.rst create mode 100644 source/object-storage/cli/swift-client/prerequisites.rst delete mode 100644 source/object-storage/tutorial-scripts/curl.rst create mode 100644 source/object-storage/tutorial-scripts/endpoint-urls.rst delete mode 100644 source/object-storage/tutorial-scripts/openstackCLI.rst diff --git a/source/identity-access-management/roles.rst b/source/identity-access-management/roles.rst index 1fbc5cd0..5c9c8e6a 100644 --- a/source/identity-access-management/roles.rst +++ b/source/identity-access-management/roles.rst @@ -1,7 +1,7 @@ .. _access_control: ############## -Access control +Access Control ############## .. _project_users: diff --git a/source/object-storage/cli/aws-cli/container-create.rst b/source/object-storage/cli/aws-cli/container-create.rst new file mode 100644 index 00000000..3837ce44 --- /dev/null +++ b/source/object-storage/cli/aws-cli/container-create.rst @@ -0,0 +1,23 @@ +.. note:: + + It is **not recommended** to create an Object Storage container + using the AWS CLI, as the :ref:`storage policy ` + for the container cannot be set. + + Containers created using the AWS CLI will always use the default storage policy + (``nz--o1--mr-r3``). + +The preferred method to create a container using the AWS CLI +is to use the ``aws s3api create-bucket`` command, +specifying the container name with ``--bucket``. + +.. code-block:: bash + + aws s3api create-bucket --bucket mycontainer-1 + +``aws s3 mb`` can also be used, +specifying the container as a bucket name using an ``s3://`` URL. + +.. code-block:: bash + + aws s3 mb s3://mycontainer-1 diff --git a/source/object-storage/cli/aws-cli/container-delete.rst b/source/object-storage/cli/aws-cli/container-delete.rst new file mode 100644 index 00000000..20b2ffa1 --- /dev/null +++ b/source/object-storage/cli/aws-cli/container-delete.rst @@ -0,0 +1,23 @@ +The preferred method is to use the ``aws s3api delete-bucket`` command, +specifying the container name with ``--bucket``. +This command can only be used to delete empty containers +(all objects must be deleted before the container itself can be deleted), +so is a safer option to ensure data is not accidentally deleted. + +.. code-block:: bash + + aws s3api delete-bucket --bucket mycontainer-1 + +An alternative is to use the ``aws s3 rb`` command, +specifying the container as a bucket name using an ``s3://`` URL. + +.. code-block:: bash + + aws s3 rb s3://mycontainer-1 + +Unlike ``aws s3api delete-bucket``, this command can be used to delete non-empty containers +using the ``--force`` option. + +.. code-block:: bash + + aws s3 rb s3://mycontainer-1 --force diff --git a/source/object-storage/cli/aws-cli/container-list.rst b/source/object-storage/cli/aws-cli/container-list.rst new file mode 100644 index 00000000..90db1239 --- /dev/null +++ b/source/object-storage/cli/aws-cli/container-list.rst @@ -0,0 +1,24 @@ +Use either the ``aws s3 ls`` or the ``aws s3api list-buckets`` commands to return a list of all containers. + +.. code-block:: console + + $ aws s3 ls + 2009-02-04 05:45:09 mycontainer-1 + 2009-02-04 05:45:09 mycontainer-2 + $ aws s3api list-buckets + { + "Buckets": [ + { + "Name": "mycontainer-1", + "CreationDate": "2009-02-03T16:45:09+00:00" + }, + { + "Name": "mycontainer-2", + "CreationDate": "2009-02-03T16:45:09+00:00" + } + ], + "Owner": { + "DisplayName": "example.com:test@example.com", + "ID": "example.com:test@example.com" + } + } diff --git a/source/object-storage/cli/aws-cli/container-show.rst b/source/object-storage/cli/aws-cli/container-show.rst new file mode 100644 index 00000000..c2a37c48 --- /dev/null +++ b/source/object-storage/cli/aws-cli/container-show.rst @@ -0,0 +1,3 @@ +Object Storage container details are not available using the AWS CLI. + +To get container details on the command line, use the Catalyst Cloud Client. diff --git a/source/object-storage/cli/aws-cli/object-create.rst b/source/object-storage/cli/aws-cli/object-create.rst new file mode 100644 index 00000000..17b5e979 --- /dev/null +++ b/source/object-storage/cli/aws-cli/object-create.rst @@ -0,0 +1,20 @@ +The ``aws s3 cp`` command can be used to upload an object to a container, +by specifying the local source path as the first argument +and the destination ``s3://`` URL for the object as the second argument. + +.. code-block:: console + + $ aws s3 cp file-1.txt s3://mycontainer-1/file-1.txt + upload: ./file-1.txt to s3://mycontainer-1/file-1.txt + +Alternatively ``aws s3api put-object`` can be used as shown below, +using ``--body`` to specify the source file path, +``--bucket`` to specify the container name +and ``--key`` to specify the unique object name. + +.. code-block:: console + + $ aws s3api put-object --body file-1.txt --bucket mycontainer-1 --key file-1.txt + { + "ETag": "\"746308829575e17c3331bbcb00c0898b\"" + } diff --git a/source/object-storage/cli/aws-cli/object-delete.rst b/source/object-storage/cli/aws-cli/object-delete.rst new file mode 100644 index 00000000..73796208 --- /dev/null +++ b/source/object-storage/cli/aws-cli/object-delete.rst @@ -0,0 +1,15 @@ +The ``aws s3 rm`` command can be used to delete an object from a container, +specifying the ``s3://`` URL for the object. + +.. code-block:: console + + $ aws s3 rm s3://mycontainer-1/file-1.txt + delete: s3://mycontainer-1/file-1.txt + +Alternatively ``aws s3api delete-object`` can be used as shown below, +using ``--bucket`` to specify the container name +and ``--key`` to specify the unique object name. + +.. code-block:: bash + + aws s3api delete-object --bucket mycontainer-1 --key file-1.txt diff --git a/source/object-storage/cli/aws-cli/object-list.rst b/source/object-storage/cli/aws-cli/object-list.rst new file mode 100644 index 00000000..84312be2 --- /dev/null +++ b/source/object-storage/cli/aws-cli/object-list.rst @@ -0,0 +1,57 @@ +A list of objects can be returned using the ``aws s3 ls`` command, +specifying the container as a bucket name using an ``s3://`` URL. + +.. code-block:: console + + $ aws s3 ls s3://mycontainer-1 + PRE images/ + 2024-08-16 14:22:25 14 file-1.txt + 2023-08-25 11:31:27 1908226 image-1.png + +Just like on AWS, objects with directory-style prefixes (e.g. ``images/image-2.png``) +are not listed by default. Use the ``--recursive`` option to recursively list all objects +in the container. + +.. code-block:: console + + $ aws s3 ls s3://mycontainer-1 + 2024-08-16 14:22:25 14 file-1.txt + 2023-08-25 11:31:27 1908226 image-1.png + 2024-08-16 15:55:09 723424 images/image-2.png + +For more information on how to use the ``aws s3 ls`` command, see the `AWS CLI documentation `_. + +To get the list of objects in a format suitable for ingestion +by a script, use the ``aws s3api list-objects-v2`` command. + +.. code-block:: console + + $ aws s3api list-objects-v2 --bucket mycontainer-1 + { + "Contents": [ + { + "Key": "file-1.txt", + "LastModified": "2024-08-16T02:22:25.304000+00:00", + "ETag": "\"746308829575e17c3331bbcb00c0898b\"", + "Size": 14, + "StorageClass": "STANDARD" + }, + { + "Key": "image-1.png", + "LastModified": "2024-08-16T02:22:28.010000+00:00", + "ETag": "\"90cef9cada92ce2cf05cbde9499afbdb\"", + "Size": 1908226, + "StorageClass": "STANDARD" + }, + { + "Key": "images/image-2.png", + "LastModified": "2024-08-16T03:55:09.486000+00:00", + "ETag": "\"746308829575e17c3331bbcb00c0898b\"", + "Size": 723424, + "StorageClass": "STANDARD" + } + ], + "RequestCharged": null + } + +.. _`aws s3 ls`: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/ls.html diff --git a/source/object-storage/cli/aws-cli/object-save.rst b/source/object-storage/cli/aws-cli/object-save.rst new file mode 100644 index 00000000..a47e63d8 --- /dev/null +++ b/source/object-storage/cli/aws-cli/object-save.rst @@ -0,0 +1,30 @@ +The ``aws s3 cp`` command can be used to download an object from a container, +by specifying the source ``s3://`` URL for the object as the first argument +and the local destination path as the second argument. + +.. code-block:: console + + $ aws s3 cp s3://mycontainer-1/file-1.txt file-1.txt + download: s3://mycontainer-1/file-1.txt to ./file-1.txt + +The file will be saved to the target path. + +.. code-block:: console + + $ cat file-1.txt + Hello, world! + +Alternatively ``aws s3api get-object`` can be used as shown below, +using ``--bucket`` to specify the container name +and ``--key`` to specify the unique object name. + +.. code-block:: console + + $ aws s3api get-object --bucket mycontainer-1 --key file-1.txt file-1.txt + { + "LastModified": "2024-08-16T04:28:01+00:00", + "ContentLength": 14, + "ETag": "\"746308829575e17c3331bbcb00c0898b\"", + "ContentType": "binary/octet-stream", + "Metadata": {} + } diff --git a/source/object-storage/cli/aws-cli/object-show.rst b/source/object-storage/cli/aws-cli/object-show.rst new file mode 100644 index 00000000..b4c02505 --- /dev/null +++ b/source/object-storage/cli/aws-cli/object-show.rst @@ -0,0 +1,16 @@ +Some of the object metadata is available from the S3 CLI. + +To get this metadata run the ``aws s3api head-object`` command, +using ``--bucket`` to specify the container name +and ``--key`` to specify the unique object name. + +.. code-block:: console + + $ aws s3api head-object --bucket mycontainer-1 --key file-1.txt + { + "LastModified": "2024-08-16T02:22:26+00:00", + "ContentLength": 14, + "ETag": "\"746308829575e17c3331bbcb00c0898b\"", + "ContentType": "text/plain", + "Metadata": {} + } diff --git a/source/object-storage/cli/aws-cli/prerequisites.rst b/source/object-storage/cli/aws-cli/prerequisites.rst new file mode 100644 index 00000000..5b580673 --- /dev/null +++ b/source/object-storage/cli/aws-cli/prerequisites.rst @@ -0,0 +1,76 @@ +Once you have downloaded and installed the AWS CLI +(see `AWS CLI - Getting Started `_ for more information), +you will need to use the Catalyst Cloud Client to create the credentials for the AWS CLI. + +First, run ``openstack ec2 credentials create`` to create an EC2 credential. + +.. code-block:: console + + $ openstack ec2 credentials create + +-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ + | Field | Value | + +-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ + | access | ee55dd44cc33bb2211aaee55dd44cc33 | + | access_token_id | None | + | app_cred_id | None | + | links | {'self': 'https://api.nz-por-1.catalystcloud.io:5000/v3/users/1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a/credentials/OS-EC2/e5d4c3b2a1e5d4c3b2a1e5d4c3b2a1e5'} | + | project_id | 1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a | + | secret | 11aa22bb33cc44dd55ee11aa22bb33cc | + | trust_id | None | + | user_id | e5d4c3b2a1e5d4c3b2a1e5d4c3b2a1e5 | + +-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ + +If you have an existing EC2 credential you'd like to use instead, +run ``openstack ec2 credentials list`` to fetch the access key ID and secret access key for the credendial. + +.. code-block:: console + + $ openstack ec2 credentials list + +----------------------------------+----------------------------------+----------------------------------+----------------------------------+ + | Access | Secret | Project ID | User ID | + +----------------------------------+----------------------------------+----------------------------------+----------------------------------+ + | ee55dd44cc33bb2211aaee55dd44cc33 | 11aa22bb33cc44dd55ee11aa22bb33cc | 1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a | e5d4c3b2a1e5d4c3b2a1e5d4c3b2a1e5 | + +----------------------------------+----------------------------------+----------------------------------+----------------------------------+ + +Take the values for the ``access`` and ``secret`` fields, and use them to define the following environment variables +in your terminal session. + +.. tabs:: + + .. group-tab:: Linux / macOS + + .. code-block:: bash + + export AWS_ACCESS_KEY_ID=${access} + export AWS_SECRET_ACCESS_KEY=${secret} + + .. group-tab:: Windows + + .. code-block:: powershell + + $Env:AWS_ACCESS_KEY_ID = "${access}" + $Env:AWS_SECRET_ACCESS_KEY = "${secret}" + +In addition, you will need to set the ``AWS_ENDPOINT_URL`` environment variable to tell the AWS CLI +the location of the Catalyst Cloud Object Storage S3 API. + +.. include:: tutorial-scripts/endpoint-urls.rst + +Once you have determined the correct endpoint URL to use, set the ``AWS_ENDPOINT_URL`` +environment variable in your terminal session using the following command. + +.. tabs:: + + .. group-tab:: Linux / macOS + + .. code-block:: bash + + export AWS_ENDPOINT_URL=${endpoint_url} + + .. group-tab:: Windows + + .. code-block:: powershell + + $Env:AWS_ENDPOINT_URL = "${endpoint_url}" + +At this point, you should now be able to use the AWS CLI to interact with Catalyst Cloud Object Storage. diff --git a/source/object-storage/cli/catalystcloud-client/container-create.rst b/source/object-storage/cli/catalystcloud-client/container-create.rst new file mode 100644 index 00000000..bfb27ded --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/container-create.rst @@ -0,0 +1,30 @@ +To create a new Object Storage container, run ``openstack container create`` +followed by the unique name for the new container. + +.. code-block:: console + + $ openstack container create mycontainer-1 + +---------------------------------------+---------------+------------------------------------+ + | account | container | x-trans-id | + +---------------------------------------+---------------+------------------------------------+ + | AUTH_1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a | mycontainer-1 | tx415f9620d24e4ab28f01f-0066beb32f | + +---------------------------------------+---------------+------------------------------------+ + +If you'd like to create the container using a specific +:ref:`storage policy `, use the ``--storage-policy`` option +followed by the storage policy name. + +.. code-block:: console + + $ openstack container create mycontainer-2 --storage-policy nz-hlz-1--o1--sr-r3 + +---------------------------------------+---------------+------------------------------------+ + | account | container | x-trans-id | + +---------------------------------------+---------------+------------------------------------+ + | AUTH_1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a | mycontainer-2 | tx580e7f6582674260b9261-0066beb4a6 | + +---------------------------------------+---------------+------------------------------------+ + +Other useful options: + +* ``--public`` - Make the container publicly accessible. + **Be careful when using this option, as no authentication + will be required to access the container's objects.** diff --git a/source/object-storage/cli/catalystcloud-client/container-delete.rst b/source/object-storage/cli/catalystcloud-client/container-delete.rst new file mode 100644 index 00000000..71232aca --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/container-delete.rst @@ -0,0 +1,14 @@ +To delete one or more containers, use the ``openstack container delete`` command, +followed by the names of the containers to delete. + +.. code-block:: bash + + openstack container delete mycontainer-1 + +By default, only containers that contain no objects can be deleted. +To delete the containers **and all of the objects contained within them**, +use the ``--recursive`` option. + +.. code-block:: bash + + openstack container delete mycontainer-1 --recursive diff --git a/source/object-storage/cli/catalystcloud-client/container-list.rst b/source/object-storage/cli/catalystcloud-client/container-list.rst new file mode 100644 index 00000000..a113c90e --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/container-list.rst @@ -0,0 +1,7 @@ +Use the ``openstack container list`` command to return a list of all containers. + +.. code-block:: console + + $ openstack container list + mycontainer-1 + mycontainer-2 diff --git a/source/object-storage/cli/catalystcloud-client/container-show.rst b/source/object-storage/cli/catalystcloud-client/container-show.rst new file mode 100644 index 00000000..2d049d36 --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/container-show.rst @@ -0,0 +1,15 @@ +To view the details for a specific container, +use ``openstack container show`` followed by the container name. + +.. code-block:: console + + $ openstack container show mycontainer-1 + +----------------+---------------------------------------+ + | Field | Value | + +----------------+---------------------------------------+ + | account | AUTH_1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a | + | bytes_used | 1908226 | + | container | mycontainer-1 | + | object_count | 1 | + | storage_policy | nz--o1--mr-r3 | + +----------------+---------------------------------------+ diff --git a/source/object-storage/cli/catalystcloud-client/object-create.rst b/source/object-storage/cli/catalystcloud-client/object-create.rst new file mode 100644 index 00000000..5650f03b --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/object-create.rst @@ -0,0 +1,36 @@ +To upload a new object to a container, run ``openstack object create``, +followed by the container name and the path of the file to upload. + +.. code-block:: console + + $ openstack object create mycontainer-1 file-1.txt + +------------+---------------+----------------------------------+ + | object | container | etag | + +------------+---------------+----------------------------------+ + | file-1.txt | mycontainer-1 | 746308829575e17c3331bbcb00c0898b | + +------------+---------------+----------------------------------+ + +Use the ``--name`` option to upload the object to the container +using a different name. + +.. code-block:: console + + $ openstack object create mycontainer-1 file-1.txt --name custom-name.txt + +-----------------+---------------+----------------------------------+ + | object | container | etag | + +-----------------+---------------+----------------------------------+ + | custom-name.txt | mycontainer-1 | 746308829575e17c3331bbcb00c0898b | + +-----------------+---------------+----------------------------------+ + +You can also upload multiple files at once in a single command. +Note that ``--name`` is not usable when uploading multiple files. + +.. code-block:: console + + $ openstack object create mycontainer-1 file-1.txt file-2.txt + +------------+---------------+----------------------------------+ + | object | container | etag | + +------------+---------------+----------------------------------+ + | file-1.txt | mycontainer-1 | 746308829575e17c3331bbcb00c0898b | + | file-2.txt | mycontainer-1 | 90cef9cada92ce2cf05cbde9499afbdb | + +------------+---------------+----------------------------------+ diff --git a/source/object-storage/cli/catalystcloud-client/object-delete.rst b/source/object-storage/cli/catalystcloud-client/object-delete.rst new file mode 100644 index 00000000..e0bf1dcb --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/object-delete.rst @@ -0,0 +1,7 @@ +To delete one or more objects from a container, +use ``openstack object delete``, followed by the name of the container +and the unique names of the objects to delete. + +.. code-block:: bash + + openstack object delete mycontainer-1 file-1.txt diff --git a/source/object-storage/cli/catalystcloud-client/object-list.rst b/source/object-storage/cli/catalystcloud-client/object-list.rst new file mode 100644 index 00000000..a60196da --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/object-list.rst @@ -0,0 +1,12 @@ +To view the objects stored within a container, use +``openstack object list`` followed by the container name. + +.. code-block:: console + + $ openstack object list mycontainer-1 + +-------------+ + | Name | + +-------------+ + | file-1.txt | + | image-1.png | + +-------------+ diff --git a/source/object-storage/cli/catalystcloud-client/object-save.rst b/source/object-storage/cli/catalystcloud-client/object-save.rst new file mode 100644 index 00000000..6dc325d7 --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/object-save.rst @@ -0,0 +1,20 @@ +To download a single object from a container, use the ``openstack object save`` command, +followed by the container name and the unique object name. + +.. code-block:: bash + + openstack object save mycontainer-1 file-1.txt + +The file will be saved to the local directory of the terminal session. + +.. code-block:: console + + $ cat file-1.txt + Hello, world! + +If you'd like to save the file to a specific location +(or under a different name), use the ``--file`` option to specify the destination. + +.. code-block:: bash + + openstack object save mycontainer-1 file-1.txt --file /home/example/Downloads/file-1.txt diff --git a/source/object-storage/cli/catalystcloud-client/object-show.rst b/source/object-storage/cli/catalystcloud-client/object-show.rst new file mode 100644 index 00000000..0b10f552 --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/object-show.rst @@ -0,0 +1,18 @@ +To get the details of a specific object in a container, +run ``openstack object show``, specifying the container name +and the unique object name. + +.. code-block:: console + + $ openstack object show mycontainer-1 file-1.txt + +----------------+---------------------------------------+ + | Field | Value | + +----------------+---------------------------------------+ + | account | AUTH_1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a | + | container | mycontainer-1 | + | content-length | 14 | + | content-type | text/plain | + | etag | 746308829575e17c3331bbcb00c0898b | + | last-modified | Fri, 16 Aug 2024 01:58:10 GMT | + | object | file-1.txt | + +----------------+---------------------------------------+ diff --git a/source/object-storage/cli/catalystcloud-client/prerequisites.rst b/source/object-storage/cli/catalystcloud-client/prerequisites.rst new file mode 100644 index 00000000..f206ed63 --- /dev/null +++ b/source/object-storage/cli/catalystcloud-client/prerequisites.rst @@ -0,0 +1,27 @@ +Check that the ``openstack container`` and ``openstack object`` series of commands available by running the help command. + +.. code-block:: console + + $ openstack container --help + Command "container" matches: + container create + container delete + container list + container save + container set + container show + container unset + $ openstack object --help + Command "object" matches: + object create + object delete + object list + object save + object set + object show + object store account set + object store account show + object store account unset + object unset + +If you have sourced your OpenRC file, you should now be able to run these commands. diff --git a/source/object-storage/cli/curl/container-create.rst b/source/object-storage/cli/curl/container-create.rst new file mode 100644 index 00000000..72abc01a --- /dev/null +++ b/source/object-storage/cli/curl/container-create.rst @@ -0,0 +1,19 @@ +To create a new Object Storage container, run the following command, +substituting ``mycontainer-1`` for the unique name of the container: + +.. code-block:: bash + + curl -i -X PUT -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1 + +The API should return a ``201 Created`` response. + +.. code-block:: console + + $ curl -i -X PUT -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1 + HTTP/1.1 201 Created + server: nginx/1.14.0 (Ubuntu) + date: Fri, 16 Aug 2024 07:53:19 GMT + content-type: text/html; charset=UTF-8 + content-length: 0 + x-trans-id: tx200ef08930e04367b5103-0066bf056f + x-openstack-request-id: tx200ef08930e04367b5103-0066bf056f diff --git a/source/object-storage/cli/curl/container-delete.rst b/source/object-storage/cli/curl/container-delete.rst new file mode 100644 index 00000000..1332890d --- /dev/null +++ b/source/object-storage/cli/curl/container-delete.rst @@ -0,0 +1,23 @@ +To delete a container, run the following command, +substituting ``mycontainer-1`` for the unique name of the container: + +.. code-block:: bash + + curl -i -X DELETE -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1 + +The API should return a ``204 No Content`` response. + +.. code-block:: console + + $ curl -i -X DELETE -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1 + HTTP/1.1 204 No Content + server: nginx/1.14.0 (Ubuntu) + date: Fri, 16 Aug 2024 08:09:00 GMT + content-type: text/html; charset=UTF-8 + x-trans-id: tx813318cbeb404fafbacec-0066bf091c + x-openstack-request-id: tx813318cbeb404fafbacec-0066bf091c + +.. note:: + + Using this command you can only delete **empty** containers. + If the container has objects in it, delete the objects first. diff --git a/source/object-storage/cli/curl/container-list.rst b/source/object-storage/cli/curl/container-list.rst new file mode 100644 index 00000000..4d88da2d --- /dev/null +++ b/source/object-storage/cli/curl/container-list.rst @@ -0,0 +1,47 @@ +To list all existing containers on your project, run the following command: + +.. code-block:: bash + + curl -i -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL + +The output will be as shown below. A number of related headers are also shown with useful information. + +.. code-block:: console + + $ curl -i -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL + HTTP/1.1 200 OK + server: nginx/1.14.0 (Ubuntu) + date: Fri, 16 Aug 2024 07:56:23 GMT + content-type: text/plain; charset=utf-8 + content-length: 86 + x-account-container-count: 2 + x-account-object-count: 4 + x-account-bytes-used: 1908270 + x-timestamp: 1692919874.18591 + x-account-storage-policy-nz--o1--mr-r3-container-count: 2 + x-account-storage-policy-nz--o1--mr-r3-object-count: 4 + x-account-storage-policy-nz--o1--mr-r3-bytes-used: 1908270 + x-account-storage-policy-nz-hlz-1--o1--sr-r3-container-count: 0 + x-account-storage-policy-nz-hlz-1--o1--sr-r3-object-count: 0 + x-account-storage-policy-nz-hlz-1--o1--sr-r3-bytes-used: 0 + x-account-storage-policy-nz-por-1--o1--sr-r3-container-count: 0 + x-account-storage-policy-nz-por-1--o1--sr-r3-object-count: 0 + x-account-storage-policy-nz-por-1--o1--sr-r3-bytes-used: 0 + x-account-storage-policy-nz-wlg-2--o1--sr-r3-container-count: 0 + x-account-storage-policy-nz-wlg-2--o1--sr-r3-object-count: 0 + x-account-storage-policy-nz-wlg-2--o1--sr-r3-bytes-used: 0 + accept-ranges: bytes + x-account-project-domain-id: default + vary: Accept + x-trans-id: tx3416961483f74f3b836d6-0066bf0627 + x-openstack-request-id: tx3416961483f74f3b836d6-0066bf0627 + + mycontainer-1 + mycontainer-2 + +Responses can also be formatted in JSON or XML. + +.. code-block:: bash + + curl -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL?format=json + curl -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL?format=xml diff --git a/source/object-storage/cli/curl/container-show.rst b/source/object-storage/cli/curl/container-show.rst new file mode 100644 index 00000000..511e340c --- /dev/null +++ b/source/object-storage/cli/curl/container-show.rst @@ -0,0 +1,26 @@ +To view the details for a specific container, run the following command, +substituting ``mycontainer-1`` for the container name: + +.. code-block:: bash + + curl -i --head -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1 + +The headers containing the container metadata will be returned. + +.. code-block:: console + + $ curl -i --head -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1 + HTTP/1.1 200 OK + server: nginx/1.14.0 (Ubuntu) + date: Fri, 16 Aug 2024 08:02:19 GMT + content-type: text/plain; charset=utf-8 + content-length: 22 + x-container-object-count: 2 + x-container-bytes-used: 28 + x-timestamp: 1723782306.51716 + last-modified: Fri, 16 Aug 2024 07:55:28 GMT + accept-ranges: bytes + x-storage-policy: nz--o1--mr-r3 + vary: Accept + x-trans-id: tx3ca25d3285344a0188612-0066bf078a + x-openstack-request-id: tx3ca25d3285344a0188612-0066bf078a diff --git a/source/object-storage/cli/curl/object-create.rst b/source/object-storage/cli/curl/object-create.rst new file mode 100644 index 00000000..674eb2d8 --- /dev/null +++ b/source/object-storage/cli/curl/object-create.rst @@ -0,0 +1,28 @@ +To upload a new object to a container, run ``openstack object create``, +run the following command. + +In the URL, substitute ``mycontainer-1`` for the target container, +and ``file-1.txt`` for the unique name for the object to create. +The value of ``--upload-file`` should be the path of the file to upload. + +.. code-block:: bash + + curl -i -X PUT -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1/file-1.txt --upload-file ./file-1.txt + +If the upload is successful, you will receive a ``201 Created`` response, +along with an ``etag`` header containing the generated ETag for the object. + +.. code-block:: console + + $ curl -i -X PUT -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1/file-1.txt --upload-file ./file-1.txt + HTTP/1.1 100 Continue + + HTTP/1.1 201 Created + server: nginx/1.14.0 (Ubuntu) + date: Fri, 16 Aug 2024 12:06:36 GMT + content-type: text/html; charset=UTF-8 + content-length: 0 + etag: 746308829575e17c3331bbcb00c0898b + last-modified: Fri, 16 Aug 2024 12:06:37 GMT + x-trans-id: tx9a26f494bfc148a8a5ab6-0066bf40cb + x-openstack-request-id: tx9a26f494bfc148a8a5ab6-0066bf40cb diff --git a/source/object-storage/cli/curl/object-delete.rst b/source/object-storage/cli/curl/object-delete.rst new file mode 100644 index 00000000..53b0a998 --- /dev/null +++ b/source/object-storage/cli/curl/object-delete.rst @@ -0,0 +1,19 @@ +To delete an object from a container, run the following command, +substituting ``mycontainer-1`` for the container name +and ``file-1.txt`` for the unique object name: + +.. code-block:: bash + + curl -i -X DELETE -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1/file-1.txt + +The API should return a ``204 No Content`` response. + +.. code-block:: console + + $ curl -i -X DELETE -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1/file-1.txt + HTTP/1.1 204 No Content + server: nginx/1.14.0 (Ubuntu) + date: Fri, 16 Aug 2024 12:19:49 GMT + content-type: text/html; charset=UTF-8 + x-trans-id: tx89613718a38a4a2db45b4-0066bf43e5 + x-openstack-request-id: tx89613718a38a4a2db45b4-0066bf43e5 diff --git a/source/object-storage/cli/curl/object-list.rst b/source/object-storage/cli/curl/object-list.rst new file mode 100644 index 00000000..303be9a7 --- /dev/null +++ b/source/object-storage/cli/curl/object-list.rst @@ -0,0 +1,38 @@ +To view the objects stored within a container, run the following command, +substituting ``mycontainer-1`` for the container name: + +.. code-block:: bash + + curl -i -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1 + +The body of the response will contain a list of the objects stored within the +container, one per line. +A number of related headers are also included with useful information. + +.. code-block:: console + + $ curl -i -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1 + HTTP/1.1 200 OK + server: nginx/1.14.0 (Ubuntu) + date: Fri, 16 Aug 2024 08:02:19 GMT + content-type: text/plain; charset=utf-8 + content-length: 22 + x-container-object-count: 2 + x-container-bytes-used: 28 + x-timestamp: 1723782306.51716 + last-modified: Fri, 16 Aug 2024 07:55:28 GMT + accept-ranges: bytes + x-storage-policy: nz--o1--mr-r3 + vary: Accept + x-trans-id: tx3ca25d3285344a0188612-0066bf078a + x-openstack-request-id: tx3ca25d3285344a0188612-0066bf078a + + file-1.txt + image-1.png + +Responses can also be formatted in JSON or XML. + +.. code-block:: bash + + curl -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1?format=json + curl -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1?format=xml diff --git a/source/object-storage/cli/curl/object-save.rst b/source/object-storage/cli/curl/object-save.rst new file mode 100644 index 00000000..1042ebba --- /dev/null +++ b/source/object-storage/cli/curl/object-save.rst @@ -0,0 +1,44 @@ +To return the contents of an object, run the following command, +substituting ``mycontainer-1`` for the container name +and ``file-1.txt`` for the unique object name. + +.. code-block:: bash + + curl -i -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1/file-1.txt + +The contents of the file, and related headers, will be output to the console. + +.. code-block:: console + + $ curl -i -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1/file-1.txt + HTTP/1.1 200 OK + server: nginx/1.14.0 (Ubuntu) + date: Fri, 16 Aug 2024 11:54:40 GMT + content-type: text/plain + content-length: 14 + x-object-meta-mtime: 1723794919.810479 + etag: 746308829575e17c3331bbcb00c0898b + last-modified: Fri, 16 Aug 2024 11:54:35 GMT + x-timestamp: 1723809274.03068 + accept-ranges: bytes + x-trans-id: txa3cd4a2343c644d994397-0066bf3e00 + x-openstack-request-id: txa3cd4a2343c644d994397-0066bf3e00 + + Hello, world! + +If you'd like to save the object to a file use the following command instead, +setting the value of ``--output`` to the destination file to save the object to. + +.. code-block:: console + + $ curl -X GET -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1/file-1.txt --output file-1.txt + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 100 14 100 14 0 0 10 0 0:00:01 0:00:01 --:--:-- 10 + +The object will now be saved to the specified file. + +.. code-block:: console + + $ cat file-1.txt + Hello, world! diff --git a/source/object-storage/cli/curl/object-show.rst b/source/object-storage/cli/curl/object-show.rst new file mode 100644 index 00000000..a7048d69 --- /dev/null +++ b/source/object-storage/cli/curl/object-show.rst @@ -0,0 +1,28 @@ +To get the details of a specific object in a container, +run the following command, +substituting ``mycontainer-1`` for the container name +and ``file-1.txt`` for the unique object name: + +.. code-block:: bash + + curl -i --head -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1/file-1.txt + +The headers containing the container metadata will be returned. + +.. code-block:: console + + $ curl -i --head -H "X-Auth-Token: $OS_TOKEN" $OS_STORAGE_URL/mycontainer-1/file-1.txt + HTTP/1.1 200 OK + server: nginx/1.14.0 (Ubuntu) + date: Fri, 16 Aug 2024 08:02:19 GMT + content-type: text/plain; charset=utf-8 + content-length: 22 + x-container-object-count: 2 + x-container-bytes-used: 28 + x-timestamp: 1723782306.51716 + last-modified: Fri, 16 Aug 2024 07:55:28 GMT + accept-ranges: bytes + x-storage-policy: nz--o1--mr-r3 + vary: Accept + x-trans-id: tx3ca25d3285344a0188612-0066bf078a + x-openstack-request-id: tx3ca25d3285344a0188612-0066bf078a diff --git a/source/object-storage/cli/curl/prerequisites.rst b/source/object-storage/cli/curl/prerequisites.rst new file mode 100644 index 00000000..efb1805d --- /dev/null +++ b/source/object-storage/cli/curl/prerequisites.rst @@ -0,0 +1,23 @@ +The easiest way to interact with Catalyst Cloud using cURL is to use +Swift API. + +This uses two environment variables in your terminal session: +the **storage URL** (``OS_STORAGE_URL``), which we will configure now, +and the **auth token** (``OS_TOKEN``), which is set when you source the +OpenRC file for your Catalyst Cloud project (which we have already done). + +To get API requests using cURL working we now need to set the ``OS_STORAGE_URL`` +environment variable, which stores the region-specific object storage endpoint URL +for your project. + +.. include:: tutorial-scripts/endpoint-urls.rst + +Once you have determined the correct endpoint URL, set the ``OS_STORAGE_URL`` +environment variable, substituting ``${endpoint_url}`` below with the endpoint URL: + +.. code-block:: bash + + export OS_STORAGE_URL="${endpoint_url}/v1/AUTH_${OS_PROJECT_ID}" + +At this point, you will now be able to use the instructions below to send +API commands using cURL. diff --git a/source/object-storage/cli/swift-client/container-create.rst b/source/object-storage/cli/swift-client/container-create.rst new file mode 100644 index 00000000..9977c141 --- /dev/null +++ b/source/object-storage/cli/swift-client/container-create.rst @@ -0,0 +1,14 @@ +To create a new Object Storage container, run ``swift post`` +followed by the unique name for the new container. + +.. code-block:: bash + + swift post mycontainer-1 + +If you'd like to create the container using a specific +:ref:`storage policy `, +use the ``--header`` option to specify the ``X-Storage-Policy`` header as shown below. + +.. code-block:: bash + + swift post mycontainer-1 --header 'X-Storage-Policy: nz-hlz-1--o1--sr-r3' diff --git a/source/object-storage/cli/swift-client/container-delete.rst b/source/object-storage/cli/swift-client/container-delete.rst new file mode 100644 index 00000000..dc3b037d --- /dev/null +++ b/source/object-storage/cli/swift-client/container-delete.rst @@ -0,0 +1,19 @@ +To delete a container, use the ``swift delete`` command, +followed by the name of the container to delete. + +Any objects that exist in the container will also be deleted. + +.. code-block:: console + + $ swift delete mycontainer-1 + file-2.txt + file-1.txt + mycontainer-1 + +.. warning:: + + There is an ``--all`` option which can be used to delete + **all objects and containers** in your project. + This option is **NOT** required to delete a container. + **Do not use it unless you really intend to delete all containers + from a project.** diff --git a/source/object-storage/cli/swift-client/container-list.rst b/source/object-storage/cli/swift-client/container-list.rst new file mode 100644 index 00000000..41b56a68 --- /dev/null +++ b/source/object-storage/cli/swift-client/container-list.rst @@ -0,0 +1,7 @@ +Use the ``swift list`` command to return a list of all containers. + +.. code-block:: console + + $ swift list + mycontainer-1 + mycontainer-2 diff --git a/source/object-storage/cli/swift-client/container-show.rst b/source/object-storage/cli/swift-client/container-show.rst new file mode 100644 index 00000000..675f214c --- /dev/null +++ b/source/object-storage/cli/swift-client/container-show.rst @@ -0,0 +1,23 @@ +To view the details for a specific container, +use ``swift stat`` followed by the container name. + +.. code-block:: console + + $ swift stat mycontainer-1 + Account: AUTH_1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a + Container: mycontainer-1 + Objects: 2 + Bytes: 2453477 + Read ACL: + Write ACL: + Sync To: + Sync Key: + Server: nginx/1.14.0 (Ubuntu) + Content-Type: text/plain; charset=utf-8 + X-Timestamp: 1723782306.51716 + Last-Modified: Fri, 16 Aug 2024 04:25:07 GMT + Accept-Ranges: bytes + X-Storage-Policy: nz--o1--mr-r3 + Vary: Accept + X-Trans-Id: tx9abcd62799254073ad405-0066bef1b2 + X-Openstack-Request-Id: tx9abcd62799254073ad405-0066bef1b2 diff --git a/source/object-storage/cli/swift-client/object-create.rst b/source/object-storage/cli/swift-client/object-create.rst new file mode 100644 index 00000000..58b704d8 --- /dev/null +++ b/source/object-storage/cli/swift-client/object-create.rst @@ -0,0 +1,24 @@ +To upload a new object to a container, run ``swift upload``, +followed by the container name and the path of the file to upload. + +.. code-block:: console + + $ swift upload mycontainer-1 file-1.txt + file-1.txt + +Use the ``--object-name`` option to upload the object to the container +using a different name. + +.. code-block:: console + + $ swift upload mycontainer-1 file-1.txt --object-name custom-name.txt + custom-name.txt + +You can also upload multiple files at once in a single command. +Note that ``--object-name`` is not usable when uploading multiple files. + +.. code-block:: console + + $ swift upload mycontainer-1 file-1.txt file-2.txt + file-1.txt + file-2.txt diff --git a/source/object-storage/cli/swift-client/object-delete.rst b/source/object-storage/cli/swift-client/object-delete.rst new file mode 100644 index 00000000..2cb7f15f --- /dev/null +++ b/source/object-storage/cli/swift-client/object-delete.rst @@ -0,0 +1,8 @@ +To delete one or more objects from a container, +use ``swift delete``, followed by the name of the container +and the unique names of the objects to delete. + +.. code-block:: console + + $ swift delete mycontainer-1 file-1.txt + file-1.txt diff --git a/source/object-storage/cli/swift-client/object-list.rst b/source/object-storage/cli/swift-client/object-list.rst new file mode 100644 index 00000000..44067d5f --- /dev/null +++ b/source/object-storage/cli/swift-client/object-list.rst @@ -0,0 +1,8 @@ +To view the objects stored within a container, use +``swift list`` followed by the container name. + +.. code-block:: console + + $ swift list mycontainer-1 + file-1.txt + image-1.png diff --git a/source/object-storage/cli/swift-client/object-save.rst b/source/object-storage/cli/swift-client/object-save.rst new file mode 100644 index 00000000..991f8b0c --- /dev/null +++ b/source/object-storage/cli/swift-client/object-save.rst @@ -0,0 +1,39 @@ +To download a single object from a container, use the ``swift download`` command, +followed by the container name and the unique object name. + +.. code-block:: console + + $ swift download mycontainer-1 file-1.txt + file-1.txt [auth 0.000s, headers 0.844s, total 0.845s, 0.000 MB/s] + +The file will be saved to the local directory of the terminal session. + +.. code-block:: console + + $ cat file-1.txt + Hello, world! + +If you'd like to save the file to a specific location +(or under a different name), use the ``--output`` option to specify the destination. + +.. code-block:: bash + + $ swift download mycontainer-1 file-1.txt --output /home/example/Downloads/file-1.txt + file-1.txt [auth 0.000s, headers 0.844s, total 0.845s, 0.000 MB/s] + +Multiple objects can be downloaded at once by passing additional arguments. + +.. code-block:: console + + $ swift download mycontainer-1 file-1.txt file-2.txt + file-2.txt [auth 0.000s, headers 0.279s, total 0.279s, 0.000 MB/s] + file-1.txt [auth 0.000s, headers 0.401s, total 0.402s, 0.000 MB/s] + +When downloading multiple objects, use the ``--output-dir`` option to specify +the target directory to download files to: + +.. code-block:: console + + $ swift download mycontainer-1 file-1.txt file-2.txt --output-dir /home/example/Downloads + file-2.txt [auth 0.000s, headers 0.279s, total 0.279s, 0.000 MB/s] + file-1.txt [auth 0.000s, headers 0.401s, total 0.402s, 0.000 MB/s] diff --git a/source/object-storage/cli/swift-client/object-show.rst b/source/object-storage/cli/swift-client/object-show.rst new file mode 100644 index 00000000..c44a63fe --- /dev/null +++ b/source/object-storage/cli/swift-client/object-show.rst @@ -0,0 +1,19 @@ +To get the details of a specific object in a container, +run ``swift stat``, specifying the container name +and the unique object name. + +.. code-block:: console + + $ swift stat test-container-mr file-1.txt + Account: AUTH_1a2b3c4d5e1a2b3c4d5e1a2b3c4d5e1a + Container: test-container-mr + Object: file-1.txt + Content Type: text/plain + Content Length: 14 + Last Modified: Fri, 16 Aug 2024 02:22:26 GMT + ETag: 746308829575e17c3331bbcb00c0898b + Server: nginx/1.14.0 (Ubuntu) + X-Timestamp: 1723774945.30427 + Accept-Ranges: bytes + X-Trans-Id: txa8dcd12eb0cd4c7caa153-0066bef7cb + X-Openstack-Request-Id: txa8dcd12eb0cd4c7caa153-0066bef7cb diff --git a/source/object-storage/cli/swift-client/prerequisites.rst b/source/object-storage/cli/swift-client/prerequisites.rst new file mode 100644 index 00000000..3c3711ae --- /dev/null +++ b/source/object-storage/cli/swift-client/prerequisites.rst @@ -0,0 +1,28 @@ +Swift Client is automatically installed when installing the +Catalyst Cloud Client package. To use it, some minor configuration is required. + +In addition to sourcing the OpenRC for your Catalyst Cloud project in your terminal session, +the ``OS_STORAGE_URL`` environment variable needs to be set, which stores the region-specific +object storage endpoint URL for your project. + +.. include:: tutorial-scripts/endpoint-urls.rst + +Once you have determined the correct endpoint URL, set the ``OS_STORAGE_URL`` +environment variable, substituting ``${endpoint_url}`` below with the endpoint URL: + +.. tabs:: + + .. group-tab:: Linux / macOS + + .. code-block:: bash + + export OS_STORAGE_URL="${endpoint_url}/v1/AUTH_${OS_PROJECT_ID}" + + .. group-tab:: Windows + + .. code-block:: powershell + + $Env:OS_STORAGE_URL = "${endpoint_url}/v1/AUTH_${Env:OS_PROJECT_ID}" + +At this point, you should now be able to use Swift Client to interact with +Catalyst Cloud Object Storage. diff --git a/source/object-storage/tutorial-scripts/curl.rst b/source/object-storage/tutorial-scripts/curl.rst deleted file mode 100644 index 7b5b293a..00000000 --- a/source/object-storage/tutorial-scripts/curl.rst +++ /dev/null @@ -1,104 +0,0 @@ -To access object storage using cURL it is necessary to provide credentials -to authenticate any requests you make. - -This can be done by sourcing your OpenRC file and retrieving your account -specific details via the Swift command line tools; then exporting the required -variables as shown below. - - -.. code-block:: bash - - $ source openstack-openrc.sh - - # we then need to create a OS_STORAGE_URL environment variable. - # To create this variable we will need to find our project ID: - - $ openstack project show - +-------------+----------------------------------+ - | Field | Value | - +-------------+----------------------------------+ - | description | | - | domain_id | default | - | enabled | True | - | id | 1xxxxxxxxxxxxxxxxxxxxxxxxxxxxe54 | - | tags | [] | - +-------------+----------------------------------+ - -We then export this ID with the storage API for the region we are working in. -For this example we will use the Porirua region: - -.. code-block:: bash - - $ export OS_STORAGE_URL="https://object-storage.nz-por-1.catalystcloud.io:443/v1/AUTH_1xxxxxxxxxxxxxxxxxxxxxxxxxxxxe54 - - # Then we grab our auth token for later use: - - $ swift stat -v - StorageURL: https://object-storage.nz-por-1.catalystcloud.io:443/v1/AUTH_1xxxxxxxxxxxxxxxxxxxxxxxxxxxxe54 - Auth Token: 5f5a043e1bd24a8fa8xxxxxxcca8e0fc - Containers: 48 - Account: AUTH_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - Objects: 156 - Bytes: 11293750551 - Containers in policy "policy-0": 48 - Objects in policy "policy-0": 156 - Bytes in policy "policy-0": 11293750551 - X-Account-Project-Domain-Id: default - Server: nginx/1.8.1 - X-Timestamp: 1466047859.45584 - X-Trans-Id: tx4bdb5d859f8c47f18b44d-00578c0e63 - Content-Type: text/plain; charset=utf-8 - Accept-Ranges: bytes - - $ export token="5f5a043e1bd24a8fa8xxxxxxcca8e0fc" - -To create a new container, use the following cURL request: - -.. code-block:: bash - - curl -i -X PUT -H "X-Auth-Token: $token" $storageURL/mycontainer - -Then run the following command to get a list of all available containers for -your project: - -.. code-block:: bash - - curl -i -X GET -H "X-Auth-Token: $token" $storageURL - -You can optionally specify alternative output formats. For example: to have XML -or JSON returned use the following syntax: - -.. code-block:: bash - - curl -i -X GET -H "X-Auth-Token: $token" $storageURL?format=xml - curl -i -X GET -H "X-Auth-Token: $token" $storageURL?format=json - -To view the objects within a container, simply append the container name to -the cURL request: - -.. code-block:: bash - - curl -i -X GET -H "X-Auth-Token: $token" $storageURL/mycontainer - -To upload a file to your container, use the following cURL format: - -.. code-block:: bash - - curl -i -T -X PUT -H "X-Auth-Token: $token" $storageURL/mycontainer - -To delete a file from your container, use this code: - -.. code-block:: bash - - curl -X DELETE -H "X-Auth-Token: " /mycontainer/myobject - -Finally, to delete a container you can use the following syntax. - -.. Note:: - - A container must be empty before you try and delete it. Otherwise the - operation will fail. - -.. code-block:: bash - - curl -X DELETE -H "X-Auth-Token: " /mycontainer diff --git a/source/object-storage/tutorial-scripts/endpoint-urls.rst b/source/object-storage/tutorial-scripts/endpoint-urls.rst new file mode 100644 index 00000000..03921885 --- /dev/null +++ b/source/object-storage/tutorial-scripts/endpoint-urls.rst @@ -0,0 +1,32 @@ +Catalyst Cloud has unique Object Storage endpoint URLs for each region. + +.. list-table:: + :header-rows: 1 + + * - Region + - Endpoint URL + * - nz-por-1 + - ``https://object-storage.nz-por-1.catalystcloud.io:443`` + * - nz-hlz-1 + - ``https://object-storage.nz-hlz-1.catalystcloud.io:443`` + * - nz_wlg_2 + - ``https://object-storage.nz-wlg-2.catalystcloud.io:443`` + +The endpoint URL you will need to use depends on your use case. + +* If the container uses a + :ref:`single-region storage policy `, + use the endpoint URL for the region the container is located in. + + * For example, if the container is located in the ``nz-hlz-1`` region, + use the endpoint URL for the ``nz-hlz-1`` region. + +* If the container uses the multi-region replication policy + **AND** the workload (e.g. compute instance) accessing the container + is also hosted on Catalyst Cloud, set this to the same region in which + the workload is located. + + * For example, if the workload is hosted in the ``nz-hlz-1`` region, + use the endpoint URL for the ``nz-hlz-1`` region. + +* If none of the above apply, use the endpoint for the ``nz-por-1`` region. diff --git a/source/object-storage/tutorial-scripts/heat.rst b/source/object-storage/tutorial-scripts/heat.rst index afb6540a..12022900 100644 --- a/source/object-storage/tutorial-scripts/heat.rst +++ b/source/object-storage/tutorial-scripts/heat.rst @@ -1,18 +1,19 @@ -Using Heat you are able to manage a large number of resources programmatically, +Using the Catalyst Cloud :ref:`Orchestration ` service +you are able to manage a large number of resources programmatically, by utilizing a **stack** that will construct and monitor your resources for you. You can create a stack by using a pre-designed template. The -following example assumes that you have some knowledge of Heat and how to +following example assumes that you have some knowledge of Orchestration and how to use these template files. The following code snippet contains the minimum required components to -construct an object storage container using Heat: +construct an object storage container using Orchestration: .. code-block:: bash heat_template_version: 2015-04-30 description: > - Creating a swift container using HEAT + Creating a swift container using Orchestration resources: @@ -20,13 +21,13 @@ construct an object storage container using Heat: type: OS::Swift::Container properties: PurgeOnDelete: FALSE - name: heat-container + name: orchestration-container For more information on object storage containers and what customization options you can select for them, please see the -`Openstack Heat`_ documentation. +`OpenStack Heat`_ documentation. -.. _Openstack Heat: https://docs.openstack.org/heat/latest/template_guide/openstack.html#OS::Swift::Container +.. _OpenStack Heat: https://docs.openstack.org/heat/latest/template_guide/openstack.html#OS::Swift::Container Once you have your template constructed, you should make sure to validate it before creating any resources. You can do this by running the following code: diff --git a/source/object-storage/tutorial-scripts/openstackCLI.rst b/source/object-storage/tutorial-scripts/openstackCLI.rst deleted file mode 100644 index 8b505139..00000000 --- a/source/object-storage/tutorial-scripts/openstackCLI.rst +++ /dev/null @@ -1,86 +0,0 @@ -The following is a list of the most commonly used commands that will help you -interact with the object storage service via the openstack command line. - -| - -To view the current containers on your project, you can use the ``openstack -container list`` command: - -.. code-block:: bash - - $ openstack container list - mycontainer-1 - mycontainer-2 - -| - -To view the objects stored within a container: -``openstack object list `` - -.. code-block:: bash - - $ openstack object list mycontainer-1 - +-------------+ - | Name | - +-------------+ - | file-1.txt | - | image-1.png | - +-------------+ - -| - -To create a new container: ``openstack container create `` - -.. code-block:: bash - - $ openstack container create mynewcontainer - - +---------+----------------+----------------------------------------------------+ - | account | container | x-trans-id | - +---------+----------------+----------------------------------------------------+ - | v1 | mynewcontainer | tx000000000000000146531-0057bb8fc9-2836950-default | - +---------+----------------+----------------------------------------------------+ - -| - -To add a new object to a container: -``openstack object create `` - -.. code-block:: bash - - $ openstack object create mynewcontainer hello.txt - - +-----------+----------------+----------------------------------+ - | object | container | etag | - +-----------+----------------+----------------------------------+ - | hello.txt | mynewcontainer | d41d8cd98f00b204xxxxxx98ecf8427e | - +-----------+----------------+----------------------------------+ - -| - -To delete an object from a container: ``openstack object delete `` - -.. code-block:: bash - - $ openstack object delete mynewcontainer hello.txt - -| - -To delete a container: ``openstack container delete `` - -.. note:: - - this will only work if the container does not contain any objects. - -.. code-block:: bash - - $ openstack container delete mycontainer-1 - -| - -To delete a container and all of the objects within the container: -``openstack container delete --recursive `` - -.. code-block:: bash - - $ openstack container delete --recursive mycontainer-1 diff --git a/source/object-storage/tutorial-scripts/swiftAPI.rst b/source/object-storage/tutorial-scripts/swiftAPI.rst index 6b332dfb..acd731c1 100644 --- a/source/object-storage/tutorial-scripts/swiftAPI.rst +++ b/source/object-storage/tutorial-scripts/swiftAPI.rst @@ -8,27 +8,36 @@ official `OpenStack documentation

API endpoints

-+----------+---------+--------------------------------------------------------------------------+ -| Region | Version | Endpoint | -+==========+=========+==========================================================================+ -| nz-por-1 | 1 | https://object-storage.nz-por-1.catalystcloud.io:443/v1/AUTH_%tenantid% | -+----------+---------+--------------------------------------------------------------------------+ -| | 3 | https://api.nz-por-1.catalystcloud.io:5000 | -+----------+---------+--------------------------------------------------------------------------+ -| nz_wlg_2 | 1 | https://object-storage.nz-wlg-2.catalystcloud.io:443/v1/AUTH_%tenantid% | -+----------+---------+--------------------------------------------------------------------------+ -| | 3 | https://api.cloud.catalyst.net.nz:5000 | -+----------+---------+--------------------------------------------------------------------------+ -| nz-hlz-1 | 1 | https://object-storage.nz-hlz-1.catalystcloud.io:443/v1/AUTH_%tenantid% | -+----------+---------+--------------------------------------------------------------------------+ -| | 3 | https://api.nz-hlz-1.catalystcloud.io:5000 | -+----------+---------+--------------------------------------------------------------------------+ +.. list-table:: + :header-rows: 1 + + * - Region + - API Version + - Endpoint + * - nz-por-1 + - 1 + - ``https://object-storage.nz-por-1.catalystcloud.io:443/v1/AUTH_%project_id%`` + * - + - 3 + - ``https://api.nz-por-1.catalystcloud.io:5000`` + * - nz-hlz-1 + - 1 + - ``https://object-storage.nz-hlz-1.catalystcloud.io:443/v1/AUTH_%project_id%`` + * - + - 3 + - ``https://api.nz-hlz-1.catalystcloud.io:5000`` + * - nz_wlg_2 + - 1 + - ``https://object-storage.nz-wlg-2.catalystcloud.io:443/v1/AUTH_%project_id%`` + * - + - 3 + - ``https://api.cloud.catalyst.net.nz:5000`` .. raw:: html -

Requirements

+

Requirements

-

Sourcing the correct environment variables

+

Sourcing the correct environment variables

Like the the other methods in this tutorial section, you will need to source an openRC file to interact with the object storage APIs. However, there is @@ -173,4 +182,3 @@ Replace the starting section of the previous file with the following: # ...You will then need to remove the previous piece of code that created a "conn=swiftclient.Connection" using the os_token variable. - diff --git a/source/object-storage/using-container.rst b/source/object-storage/using-container.rst index 1e51baeb..8120c72f 100644 --- a/source/object-storage/using-container.rst +++ b/source/object-storage/using-container.rst @@ -9,9 +9,9 @@ object storage service. The following sections cover the most common tools that people use. Each of these examples shows some of the basic commands that you can use to create and edit your object storage containers. -***************** -Via the dashboard -***************** +******************* +Using the dashboard +******************* When using the object storage service, your data must be stored in a container (also referred to as a bucket.) So our first step is to create at least one @@ -24,11 +24,6 @@ the "Containers" section on the dashboard and click "Create Container". You will need to provide a name for the container as well as select the appropriate access level and replication policy type before clicking "Submit". -.. warning:: - - Please do not use an underscore "``_``" when naming your container. The - reasoning for this is explained in the programmatic examples below. - .. note:: Setting "Public" level access on a container means that anyone @@ -58,57 +53,398 @@ the container is now 5 Bytes. .. _object-storage-programmatic-methods: -************************ -Via programmatic methods -************************ +************* +Using the CLI +************* + +The Object Storage service is usable from the CLI using a number of different applications. + +Documentation for the following applications are provided: + +* The Catalyst Cloud Client provides both an easy to use and flexible + interface for using Object Storage, that requires no additional configuration + once you have authenticated your terminal session. +* `Swift Client`_ is the official API client library for Swift, + the system powering Catalyst Cloud's Object Storage service. + It also provides a command-line interface that can be used + as an alternative to the ``openstack container`` + and ``openstack object`` series of commands provided by + Catalyst Cloud Client. Swift Client is more complicated to use, + but provides the most flexible set of commands for interacting + with Object Storage. +* The AWS CLI, with some configuration, can be used to interact with the Object Storage S3 API, + making the wide ecosystem of applications that use AWS CLI available with Catalyst Cloud. +* cURL can be used to directly interact with the Swift API, for full control over API requests. + +.. _`Swift Client`: https://docs.openstack.org/python-swiftclient/latest/introduction.html Prerequisites ============= -For several of the methods detailed below, you will have to prepare your -command line environment before continuing with the examples. The key things -that need to be prepared are: +.. TODO(callumdickinson): Update the below with the new Catalyst Cloud Client documentation link. -* You must :ref:`Source an OpenRC file `. -* You must ensure that you have the correct role for using object storage on - your cloud project. See :ref:`here` for more details. +First, make sure that you have installed the Catalyst Cloud Client +to your system and sourced your OpenRC file in your terminal session, +as shown in the :ref:`CLI Configuration ` documentation. -Once you have met these requirements, you can continue with any of the -method you below: +In addition, you will need to ensure that you have the correct role +for using the Object Storage service on your Catalyst Cloud project. +For more information, see :ref:`access_control`. -.. warning:: +The following steps are specific to the API client you will be using. + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client + + .. include:: cli/catalystcloud-client/prerequisites.rst + + .. group-tab:: Swift Client + + .. include:: cli/swift-client/prerequisites.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/prerequisites.rst + + .. group-tab:: cURL + + .. include:: cli/curl/prerequisites.rst + +Listing containers +================== + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client + + .. include:: cli/catalystcloud-client/container-list.rst + + .. group-tab:: Swift Client + + .. include:: cli/swift-client/container-list.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/container-list.rst + + .. group-tab:: cURL + + .. include:: cli/curl/container-list.rst + +Get container details +===================== + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client + + .. include:: cli/catalystcloud-client/container-show.rst + + .. group-tab:: Swift Client + + .. include:: cli/swift-client/container-show.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/container-show.rst + + .. group-tab:: cURL + + .. include:: cli/curl/container-show.rst + +Creating containers +=================== + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client + + .. include:: cli/catalystcloud-client/container-create.rst + + .. group-tab:: Swift Client + + .. include:: cli/swift-client/container-create.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/container-create.rst + + .. group-tab:: cURL + + .. include:: cli/curl/container-create.rst + +Deleting containers +=================== + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client + + .. include:: cli/catalystcloud-client/container-delete.rst + + .. group-tab:: Swift Client + + .. include:: cli/swift-client/container-delete.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/container-delete.rst + + .. group-tab:: cURL + + .. include:: cli/curl/container-delete.rst + +Listing objects +=============== + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client + + .. include:: cli/catalystcloud-client/object-list.rst + + .. group-tab:: Swift Client + + .. include:: cli/swift-client/object-list.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/object-list.rst + + .. group-tab:: cURL + + .. include:: cli/curl/object-list.rst + +Get object details +================== + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client - Please do not use an underscore "``_``" when naming your containers. This will - cause an error with the S3 api and you will receive errors through your CLI - when using this method. As a general practice we recommend avoiding - the use of an underscore regardless of the method you choose below. + .. include:: cli/catalystcloud-client/object-show.rst -| + .. group-tab:: Swift Client + + .. include:: cli/swift-client/object-show.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/object-show.rst + + .. group-tab:: cURL + + .. include:: cli/curl/object-show.rst + +Downloading objects +=================== + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client + + .. include:: cli/catalystcloud-client/object-save.rst + + .. group-tab:: Swift Client + + .. include:: cli/swift-client/object-save.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/object-save.rst + + .. group-tab:: cURL + + .. include:: cli/curl/object-save.rst + +Uploading objects +================= + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client + + .. include:: cli/catalystcloud-client/object-create.rst + + .. group-tab:: Swift Client + + .. include:: cli/swift-client/object-create.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/object-create.rst + + .. group-tab:: cURL + + .. include:: cli/curl/object-create.rst + +Deleting objects +================ + +.. tabs:: + + .. group-tab:: Catalyst Cloud Client + + .. include:: cli/catalystcloud-client/object-delete.rst + + .. group-tab:: Swift Client + + .. include:: cli/swift-client/object-delete.rst + + .. group-tab:: AWS CLI + + .. include:: cli/aws-cli/object-delete.rst + + .. group-tab:: cURL + + .. include:: cli/curl/object-delete.rst .. _s3-api-documentation: +************** +Using the APIs +************** + +Catalyst Cloud provides two APIs which can be used to interact +with the Object Storage service: + +* `Object Storage API`_ (also known as Swift API), the native object storage API + for OpenStack-based clouds. +* S3 API, the API for Amazon's Simple Storage Service (S3). We provide enough functionality + to make our Object Storage service usable with most applications that only support S3. + For more information on API compatibility, see the `API Comparison Matrix`_. + +Below are some useful instructions for using the respective Python-based API client libraries +for interacting with Catalyst Cloud Object Storage. + +.. tabs:: + + .. tab:: Swift API + + .. include:: tutorial-scripts/swiftAPI.rst + + .. tab:: S3 API + + .. include:: tutorial-scripts/s3api.rst + +.. _`Object Storage API`: https://docs.openstack.org/api-ref/object-store +.. _`API Comparison Matrix`: https://docs.openstack.org/swift/latest/s3_compat.html + +********************************************** +Using orchestration / configuration management +********************************************** + +A number of orchestration and configuration management tools +can be used to automatically create and manage Catalyst Cloud +Object Storage containers and objects. + .. tabs:: - .. tab:: Openstack CLI + .. tab:: Orchestration + + .. include:: tutorial-scripts/heat.rst + + .. tab:: Terraform + + .. include:: tutorial-scripts/terraform.rst - .. include:: tutorial-scripts/openstackCLI.rst +***************** +Naming containers +***************** - .. tab:: Swift API +Object Storage containers on Catalyst Cloud must have a unique name within the project. +Depending on how you interact with your containers, some additional limitations are placed on container names. - .. include:: tutorial-scripts/swiftAPI.rst +In general it is recommended that container names: - .. tab:: S3 API +* Are between 3 and 63 characters long. +* Begin and end with either a letter or a number. +* Consist only of lower-case English letters, numbers or hypens (``-``). + When using S3 API, underscores (``_``) cannot be used (and are otherwise not recommended). - .. include:: tutorial-scripts/s3api.rst +This is fairly restrictive, but it ensures that your containers are compatible +with all available APIs and planned new features. +If this is not a concern, check the naming requirements for your chosen API +clients below to find out the current limitations on container names. - .. tab:: cURL +.. warning:: - .. include:: tutorial-scripts/curl.rst + Object Storage containers **cannot be renamed** once they have been created, + so make sure your chosen container name satisfies your particular use case + **before** creating it. + +.. tabs:: - .. tab:: Heat Orchestration + .. group-tab:: Dashboard / Catalyst Cloud Client / Swift API - .. include:: tutorial-scripts/heat.rst + When creating and using Object Storage containers with either the dashboard, + the Catalyst Cloud Client or the Swift API, you are able to use any name + that complies with the following rules: + + * The name is between 1 to 256 characters long. + * The name can start with any character and contain any pattern, **except** a slash (``/``) character + (used to delimit containers and object names in the API). + * The name's character set must be UTF-8 (non-English characters are allowed). + + .. note:: + + In the future we plan to introduce the ability to use subdomains + to reference Object Storage containers in URLs. + + If you wish to use this feature when it becomes available, + make sure your container names comply with domain name requirements. + Using the more restrictive naming guidelines outlined at the top of this section + will ensure that your containers will be supported. + + .. group-tab:: S3 API + + In addition to the above requirements, when using the S3 API it is **highly recommended** + to follow the Amazon S3 `Bucket Naming Rules`_ to ensure full compatibility. + + .. note:: + + When performing `path-style requests`_ (currently the only supported method on Catalyst Cloud), + periods (``.``) are allowed in container names. + + `Virtual-host-style requests`_, once support has been introduced, will **not** support + periods in container names. This is because the container name is part of + the URL, so the name must comply with domain name requirements. + If you wish to use virtual-host-style requests in the future, please make sure + your container names do not contain periods. + + .. _`Bucket Naming Rules`: https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html + .. _`path-style requests`: https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access + .. _`virtual-host-style requests`: https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access + +************** +Naming objects +************** + +Object names must be unique within the container. + +Generally there are no limitations on what you can name your objects. +Any UTF-8 character can be used (including non-English ones). +This is an important difference from filesystem-based storage, +which is much more restrictive in terms of what you can name files. + +Depending on your application, special handling may be required for +certain characters, such as: + +* Symbol characters, e.g. ``&``, ``$``, ``@``, ``=``, ``;``, ``:``, ``+``, ``?`` +* Whitespace +* ASCII character ranges 00-1F hex (0-31 decimal) and 7F (127 decimal) + +It is recommended to avoid using the following characters due to the +way certain applications handle them: + +* Parsing symbols, e.g. ``\``, ``{``, ``}``, ``^``, ``%``, `````, ``[``, ``]``, + ``"``, ``<``, ``>``, ``~``, ``#``, ``|`` +* Non-printable ASCII characters (128-255 decimal) + +.. note:: - .. tab:: Terraform + When using the S3 API, make sure the object name complies with the Amazon S3 + `Object Key Naming Guidelines`_. - .. include:: tutorial-scripts/terraform.rst + .. _`Object Key Naming Guidelines`: https://docs.aws.amazon.com/en_us/AmazonS3/latest/userguide/object-keys.html#object-key-guidelines