diff --git a/README.md b/README.md index 01d3cefef..906ccf97e 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ Skyplane is a tool for blazingly fast bulk data transfers between object stores Skyplane is: 1. 🔥 Blazing fast ([110x faster than AWS DataSync](https://skyplane.org/en/latest/benchmark.html)) 2. 🤑 Cheap (4x cheaper than rsync) -3. 🌐 Universal (AWS, Azure, IBM and GCP) +3. 🌐 Universal (AWS, Azure, IBM, SCP and GCP) You can use Skyplane to transfer data: * between object stores within a cloud provider (e.g. AWS us-east-1 to AWS us-west-2) @@ -45,6 +45,7 @@ Skyplane currently supports the following source and destination endpoints (any | Google Storage | :white_check_mark: | :white_check_mark: | | Azure Blob Storage | :white_check_mark: | :white_check_mark: | | IBM Cloud Object Storage | :white_check_mark: | :white_check_mark: | +| Samsung Cloud Platform Object Storage | :white_check_mark: | :white_check_mark: | | Local Disk | :white_check_mark: | (in progress) | Skyplane is an actively developed project. It will have 🔪 SHARP EDGES 🔪. Please file an issue or ask the contributors via [the #help channel on our Slack](https://join.slack.com/t/skyplaneworkspace/shared_invite/zt-1cxmedcuc-GwIXLGyHTyOYELq7KoOl6Q) if you encounter bugs. @@ -67,10 +68,11 @@ $ pip install "skyplane[aws]" # $ pip install "skyplane[azure]" # $ pip install "skyplane[gcp]" # $ pip install "skyplane[ibmcloud]" +# $ pip install "skyplane[scp]" # $ pip install "skyplane[all]" ``` -Skyplane supports AWS, Azure, IBM and GCP. You can install Skyplane with support for one or more of these clouds by specifying the corresponding extras. To install two out of three clouds, you can run `pip install "skyplane[aws,azure]"`. +Skyplane supports AWS, Azure, IBM, SCP and GCP. You can install Skyplane with support for one or more of these clouds by specifying the corresponding extras. To install two out of three clouds, you can run `pip install "skyplane[aws,azure]"`. *GCP support on the M1 Mac*: If you are using an M1 Mac with the arm64 architecture and want to install GCP support for Skyplane, you will need to install as follows `GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install "skyplane[aws,gcp]"` @@ -110,10 +112,10 @@ IBM IAM key and credentials to your IBM Cloud object storage ---> For SCP: $ # Create directory if required $ mkdir -p ~/.scp -$ # Add the lines for "access_key", "secret_key", and "project_id" to scp_credential file -$ echo "access_key = " >> ~/.scp/scp_credential -$ echo "secret_key = " >> ~/.scp/scp_credential -$ echo "project_id = " >> ~/.scp/scp_credential +$ # Add the lines for "scp_access_key", "scp_secret_key", and "scp_project_id" to scp_credential file +$ echo "scp_access_key = " >> ~/.scp/scp_credential +$ echo "scp_secret_key = " >> ~/.scp/scp_credential +$ echo "scp_project_id = " >> ~/.scp/scp_credential ``` After authenticating with each cloud provider, you can run `skyplane init` to create a configuration file for Skyplane. diff --git a/docs/_api/skyplane.api.config.rst b/docs/_api/skyplane.api.config.rst index 70bcc6708..bc48496e6 100644 --- a/docs/_api/skyplane.api.config.rst +++ b/docs/_api/skyplane.api.config.rst @@ -22,6 +22,7 @@ AzureConfig GCPConfig IBMCloudConfig + SCPConfig TransferConfig diff --git a/docs/architecture.md b/docs/architecture.md index ff8a6f19f..54c1d2111 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -38,5 +38,8 @@ While GCP VPCs are Global, in AWS for every region that is involved in a transfe Firewall support for Azure is in the roadmap. +### SCP +Similar to AWS, in SCP(Samsung Cloud Platform) for every location(region) that is involved in a transfer, Skyplane creates a `skyplane` [VPC](https://github.com/skyplane-project/skyplane/blob/main/skyplane/compute/scp/scp_network.py#L267), subnet, security group(SG). When the Gateway instance is created, port 22 is initially [registered](https://github.com/skyplane-project/skyplane/blob/main/skyplane/compute/scp/scp_cloud_provider.py#L194), and subsequently, firewall rules for all IPs are [registered](https://github.com/skyplane-project/skyplane/blob/main/skyplane/compute/scp/scp_cloud_provider.py#L225) for data relay. After the transfer, the firewalls are [deleted](https://github.com/skyplane-project/skyplane/blob/main/skyplane/compute/scp/scp_cloud_provider.py#L228). + ## Large Objects Skyplane breaks large objects into smaller sub-parts (currently AWS and GCP only) to improve transfer parallelism (also known as [striping](https://ieeexplore.ieee.org/document/1560006)). diff --git a/docs/configure.md b/docs/configure.md index 0f8e817e0..1bcd25dcf 100644 --- a/docs/configure.md +++ b/docs/configure.md @@ -55,7 +55,8 @@ By default, Skyplane will use a maximum of 1 VM in each region. This limit is co * `ibmcloud_use_spot_instances`: If set, IBM will use spot instances instead of on-demand instances. (default False) * `ibmcloud_default_region`: IBM region to use for provisioning. (default us-east) * `ibmcloud_instance_class`: IBM instance class to use for provisioning. (default bx2-2x8) - +* `scp_instance_class`: SCP instance class to use for provisioning. (default h1v32m128) +* `scp_default_region`: SCP region to use for provisioning. (default KR-WEST-1) For Azure: $ az login + ---> For SCP: + $ # Create directory if required + $ mkdir -p ~/.scp + $ # Add the lines for "scp_access_key", "scp_secret_key", and "scp_project_id" to scp_credential file + $ echo "scp_access_key = " >> ~/.scp/scp_credential + $ echo "scp_secret_key = " >> ~/.scp/scp_credential + $ echo "scp_project_id = " >> ~/.scp/scp_credential + Now, you can initialize Skyplane with your desired cloud providers. Skyplane autodetects cloud credentials and valid regions from your CLI environment. .. code-block:: bash diff --git a/docs/permissions.rst b/docs/permissions.rst index 3711a507c..6936ca027 100644 --- a/docs/permissions.rst +++ b/docs/permissions.rst @@ -40,4 +40,12 @@ Your Azure account must have the following roles: Within Azure, it is not sufficient to have just the :code:`Owner` role to be able to access and write to containers in storage. The VMs that Skyplane provisions are assigned the sufficient storage permissions, but to be able to interact with Azure storage locally, check to make sure your personal Azure account has the roles listed above. +SCP +----------------------------- +To ensure smooth authentication key usage and access to Object Storage, please verify the following permissions: + +- :code:`Access allowance IP for Your SCP account: Disabled` +- :code:`Object Storage access control feature: Disabled` + +Configure the newly created Gateway instances to enable authentication and Object Storage access. diff --git a/docs/summary.md b/docs/summary.md index 859ebea05..b6cc637cf 100644 --- a/docs/summary.md +++ b/docs/summary.md @@ -3,7 +3,7 @@ ``` pip install skyplane[aws] skyplane init -skyplane [sync/cp] [local/s3/gs/azure]://mybucket/big_dataset [local/s3/gs/azure]://mybucket2/ +skyplane [sync/cp] [local/s3/gs/azure/scp]://mybucket/big_dataset [local/s3/gs/azure/scp]://mybucket2/ ``` Skyplane is a tool for blazingly fast bulk data transfers between object stores in the cloud. It provisions a fleet of VMs in the cloud to transfer data in parallel while using compression and bandwidth tiering to reduce cost. @@ -11,7 +11,7 @@ Skyplane is a tool for blazingly fast bulk data transfers between object stores Skyplane is: 1. 🔥 Blazing fast ([110x faster than AWS DataSync](https://skyplane.org/en/latest/benchmark.html)) 2. 🤑 Cheap (4x cheaper than rsync) -3. 🌐 Universal (AWS, Azure and GCP) +3. 🌐 Universal (AWS, Azure, IBM, SCP and GCP) You can use Skyplane to transfer data: * between object stores within a cloud provider (e.g. AWS us-east-1 to AWS us-west-2) @@ -26,6 +26,7 @@ Skyplane currently supports the following source and destination endpoints (any | Google Storage | ✅ | ✅ | | Azure Blob Storage | ✅ | ✅ | | Cloudflare R2 | ✅ | ✅ | +| Samsung Cloud Platform Object Storage | ✅ | ✅ | | Local Disk | ✅ | (in progress) | Skyplane is an actively developed project. It will have 🔪 SHARP EDGES 🔪. Please file an issue or ask the contributors via [the #help channel on our Slack](https://join.slack.com/t/skyplaneworkspace/shared_invite/zt-1cxmedcuc-GwIXLGyHTyOYELq7KoOl6Q) if you encounter bugs.