docs: add aws deployment details (#101)

* docs: add aws deployment details * fix(linting): code formatting * chore: add documentation for providers login. * docs: add security details. * docs: add security information. * docs: add sdk reference * fix(linting): code formatting --------- Co-authored-by: Fabiana Clemente <[email protected]> Co-authored-by: Azory YData Bot <[email protected]>
ydataai · Jun 18, 2024 · 340a4f8 · 340a4f8
1 parent 8489a91
commit 340a4f8
Show file tree

Hide file tree

Showing 54 changed files with 893 additions and 0 deletions.
diff --git a/docs/assets/deployment_security/aws/aknowledge_cloud_formation.png b/docs/assets/deployment_security/aws/aknowledge_cloud_formation.png
diff --git a/docs/assets/deployment_security/aws/allow_analytics.png b/docs/assets/deployment_security/aws/allow_analytics.png
diff --git a/docs/assets/deployment_security/aws/auth_providers.png b/docs/assets/deployment_security/aws/auth_providers.png
diff --git a/docs/assets/deployment_security/aws/aws_acm_certificate_1.png b/docs/assets/deployment_security/aws/aws_acm_certificate_1.png
diff --git a/docs/assets/deployment_security/aws/aws_acm_certificate_2.png b/docs/assets/deployment_security/aws/aws_acm_certificate_2.png
diff --git a/docs/assets/deployment_security/aws/aws_eks_service.png b/docs/assets/deployment_security/aws/aws_eks_service.png
diff --git a/docs/assets/deployment_security/aws/aws_iam_role.png b/docs/assets/deployment_security/aws/aws_iam_role.png
diff --git a/docs/assets/deployment_security/aws/aws_import_acm_certificate.png b/docs/assets/deployment_security/aws/aws_import_acm_certificate.png
diff --git a/docs/assets/deployment_security/aws/aws_import_acm_certificate_2.png b/docs/assets/deployment_security/aws/aws_import_acm_certificate_2.png
diff --git a/docs/assets/deployment_security/aws/aws_route53_ex.png b/docs/assets/deployment_security/aws/aws_route53_ex.png
diff --git a/docs/assets/deployment_security/aws/aws_stack_default.png b/docs/assets/deployment_security/aws/aws_stack_default.png
diff --git a/docs/assets/deployment_security/aws/bastion_host_app_rules.png b/docs/assets/deployment_security/aws/bastion_host_app_rules.png
diff --git a/docs/assets/deployment_security/aws/bastion_host_info.png b/docs/assets/deployment_security/aws/bastion_host_info.png
diff --git a/docs/assets/deployment_security/aws/bastion_host_security_group.png b/docs/assets/deployment_security/aws/bastion_host_security_group.png
diff --git a/docs/assets/deployment_security/aws/cancel_subscription.png b/docs/assets/deployment_security/aws/cancel_subscription.png
diff --git a/docs/assets/deployment_security/aws/cloud_formation_new_link.png b/docs/assets/deployment_security/aws/cloud_formation_new_link.png
diff --git a/docs/assets/deployment_security/aws/cloud_formation_registry.png b/docs/assets/deployment_security/aws/cloud_formation_registry.png
diff --git a/docs/assets/deployment_security/aws/cognito_auth.png b/docs/assets/deployment_security/aws/cognito_auth.png
diff --git a/docs/assets/deployment_security/aws/copy_link.png b/docs/assets/deployment_security/aws/copy_link.png
diff --git a/docs/assets/deployment_security/aws/delete_stack.png b/docs/assets/deployment_security/aws/delete_stack.png
diff --git a/docs/assets/deployment_security/aws/dns_configuration.png b/docs/assets/deployment_security/aws/dns_configuration.png
diff --git a/docs/assets/deployment_security/aws/eks_cluster_delete.png b/docs/assets/deployment_security/aws/eks_cluster_delete.png
diff --git a/docs/assets/deployment_security/aws/init_platform_install.png b/docs/assets/deployment_security/aws/init_platform_install.png
diff --git a/docs/assets/deployment_security/aws/install_completed.png b/docs/assets/deployment_security/aws/install_completed.png
diff --git a/docs/assets/deployment_security/aws/launch_more_software.png b/docs/assets/deployment_security/aws/launch_more_software.png
diff --git a/docs/assets/deployment_security/aws/network_default_parameters.png b/docs/assets/deployment_security/aws/network_default_parameters.png
diff --git a/docs/assets/deployment_security/aws/submit_cloud_formation.png b/docs/assets/deployment_security/aws/submit_cloud_formation.png
diff --git a/docs/assets/deployment_security/aws/update_launch.png b/docs/assets/deployment_security/aws/update_launch.png
diff --git a/docs/assets/deployment_security/aws/ydata_subscription.png b/docs/assets/deployment_security/aws/ydata_subscription.png
diff --git a/docs/assets/deployment_security/login_support/cognito_create_users.png b/docs/assets/deployment_security/login_support/cognito_create_users.png
diff --git a/docs/assets/deployment_security/login_support/cognito_private_dns.png b/docs/assets/deployment_security/login_support/cognito_private_dns.png
diff --git a/docs/assets/deployment_security/login_support/cognito_public_dns.png b/docs/assets/deployment_security/login_support/cognito_public_dns.png
diff --git a/docs/assets/deployment_security/login_support/github_custom_dns.png b/docs/assets/deployment_security/login_support/github_custom_dns.png
diff --git a/docs/assets/deployment_security/login_support/github_public_dns.png b/docs/assets/deployment_security/login_support/github_public_dns.png
diff --git a/docs/assets/deployment_security/login_support/google_custom_domain_dns.png b/docs/assets/deployment_security/login_support/google_custom_domain_dns.png
diff --git a/docs/assets/deployment_security/login_support/google_public_dns.png b/docs/assets/deployment_security/login_support/google_public_dns.png
diff --git a/docs/assets/deployment_security/login_support/msft_access_users_groups.png b/docs/assets/deployment_security/login_support/msft_access_users_groups.png
diff --git a/docs/assets/deployment_security/login_support/msft_assignment_required.png b/docs/assets/deployment_security/login_support/msft_assignment_required.png
diff --git a/docs/assets/deployment_security/login_support/msft_consent.png b/docs/assets/deployment_security/login_support/msft_consent.png
diff --git a/docs/deployment_and_security/deployment/aws/bastion_host.md b/docs/deployment_and_security/deployment/aws/bastion_host.md
@@ -0,0 +1,24 @@
+# Bastion host
+During the installation, the user will be prompt with the possibility of allowing the creation of a bastion host.
+This bastion host is used by YData to give a closer support to the users.
+If you allow the creation of this bastion host, an EC2 will be created during installation with NO ingress rules on his security group.
+
+In case is needed, you will need to send the bastion host Elastic IP to YData Fabric and add an ingress rule to the security group as explained below.
+In the *CloudFormation* outputs you can find the relevant information of the EC2 bastion host, such as, elastic IP, the EC2 instance ID and the security group ID:
+
+![bastion host](../../../assets/deployment_security/aws/bastion_host_info.png){: style="width:65%"}
+
+## Setting the SG ingress rule
+- To give access to the bastion host, please go to the **EC2 service → Security Groups**.
+- You can search for the security group ID provided on the template outputs:
+
+![bastion host](../../../assets/deployment_security/aws/bastion_host_security_group.png){: style="width:65%"}
+
+- Go to the *"Inbound rules"* tab and click *"Edit"* inbound rules.
+- You can then, add an **inbound rule to allow the access** to the bastion host and click *Save* rules, as per the image below.
+
+![bastion host rules](../../../assets/deployment_security/aws/bastion_host_app_rules.png){: style="width:65%"}
+
+- For single IP source, an IP will be given to you on the support time via email.
+## Removing the SG ingress rule
+- As soon the support for the specific case ends, you must **remove the SG ingress rule** and click *Save* rules.
diff --git a/docs/deployment_and_security/deployment/aws/clean.md b/docs/deployment_and_security/deployment/aws/clean.md
@@ -0,0 +1,35 @@
+# Clean
+
+The following procedure explains how to delete the platform. The full procedure takes around 45m to 1h to be completed.
+To clean up **YData Fabric**, you will need to delete the CloudFormation stack and remove the subscription.
+
+Please take in consideration that this will delete **everything associated with the installation**.
+
+## Deleting the stacks
+- Go to the regions where the product is installed
+- Go to the *CloudFormation* service
+- Select the *ydata stack*
+- Click in the **Delete** button
+
+![delete stack](../../../assets/deployment_security/aws/delete_stack.png){: style="width:65%"}
+
+- Select the Extension stack and click in the **Delete** button.
+
+!!! Note
+
+    This will disable the extension. If you are using this extension for any other project, please do not delete this stack.
+
+![EKS cluster delete](../../../assets/deployment_security/aws/eks_cluster_delete.png){: style="width:65%"}
+
+## Deleting the subscription
+- Go to the ^^[**AWS Marketplace Subscriptions](https://console.aws.amazon.com/marketplace/home?region=eu-west-1)^^** → Manage subscriptions
+- Click the *YData product*
+
+![ydata product](../../../assets/deployment_security/aws/ydata_subscription.png){: style="width:65%"}
+
+- **Actions → Cancel** subscription
+- Click the checkbox and click *Yes*, cancel subscription
+
+![cancel](../../../assets/deployment_security/aws/cancel_subscription.png){: style="width:65%"}
+
+Following the above steps completes the process of deleting YData Fabric from your AWS Cloud instance.
diff --git a/docs/deployment_and_security/deployment/aws/deploy.md b/docs/deployment_and_security/deployment/aws/deploy.md
@@ -0,0 +1,87 @@
+# Deploy
+
+## Installation process
+The following procedure explains how to install the platform using the CloudFormation template and how to connect
+to the platform after the installation. The full procedure takes around 45m to 1h to be completed.
+In order to install the platform in your account, the user must have basic knowledge with the used tools, such as CloudFormation,
+Route53 and Cognito.
+
+### 1. Configure the product
+
+!!! Note "Make sure that you comply with the pre-flight checks"
+
+    You can check the ^^[prerequisites and pre-deploy checks](pre_deploy_checklist.md)^^.
+
+Start with the basic configuration for the app installation:
+
+- Ensure you are in the right region.
+- Choose the stack name *"ydata-platform"* is the default name
+![default stack](../../../assets/deployment_security/aws/aws_stack_default.png){: style="width:75%"}
+
+#### Network
+Define your network configurations to access the platform.
+Using the `ACM Certificate ARN` **OR** the `Hosted Zone ID` and the `Domain` chosen from the preflight checklist,
+fill up the following parameters:
+![default network parameters](../../../assets/deployment_security/aws/network_default_parameters.png){: style="width:75%"}
+
+#### OAuth
+Define how your users will authenticate in the platform (you can use multiple providers).
+![auth providers](../../../assets/deployment_security/aws/auth_providers.png){: style="width:55%"}
+
+#### Analytics
+You can opt for allowing or not the collection of metrics in order to help us understand how users interact with the product.
+No user data is collected at any point.
+You can find our privacy policy at ^^[ydata.ai/privacy](https://ydata.ai/privacy)^^.
+![auth providers](../../../assets/deployment_security/aws/allow_analytics.png){: style="width:70%"}
+
+#### Bastion host
+A bastion host is created and used to give closer support to the users.
+The bastion host is only accessible on user demand, giving us access to EC2 setting an SG ingress rule.
+Set it to *"Allow"* to have it available.
+More information [here](bastion_host.md).
+
+#### Create
+
+- Check the “**I acknowledge that AWS CloudFormation might create IAM resources with custom names.”**
+- Click **Create Stack**
+
+![Acknowledge cloud formation](../../../assets/deployment_security/aws/aknowledge_cloud_formation.png){: style="width:70%"}
+
+### 2. Following the installation process
+
+Now we can follow the step-by-step for the installation of ^^[YData Fabric](https://ydata.ai/products/fabric)^^.
+
+- Click the “Create” button, the installation of the platform will start:
+
+![Init platform install](../../../assets/deployment_security/aws/init_platform_install.png){: style="width:40%"}
+
+The process will take approximately 45-60 minutes.
+
+- If the installation process occurs without any issues, you will see the **CREATE_COMPLETE** status in the stack:
+
+![Install completed](../../../assets/deployment_security/aws/install_completed.png){: style="width:40%"}
+
+- If any error occur during installation, please open a support case at ^^[support.ydata.ai](https://support.ydata.ai)^^.
+
+### 3. Post installation configuration
+
+#### DNS Configuration
+If you have your domain registered in Route53, you can check the CF Outputs, and click the domain name to access the
+platform:
+
+![DNS Configuration](../../../assets/deployment_security/aws/dns_configuration.png){: style="width:40%"}
+
+If you are using another DNS provider or a Route53 in another account, you will need to create a CNAME record pointing
+to the ALB endpoint (ALBDNSName). As an example:
+`CNAME → ydata-alb-xxxxxxxxx.eu-west-1.elb.amazonaws.com`
+
+## 4. Connecting to the platform
+To connect the platform, **please allow 20-30m so the platform is completed initialised** and access using the URL
+displayed in the CF Outputs.
+For the login process, if you choose a customer custom login provider, you need to ensure that the users are created.
+
+Otherwise, you will need to create the users in the Cognito generated by the CloudFormation stack.
+
+More information under can be found at ^^[Login providers](../login_support/login_providers.md)^^.
+
+🚀 Congratulations you are now ready to start exploring your data with **YData Fabric**!
diff --git a/docs/deployment_and_security/deployment/aws/instance_types.md b/docs/deployment_and_security/deployment/aws/instance_types.md
@@ -0,0 +1,32 @@
+# Instance types
+
+| Name | ID | Supported | System Pool | CPU MIcro Pool | CPU Small Pool | CPU Medium Pool | CPU Large Pool | CPU Compute Micro Pool | GPU MIcro Pool | GPU Compute Micro Pool | Bastion Host |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| N. Virginia | us-east-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Ohio | us-east-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| N. California | us-west-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Oregon | us-west-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Cape Town | af-south-1 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano |
+| Melbourne | ap-southeast-4 | 🔴 | - | - | - | - | - | - | - | - | - |
+| Hong Kong | ap-east-1 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano |
+| Hyderabad | ap-south-2 | 🔴 | - | - | - | - | - | - | - | - | - |
+| Jakarta | ap-southeast-3 | 🔴 | - | - | - | - | - | - | - | - | - |
+| Mumbai | ap-south-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3a.nano |
+| Osaka | ap-northeast-3 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano |
+| Seoul | ap-northeast-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Singapore | ap-southeast-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Sydney | ap-southeast-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Tokyo | ap-northeast-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Canada Central | ca-central-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Frankfurt | eu-central-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Ireland | eu-west-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| London | eu-west-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano |
+| Milan | eu-south-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3a.nano |
+| Paris | eu-west-3 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3a.nano |
+| Spain | eu-south-2 | 🔴 | - | - | - | - | - | - | - | - | - |
+| Stockholm | eu-north-1 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano |
+| Zurich | eu-central-2 | 🔴 | - | - | - | - | - | - | - | - | - |
+| Bahrain | me-south-1 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano |
+| UAE | me-central-1 | 🔴 | - | - | - | - | - | - | - | - | - |
+| Tel Aviv | il-central-1 | 🔴 | - | - | - | - | - | - | - | - | - |
+| São Paulo | sa-east-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3a.nano |