diff --git a/docs/assets/deployment_security/aws/aknowledge_cloud_formation.png b/docs/assets/deployment_security/aws/aknowledge_cloud_formation.png new file mode 100644 index 00000000..ac4a0061 Binary files /dev/null and b/docs/assets/deployment_security/aws/aknowledge_cloud_formation.png differ diff --git a/docs/assets/deployment_security/aws/allow_analytics.png b/docs/assets/deployment_security/aws/allow_analytics.png new file mode 100644 index 00000000..d8213f49 Binary files /dev/null and b/docs/assets/deployment_security/aws/allow_analytics.png differ diff --git a/docs/assets/deployment_security/aws/auth_providers.png b/docs/assets/deployment_security/aws/auth_providers.png new file mode 100644 index 00000000..05ad402d Binary files /dev/null and b/docs/assets/deployment_security/aws/auth_providers.png differ diff --git a/docs/assets/deployment_security/aws/aws_acm_certificate_1.png b/docs/assets/deployment_security/aws/aws_acm_certificate_1.png new file mode 100644 index 00000000..881f73bc Binary files /dev/null and b/docs/assets/deployment_security/aws/aws_acm_certificate_1.png differ diff --git a/docs/assets/deployment_security/aws/aws_acm_certificate_2.png b/docs/assets/deployment_security/aws/aws_acm_certificate_2.png new file mode 100644 index 00000000..28da3a8a Binary files /dev/null and b/docs/assets/deployment_security/aws/aws_acm_certificate_2.png differ diff --git a/docs/assets/deployment_security/aws/aws_eks_service.png b/docs/assets/deployment_security/aws/aws_eks_service.png new file mode 100644 index 00000000..302748cd Binary files /dev/null and b/docs/assets/deployment_security/aws/aws_eks_service.png differ diff --git a/docs/assets/deployment_security/aws/aws_iam_role.png b/docs/assets/deployment_security/aws/aws_iam_role.png new file mode 100644 index 00000000..cbd62ab0 Binary files /dev/null and b/docs/assets/deployment_security/aws/aws_iam_role.png differ diff --git a/docs/assets/deployment_security/aws/aws_import_acm_certificate.png b/docs/assets/deployment_security/aws/aws_import_acm_certificate.png new file mode 100644 index 00000000..4dcbd465 Binary files /dev/null and b/docs/assets/deployment_security/aws/aws_import_acm_certificate.png differ diff --git a/docs/assets/deployment_security/aws/aws_import_acm_certificate_2.png b/docs/assets/deployment_security/aws/aws_import_acm_certificate_2.png new file mode 100644 index 00000000..7e144ead Binary files /dev/null and b/docs/assets/deployment_security/aws/aws_import_acm_certificate_2.png differ diff --git a/docs/assets/deployment_security/aws/aws_route53_ex.png b/docs/assets/deployment_security/aws/aws_route53_ex.png new file mode 100644 index 00000000..1090a16c Binary files /dev/null and b/docs/assets/deployment_security/aws/aws_route53_ex.png differ diff --git a/docs/assets/deployment_security/aws/aws_stack_default.png b/docs/assets/deployment_security/aws/aws_stack_default.png new file mode 100644 index 00000000..58987dc6 Binary files /dev/null and b/docs/assets/deployment_security/aws/aws_stack_default.png differ diff --git a/docs/assets/deployment_security/aws/bastion_host_app_rules.png b/docs/assets/deployment_security/aws/bastion_host_app_rules.png new file mode 100644 index 00000000..2a8fc7bd Binary files /dev/null and b/docs/assets/deployment_security/aws/bastion_host_app_rules.png differ diff --git a/docs/assets/deployment_security/aws/bastion_host_info.png b/docs/assets/deployment_security/aws/bastion_host_info.png new file mode 100644 index 00000000..44a2345a Binary files /dev/null and b/docs/assets/deployment_security/aws/bastion_host_info.png differ diff --git a/docs/assets/deployment_security/aws/bastion_host_security_group.png b/docs/assets/deployment_security/aws/bastion_host_security_group.png new file mode 100644 index 00000000..d8465622 Binary files /dev/null and b/docs/assets/deployment_security/aws/bastion_host_security_group.png differ diff --git a/docs/assets/deployment_security/aws/cancel_subscription.png b/docs/assets/deployment_security/aws/cancel_subscription.png new file mode 100644 index 00000000..ad581143 Binary files /dev/null and b/docs/assets/deployment_security/aws/cancel_subscription.png differ diff --git a/docs/assets/deployment_security/aws/cloud_formation_new_link.png b/docs/assets/deployment_security/aws/cloud_formation_new_link.png new file mode 100644 index 00000000..37ad0bba Binary files /dev/null and b/docs/assets/deployment_security/aws/cloud_formation_new_link.png differ diff --git a/docs/assets/deployment_security/aws/cloud_formation_registry.png b/docs/assets/deployment_security/aws/cloud_formation_registry.png new file mode 100644 index 00000000..7beb6bbe Binary files /dev/null and b/docs/assets/deployment_security/aws/cloud_formation_registry.png differ diff --git a/docs/assets/deployment_security/aws/cognito_auth.png b/docs/assets/deployment_security/aws/cognito_auth.png new file mode 100644 index 00000000..744982c0 Binary files /dev/null and b/docs/assets/deployment_security/aws/cognito_auth.png differ diff --git a/docs/assets/deployment_security/aws/copy_link.png b/docs/assets/deployment_security/aws/copy_link.png new file mode 100644 index 00000000..4d6df4e1 Binary files /dev/null and b/docs/assets/deployment_security/aws/copy_link.png differ diff --git a/docs/assets/deployment_security/aws/delete_stack.png b/docs/assets/deployment_security/aws/delete_stack.png new file mode 100644 index 00000000..b6909563 Binary files /dev/null and b/docs/assets/deployment_security/aws/delete_stack.png differ diff --git a/docs/assets/deployment_security/aws/dns_configuration.png b/docs/assets/deployment_security/aws/dns_configuration.png new file mode 100644 index 00000000..19706bf3 Binary files /dev/null and b/docs/assets/deployment_security/aws/dns_configuration.png differ diff --git a/docs/assets/deployment_security/aws/eks_cluster_delete.png b/docs/assets/deployment_security/aws/eks_cluster_delete.png new file mode 100644 index 00000000..ace37c5d Binary files /dev/null and b/docs/assets/deployment_security/aws/eks_cluster_delete.png differ diff --git a/docs/assets/deployment_security/aws/init_platform_install.png b/docs/assets/deployment_security/aws/init_platform_install.png new file mode 100644 index 00000000..ddff69c6 Binary files /dev/null and b/docs/assets/deployment_security/aws/init_platform_install.png differ diff --git a/docs/assets/deployment_security/aws/install_completed.png b/docs/assets/deployment_security/aws/install_completed.png new file mode 100644 index 00000000..f51027d2 Binary files /dev/null and b/docs/assets/deployment_security/aws/install_completed.png differ diff --git a/docs/assets/deployment_security/aws/launch_more_software.png b/docs/assets/deployment_security/aws/launch_more_software.png new file mode 100644 index 00000000..9b130404 Binary files /dev/null and b/docs/assets/deployment_security/aws/launch_more_software.png differ diff --git a/docs/assets/deployment_security/aws/network_default_parameters.png b/docs/assets/deployment_security/aws/network_default_parameters.png new file mode 100644 index 00000000..cd60cb97 Binary files /dev/null and b/docs/assets/deployment_security/aws/network_default_parameters.png differ diff --git a/docs/assets/deployment_security/aws/submit_cloud_formation.png b/docs/assets/deployment_security/aws/submit_cloud_formation.png new file mode 100644 index 00000000..5d78d42c Binary files /dev/null and b/docs/assets/deployment_security/aws/submit_cloud_formation.png differ diff --git a/docs/assets/deployment_security/aws/update_launch.png b/docs/assets/deployment_security/aws/update_launch.png new file mode 100644 index 00000000..1765ff88 Binary files /dev/null and b/docs/assets/deployment_security/aws/update_launch.png differ diff --git a/docs/assets/deployment_security/aws/ydata_subscription.png b/docs/assets/deployment_security/aws/ydata_subscription.png new file mode 100644 index 00000000..20346038 Binary files /dev/null and b/docs/assets/deployment_security/aws/ydata_subscription.png differ diff --git a/docs/assets/deployment_security/login_support/cognito_create_users.png b/docs/assets/deployment_security/login_support/cognito_create_users.png new file mode 100644 index 00000000..e04ce40b Binary files /dev/null and b/docs/assets/deployment_security/login_support/cognito_create_users.png differ diff --git a/docs/assets/deployment_security/login_support/cognito_private_dns.png b/docs/assets/deployment_security/login_support/cognito_private_dns.png new file mode 100644 index 00000000..1b548b65 Binary files /dev/null and b/docs/assets/deployment_security/login_support/cognito_private_dns.png differ diff --git a/docs/assets/deployment_security/login_support/cognito_public_dns.png b/docs/assets/deployment_security/login_support/cognito_public_dns.png new file mode 100644 index 00000000..b85812b9 Binary files /dev/null and b/docs/assets/deployment_security/login_support/cognito_public_dns.png differ diff --git a/docs/assets/deployment_security/login_support/github_custom_dns.png b/docs/assets/deployment_security/login_support/github_custom_dns.png new file mode 100644 index 00000000..0696494c Binary files /dev/null and b/docs/assets/deployment_security/login_support/github_custom_dns.png differ diff --git a/docs/assets/deployment_security/login_support/github_public_dns.png b/docs/assets/deployment_security/login_support/github_public_dns.png new file mode 100644 index 00000000..c7bb199e Binary files /dev/null and b/docs/assets/deployment_security/login_support/github_public_dns.png differ diff --git a/docs/assets/deployment_security/login_support/google_custom_domain_dns.png b/docs/assets/deployment_security/login_support/google_custom_domain_dns.png new file mode 100644 index 00000000..d5275a2e Binary files /dev/null and b/docs/assets/deployment_security/login_support/google_custom_domain_dns.png differ diff --git a/docs/assets/deployment_security/login_support/google_public_dns.png b/docs/assets/deployment_security/login_support/google_public_dns.png new file mode 100644 index 00000000..25bed433 Binary files /dev/null and b/docs/assets/deployment_security/login_support/google_public_dns.png differ diff --git a/docs/assets/deployment_security/login_support/msft_access_users_groups.png b/docs/assets/deployment_security/login_support/msft_access_users_groups.png new file mode 100644 index 00000000..47dcbe2a Binary files /dev/null and b/docs/assets/deployment_security/login_support/msft_access_users_groups.png differ diff --git a/docs/assets/deployment_security/login_support/msft_assignment_required.png b/docs/assets/deployment_security/login_support/msft_assignment_required.png new file mode 100644 index 00000000..f08d7e85 Binary files /dev/null and b/docs/assets/deployment_security/login_support/msft_assignment_required.png differ diff --git a/docs/assets/deployment_security/login_support/msft_consent.png b/docs/assets/deployment_security/login_support/msft_consent.png new file mode 100644 index 00000000..db05d808 Binary files /dev/null and b/docs/assets/deployment_security/login_support/msft_consent.png differ diff --git a/docs/deployment_and_security/deployment/aws/bastion_host.md b/docs/deployment_and_security/deployment/aws/bastion_host.md new file mode 100644 index 00000000..48668b5d --- /dev/null +++ b/docs/deployment_and_security/deployment/aws/bastion_host.md @@ -0,0 +1,24 @@ +# Bastion host +During the installation, the user will be prompt with the possibility of allowing the creation of a bastion host. +This bastion host is used by YData to give a closer support to the users. +If you allow the creation of this bastion host, an EC2 will be created during installation with NO ingress rules on his security group. + +In case is needed, you will need to send the bastion host Elastic IP to YData Fabric and add an ingress rule to the security group as explained below. +In the *CloudFormation* outputs you can find the relevant information of the EC2 bastion host, such as, elastic IP, the EC2 instance ID and the security group ID: + +![bastion host](../../../assets/deployment_security/aws/bastion_host_info.png){: style="width:65%"} + +## Setting the SG ingress rule +- To give access to the bastion host, please go to the **EC2 service → Security Groups**. +- You can search for the security group ID provided on the template outputs: + +![bastion host](../../../assets/deployment_security/aws/bastion_host_security_group.png){: style="width:65%"} + +- Go to the *"Inbound rules"* tab and click *"Edit"* inbound rules. +- You can then, add an **inbound rule to allow the access** to the bastion host and click *Save* rules, as per the image below. + +![bastion host rules](../../../assets/deployment_security/aws/bastion_host_app_rules.png){: style="width:65%"} + +- For single IP source, an IP will be given to you on the support time via email. +## Removing the SG ingress rule +- As soon the support for the specific case ends, you must **remove the SG ingress rule** and click *Save* rules. diff --git a/docs/deployment_and_security/deployment/aws/clean.md b/docs/deployment_and_security/deployment/aws/clean.md new file mode 100644 index 00000000..314725f7 --- /dev/null +++ b/docs/deployment_and_security/deployment/aws/clean.md @@ -0,0 +1,35 @@ +# Clean + +The following procedure explains how to delete the platform. The full procedure takes around 45m to 1h to be completed. +To clean up **YData Fabric**, you will need to delete the CloudFormation stack and remove the subscription. + +Please take in consideration that this will delete **everything associated with the installation**. + +## Deleting the stacks +- Go to the regions where the product is installed +- Go to the *CloudFormation* service +- Select the *ydata stack* +- Click in the **Delete** button + +![delete stack](../../../assets/deployment_security/aws/delete_stack.png){: style="width:65%"} + +- Select the Extension stack and click in the **Delete** button. + +!!! Note + + This will disable the extension. If you are using this extension for any other project, please do not delete this stack. + +![EKS cluster delete](../../../assets/deployment_security/aws/eks_cluster_delete.png){: style="width:65%"} + +## Deleting the subscription +- Go to the ^^[**AWS Marketplace Subscriptions](https://console.aws.amazon.com/marketplace/home?region=eu-west-1)^^** → Manage subscriptions +- Click the *YData product* + +![ydata product](../../../assets/deployment_security/aws/ydata_subscription.png){: style="width:65%"} + +- **Actions → Cancel** subscription +- Click the checkbox and click *Yes*, cancel subscription + +![cancel](../../../assets/deployment_security/aws/cancel_subscription.png){: style="width:65%"} + +Following the above steps completes the process of deleting YData Fabric from your AWS Cloud instance. diff --git a/docs/deployment_and_security/deployment/aws/deploy.md b/docs/deployment_and_security/deployment/aws/deploy.md new file mode 100644 index 00000000..daf45f21 --- /dev/null +++ b/docs/deployment_and_security/deployment/aws/deploy.md @@ -0,0 +1,87 @@ +# Deploy + +## Installation process +The following procedure explains how to install the platform using the CloudFormation template and how to connect +to the platform after the installation. The full procedure takes around 45m to 1h to be completed. +In order to install the platform in your account, the user must have basic knowledge with the used tools, such as CloudFormation, +Route53 and Cognito. + +### 1. Configure the product + +!!! Note "Make sure that you comply with the pre-flight checks" + + You can check the ^^[prerequisites and pre-deploy checks](pre_deploy_checklist.md)^^. + +Start with the basic configuration for the app installation: + +- Ensure you are in the right region. +- Choose the stack name *"ydata-platform"* is the default name +![default stack](../../../assets/deployment_security/aws/aws_stack_default.png){: style="width:75%"} + +#### Network +Define your network configurations to access the platform. +Using the `ACM Certificate ARN` **OR** the `Hosted Zone ID` and the `Domain` chosen from the preflight checklist, +fill up the following parameters: +![default network parameters](../../../assets/deployment_security/aws/network_default_parameters.png){: style="width:75%"} + +#### OAuth +Define how your users will authenticate in the platform (you can use multiple providers). +![auth providers](../../../assets/deployment_security/aws/auth_providers.png){: style="width:55%"} + +#### Analytics +You can opt for allowing or not the collection of metrics in order to help us understand how users interact with the product. +No user data is collected at any point. +You can find our privacy policy at ^^[ydata.ai/privacy](https://ydata.ai/privacy)^^. +![auth providers](../../../assets/deployment_security/aws/allow_analytics.png){: style="width:70%"} + +#### Bastion host +A bastion host is created and used to give closer support to the users. +The bastion host is only accessible on user demand, giving us access to EC2 setting an SG ingress rule. +Set it to *"Allow"* to have it available. +More information [here](bastion_host.md). + +#### Create + +- Check the “**I acknowledge that AWS CloudFormation might create IAM resources with custom names.”** +- Click **Create Stack** + +![Acknowledge cloud formation](../../../assets/deployment_security/aws/aknowledge_cloud_formation.png){: style="width:70%"} + +### 2. Following the installation process + +Now we can follow the step-by-step for the installation of ^^[YData Fabric](https://ydata.ai/products/fabric)^^. + +- Click the “Create” button, the installation of the platform will start: + +![Init platform install](../../../assets/deployment_security/aws/init_platform_install.png){: style="width:40%"} + +The process will take approximately 45-60 minutes. + +- If the installation process occurs without any issues, you will see the **CREATE_COMPLETE** status in the stack: + +![Install completed](../../../assets/deployment_security/aws/install_completed.png){: style="width:40%"} + +- If any error occur during installation, please open a support case at ^^[support.ydata.ai](https://support.ydata.ai)^^. + +### 3. Post installation configuration + +#### DNS Configuration +If you have your domain registered in Route53, you can check the CF Outputs, and click the domain name to access the +platform: + +![DNS Configuration](../../../assets/deployment_security/aws/dns_configuration.png){: style="width:40%"} + +If you are using another DNS provider or a Route53 in another account, you will need to create a CNAME record pointing +to the ALB endpoint (ALBDNSName). As an example: +`CNAME → ydata-alb-xxxxxxxxx.eu-west-1.elb.amazonaws.com` + +## 4. Connecting to the platform +To connect the platform, **please allow 20-30m so the platform is completed initialised** and access using the URL +displayed in the CF Outputs. +For the login process, if you choose a customer custom login provider, you need to ensure that the users are created. + +Otherwise, you will need to create the users in the Cognito generated by the CloudFormation stack. + +More information under can be found at ^^[Login providers](../login_support/login_providers.md)^^. + +🚀 Congratulations you are now ready to start exploring your data with **YData Fabric**! diff --git a/docs/deployment_and_security/deployment/aws/instance_types.md b/docs/deployment_and_security/deployment/aws/instance_types.md new file mode 100644 index 00000000..0e188273 --- /dev/null +++ b/docs/deployment_and_security/deployment/aws/instance_types.md @@ -0,0 +1,32 @@ +# Instance types + +| Name | ID | Supported | System Pool | CPU MIcro Pool | CPU Small Pool | CPU Medium Pool | CPU Large Pool | CPU Compute Micro Pool | GPU MIcro Pool | GPU Compute Micro Pool | Bastion Host | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| N. Virginia | us-east-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Ohio | us-east-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| N. California | us-west-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Oregon | us-west-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Cape Town | af-south-1 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano | +| Melbourne | ap-southeast-4 | 🔴 | - | - | - | - | - | - | - | - | - | +| Hong Kong | ap-east-1 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano | +| Hyderabad | ap-south-2 | 🔴 | - | - | - | - | - | - | - | - | - | +| Jakarta | ap-southeast-3 | 🔴 | - | - | - | - | - | - | - | - | - | +| Mumbai | ap-south-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3a.nano | +| Osaka | ap-northeast-3 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano | +| Seoul | ap-northeast-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Singapore | ap-southeast-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Sydney | ap-southeast-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Tokyo | ap-northeast-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Canada Central | ca-central-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Frankfurt | eu-central-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Ireland | eu-west-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| London | eu-west-2 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g3.4xlarge | t3a.nano | +| Milan | eu-south-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3a.nano | +| Paris | eu-west-3 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3a.nano | +| Spain | eu-south-2 | 🔴 | - | - | - | - | - | - | - | - | - | +| Stockholm | eu-north-1 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano | +| Zurich | eu-central-2 | 🔴 | - | - | - | - | - | - | - | - | - | +| Bahrain | me-south-1 | ✅ | t3.2xlarge | t3.large | t3.xlarge | t3.2xlarge | m5.4xlarge | r5.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3.nano | +| UAE | me-central-1 | 🔴 | - | - | - | - | - | - | - | - | - | +| Tel Aviv | il-central-1 | 🔴 | - | - | - | - | - | - | - | - | - | +| São Paulo | sa-east-1 | ✅ | t3a.2xlarge | t3a.large | t3a.xlarge | t3a.2xlarge | m5a.4xlarge | r5a.4xlarge | g4dn.xlarge | g4dn.2xlarge | t3a.nano | diff --git a/docs/deployment_and_security/deployment/aws/pre_deploy_checklist.md b/docs/deployment_and_security/deployment/aws/pre_deploy_checklist.md new file mode 100644 index 00000000..1263d86f --- /dev/null +++ b/docs/deployment_and_security/deployment/aws/pre_deploy_checklist.md @@ -0,0 +1,177 @@ +# Checklist and Prerequisites + +Deploying [YData Fabric in the AWS cloud](https://aws.amazon.com/marketplace/pp/prodview-hgrqd5lqnqblm?sr=0-1&ref_=beagle&applicationId=AWSMPContessa) offers a scalable and efficient solution for managing and generating synthetic data. AWS provides a robust +infrastructure that ensures high availability, security, and performance, making it an ideal platform for **YData Fabric**. + +This cloud deployment allows for rapid scaling of resources to meet varying workloads, ensuring optimal performance and cost-efficiency. + +With AWS's comprehensive security features, including data encryption, network firewalls, and identity management, +your synthetic data and models are protected against unauthorized access and threats. +Additionally, AWS's global infrastructure allows for the deployment of YData Fabric in multiple regions, +ensuring low latency and high availability for users worldwide. + +!!! Note "Prerequisites" + + If you don't have an AWS account, create a ^^[free account](https://aws.amazon.com/)^^ before you begin. + +## Basic Configuration + +- **Stack name:** The name of the CloudFormation stack +- **Location**: where to install the platform and create the resources. You can check the available supported regions here: +- **Available regions: ** You can find the [aws regions where YData Fabric is available here](regions.md). + +## Permissions + +Check and add (if needed) the necessary permissions to the account and region where the platform will be installed. + +- Go to Identity and Access Management (IAM) +- Select your user or role used for deployment +- Under the permissions tab, check if you have the following permissions: + - AdministratorAccess + +**this will be updated in the future with only the necessary permissions to create and access the application.* + +*You can find [AWS official documentation here](https://docs.aws.amazon.com/iam/).* + +## Service Linked Roles +During the deployment all the required Service-Linked Roles are created by AWS by default with the **exception of the EKS +Service-Linked Role**. + +Please go to IAM → Roles +Verify that the following Service-Linked role exists in IAM: + +- `AWSServiceRoleForAmazonEKS` +![AWS IAM Role config](../../../assets/deployment_security/aws/aws_iam_role.png) + +Otherwise, please create the missing service linked role: + +- Click “Create role” +- Choose AWS service and EKS: + +![AWS EKS Service](../../../assets/deployment_security/aws/aws_eks_service.png) + +- Click “Next” → “Next” +- Click “Create role” + +*You can find ^^[AWS official documentation for service-linked roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html).*^^ + +## Quotas + +Check and set (if needed) new quotas for the region where the application will be installed. + +- Go to **Service Quotas** (ensure that you are in the right region). +- Select **AWS Services** → **Amazon Elastic Compute Cloud (Amazon EC2)** +- Check for the following quota limits: + +| Quota | Minimum | Recommended | +| --- | -- | --- | +| Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances | 50¹ | 100² | +| Running On-Demand G and VT instances | 0¹ | 20² | + + +1. *These limits are the required only for the installation of the platform. Usage is limited.* +2. *Each limit will depend on the platform usage and each client requirements.* + + +If needed, request for a new limit to the AWS support team. ^^[More on available instance types can be found here](instance_types.md)^^. + +## Network configuration + +Choose how you want to connect to the platform. + +The parameters below will be used during the deployment process. + +### DNS Configuration: + +In AWS, you will connect the platform providing your own DNS custom domain, for example: `platform.ydata.ai`. +For that, a registered domain is necessary. + +### **Domain Name** and **Route53 Hosted Zone ID** + +If you have your domain registered in Route53, you can pass the **Route53 Hosted Zone ID** and the **Domain Name**, +and the CloudFormation template will create an ACM certificate and a Route53 record pointing to the ALB used to connect the +platform. So no steps are required before or after the installation. + +### **Domain Name** and ACM Certificate ARN + +Otherwise, if you have your domain registered in another provider or in a route53 in another account, you will need to do one +of the following steps: + +=== "**Create the certificate on ACM and validate it**" + +| Request public certificate | Certificate granted | +|----------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------| +| ![Select an IDE](../../../assets/deployment_security/aws/aws_acm_certificate_1.png){: style="width:90%"} | ![Python or R](../../../assets/deployment_security/aws/aws_acm_certificate_2.png){: style="width:90%"} | + +After the certificate is requested, copy the CNAME value and name, and create the record in your DNS +provider so the certificate can be validated. + +=== "**Import the certificate to ACM**" + +| Request public certificate | Certificate granted | +|-----------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------| +| ![Select an IDE](../../../assets/deployment_security/aws/aws_import_acm_certificate.png){: style="width:90%"} | ![Python or R](../../../assets/deployment_security/aws/aws_import_acm_certificate_2.png){: style="width:90%"} | + +After the certificate is imported, ensure the certificate is validated. + +After the installation, you will need to create another CNAME record pointing to the ALB endpoint, available in the CF Outputs. + +For example: +`CNAME → ydata-alb-xxxxxxxxx.eu-west-1.elb.amazonaws.com` +![Route 53 example](../../../assets/deployment_security/aws/aws_route53_ex.png){: style="width:70%"} + +## Login Provider +In AWS you can use multiple providers to connect to the platform. +During the parameter section you can choose to create a Cognito or to use one on your own: +![Choose login provider](../../../assets/deployment_security/aws/cognito_auth.png){: style="width:90%"} + +Setting this to *True*, **unless you want to use a custom one**, you don’t need to specify any other parameters under the +*OAuth Configuration*. + +???- Warning "You can only have one Cognito" + + You can only choose one Cognito: + + - The created during the platform installation. + - One created by you, where you need to pass the credentials parameters. + + If both are set, the provided parameters will be ignored and the one created during installation will be used. + +???- Warning "Some regions do not support Cognito" + + This is not currently supported for some regions! + For this regions you will need to use the region specific template and pass your own custom oauth configuration! + + Check ^^[regions information here](regions.md)^^. + +You can log in to our app currently using the following providers - at **least one is required**, but you can choose multiple ones: + +- Google +- Microsoft +- Cognito (you own or the default created during installation) +- GitHub + +More detailed ^^[instructions for each login provider can be found here](../login_support/login_providers.md)^^. +If you required another authentication method, please fill up a support case at ^^[support.ydata.ai](https://support.ydata.ai/)^^ + +After configuring your login provider, please save the values. This values will be used during the deployment process. + +## AWSQS::EKS::Cluster +YData Fabric relies on an extension to configure the EKS cluster, so you will need to activate this extension before +proceeding to the installation. + +To activate the extension, you can use our *^^[CloudFormation template](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://ydata-public-marketplace.s3.eu-central-1.amazonaws.com/1.15.22_2.33.3/awsqs-eks-cluster.yaml&stackName=AWSQSEKSClusterExtension)*^^. +This will open the CloudFormation template ready to create. Choose the correct region on the top. And create the stack! + +To check that the extension in properly installed, go to **CloudFormation → Registry → Activated extensions** - filter by *“Privately registered”* +as depicted in the image below: +![Cloud formation extensions](../../../assets/deployment_security/aws/cloud_formation_registry.png){: style="width:75%"} + +If the resource is listed, the extension in activated. + +!!! Note + + This is extension is mandatory for the create, update and delete of the cluster. After activating this extension, + please do not make changes or delete the stack. + +As soon as the above steps are all completed, you are ^^[ready to start the deployment](deploy.md)^^. diff --git a/docs/deployment_and_security/deployment/aws/regions.md b/docs/deployment_and_security/deployment/aws/regions.md new file mode 100644 index 00000000..70eb4a27 --- /dev/null +++ b/docs/deployment_and_security/deployment/aws/regions.md @@ -0,0 +1,34 @@ +# 🌐 Regions + +**Use the Cognito Unsupported Regions template*. + +| Name | ID | Supported | Notes | +| --- | --- | --- | --- | +| N. Virginia | us-east-1 | ✅ | ➖ | +| Ohio | us-east-2 | ✅ | ➖ | +| N. California | us-west-1 | ✅ | ➖ | +| Oregon | us-west-2 | ✅ | ➖ | +| Cape Town | af-south-1 | ✅ | Cognito is not supported at the moment* | +| Melbourne | ap-southeast-4 | 🔴 | No GPU machine types available at the moment | +| Hong Kong | ap-east-1 | ✅ | Cognito is not supported at the moment* | +| Hyderabad | ap-south-2 | 🔴 | No GPU machine types available at the moment | +| Jakarta | ap-southeast-3 | 🔴 | No GPU machine types available at the moment | +| Mumbai | ap-south-1 | ✅ | ➖ | +| Osaka | ap-northeast-3 | ✅ | ➖ | +| Seoul | ap-northeast-2 | ✅ | ➖ | +| Singapore | ap-southeast-1 | ✅ | ➖ | +| Sydney | ap-southeast-2 | ✅ | ➖ | +| Tokyo | ap-northeast-1 | ✅ | ➖ | +| Canada Central | ca-central-1 | ✅ | ➖ | +| Frankfurt | eu-central-1 | ✅ | ➖ | +| Ireland | eu-west-1 | ✅ | ➖ | +| London | eu-west-2 | ✅ | ➖ | +| Milan | eu-south-1 | ✅ | ➖ | +| Paris | eu-west-3 | ✅ | ➖ | +| Spain | eu-south-2 | 🔴 | No GPU machine types available at the moment | +| Stockholm | eu-north-1 | ✅ | ➖ | +| Zurich | eu-central-2 | 🔴 | No GPU machine types available at the moment | +| Bahrain | me-south-1 | ✅ | ➖ | +| UAE | me-central-1 | 🔴 | No GPU machine types available at the moment | +| Tel Aviv | il-central-1 | 🔴 | No GPU machine types available at the moment | +| São Paulo | sa-east-1 | ✅ | ➖ | diff --git a/docs/deployment_and_security/deployment/aws/update.md b/docs/deployment_and_security/deployment/aws/update.md new file mode 100644 index 00000000..c7d42993 --- /dev/null +++ b/docs/deployment_and_security/deployment/aws/update.md @@ -0,0 +1,60 @@ +# Update Fabric + +**YData** is committed to providing our users with cutting-edge tools and features to enhance their data management and synthetic data generation capabilities. +Our solution updates policy is designed to ensure that YData Fabric remains at the forefront of technological advancements while maintaining the highest standards +of reliability, security, and user satisfaction. + +**Key Aspects of Our Update Policy** + +- **Regular Updates:** We release regular updates that include new features, performance improvements, and bug fixes. These updates are aimed at enhancing +the overall functionality and user experience of YData Fabric. +- **User Feedback Integration:** We actively seek and incorporate feedback from our user community. This ensures that our updates address real-world challenges +and meet the evolving needs of our users. +- **Seamless Deployment:** Updates are designed to be deployed seamlessly with minimal disruption to ongoing operations. +Our team provides detailed documentation and support to facilitate smooth transitions. +- **Security Enhancements:** We prioritize the security of our platform. Each update undergoes rigorous testing to ensure that it enhances the security +posture of YData Fabric without introducing vulnerabilities. +- **Compatibility and Compliance:** Updates are developed to ensure compatibility with existing systems and compliance with industry standards and regulations, +safeguarding the integrity and continuity of user operations. + +By adhering to this policy, **YData** ensures that users consistently benefit from the latest advancements in data technology, reinforcing our commitment +to innovation and excellence in the field of data science and synthetic data generation. + +All updates to Fabric are user/organization triggered and by following the next steps to update your **CloudFormation stack**. + +## 1. Get the most recent version + +- Go to the ^^[**AWS Marketplace Subscriptions](https://console.aws.amazon.com/marketplace/home?region=eu-west-1)^^** → Manage subscriptions +- Click the **YData Fabric** subscription + +![YData Configuration](../../../assets/deployment_security/aws/ydata_subscription.png){: style="width:40%"} + +- Click Launch more software. + +![launch more software](../../../assets/deployment_security/aws/launch_more_software.png){: style="width:50%"} + +- Check for new versions and click **Continue to Launch**. At this stage you will find the link for the new version. + +![update launch](../../../assets/deployment_security/aws/update_launch.png){: style="width:60%"} + +Click the deployment template associated with your installation. + +- Here you will have the new template URL. Copy the link as per the image below: + +![copy link](../../../assets/deployment_security/aws/copy_link.png){: style="width:45%"} + +- Go to the deployed *CloudFormation stack* and clink in *"Update"* button. +- Choose *“Replace current template”* and provide the new stack URL. + +![copy link](../../../assets/deployment_security/aws/cloud_formation_new_link.png){: style="width:45%"} + +- For the parameters, use the same parameters or change if needed. Click **Next → Next → Submit** + +![submit cloud formation](../../../assets/deployment_security/aws/submit_cloud_formation.png){: style="width:75%"} + +2. Following the installation process + +Now you can follow the ^^[installation process](deploy.md)^^. +Different from the initial deploy, the update process will only take approximately *15-60 minutes* depending on the update complexity. + +🚀 Congratulations you have now the latest version of **YData Fabric**! diff --git a/docs/deployment_and_security/deployment/azure.md b/docs/deployment_and_security/deployment/azure.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/deployment_and_security/deployment/google.md b/docs/deployment_and_security/deployment/google.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/deployment_and_security/deployment/login_support/login_providers.md b/docs/deployment_and_security/deployment/login_support/login_providers.md new file mode 100644 index 00000000..44fa9531 --- /dev/null +++ b/docs/deployment_and_security/deployment/login_support/login_providers.md @@ -0,0 +1,160 @@ +# Login Providers + +**YData Fabric** offers a flexible and secure authentication system, allowing users to log in using a +variety of trusted identity providers. This technical documentation provides a comprehensive guide to +configuring and managing login providers for **YData Fabric**, including Google, Microsoft, and Amazon Cognito. +By leveraging these providers, users can benefit from seamless and secure access to **YData Fabric**, ensuring a smooth and efficient user experience. + +## Google +1. Open the ^^[Google Cloud Console](https://console.cloud.google.com/)^^. +2. At the top-left, click **Menu**>**APIs & Services**>**Credentials**. +3. Click **Create Credentials**>**OAuth client ID**. +4. Click **Application type**>**Web application**. +5. In the "Name" field, type a name for the credential. This name is only shown in the Cloud Console. +6. Leave the “**Authorized JavaScript origins”** empty**.** + Add a new “**Authorized redirect URIs**” with the platform endpoint with a suffix `*/dex/callback*` + For the provided example: + + === "If you are using the **DNS Public Endpoint**" + + ![Google custom domain dns](../../../assets/deployment_security/login_support/google_custom_domain_dns.png){: style="width:40%"} + + === "Or, if you are using the **DNS Custom Domain**" + + ![Public dns](../../../assets/deployment_security/login_support/google_public_dns.png){: style="width:40%"} + +7. Click “Create” +8. Save the following credentials: + - a. Client ID + + The Client ID for the Web Application + - b. Client Secret + + The Client Secret for the Web Application + - c. APP Hosted domain + + `Google supports whitelisting allowed domains when using G Suite` + For example, for one company with the emails like person@example.com, the APP Hosted domain is example.com + +9. Use the credentials as inputs for **YData Fabric**. + +You can find more details in ^^[Google's official documentation](https://developers.google.com/workspace/guides/create-credentials#oauth-client-id)^^. + +## Microsoft +1. Open the ^^[Azure Portal](https://portal.azure.com/)^^ +2. Go to “Entra ID” +3. Click “App registrations” +4. Click “New registration” +5. Choose a name +6. For the supported account types, choose the most appropriated choice for you. +7. For the Redirect URI, choose “Web”, and fill with the platform endpoint with a suffix `*/dex/callback*`. +For the provided example: + + === "If you are using the **DNS Public Endpoint**" + + ![Google custom domain dns](../../../assets/deployment_security/login_support/google_custom_domain_dns.png){: style="width:40%"} + + === "Or, if you are using the **DNS Custom Domain**" + + ![Public dns](../../../assets/deployment_security/login_support/google_public_dns.png){: style="width:40%"} + +8. Click “Register” +9. Go to “Certificates & Secrets”, generate a new secret and save the value **(not the secret id)**. Please choose a large expiration date. This value cannot be changed after the installation of the platform. +10. Go to “Overview” and save the following credentials: + - a. Client ID + + The Application (client) ID + + - b. Client Secret + + The secret generated in step 9 **(not the secret id)**. + + - c. Tenant ID + + The Directory (tenant) ID + +11. Use the credentials as inputs for **YData Fabric**. + +### Consent workflow +The admin consent workflow is necessary to configure, so you can access the platform using the app registered above. + +1. Open the ^^[Azure Portal](https://portal.azure.com/)^^ +2. Go to “Azure Active Directory” +3. Click "Enterprise applications” +4. Open the “Consent and permissions” page → “User consent settings” +5. Check with the AD administrator if an administrator is required to login to the app, or if all users can consent for the apps. +![Consent flow](../../../assets/deployment_security/login_support/msft_consent.png){: style="width:75%"} + +### Give access only to a set of users and/or groups +1. In order to give access only to a set of users or groups, open your app and click the link “Managed application in local directory” on the right side: +![Access group/users](../../../assets/deployment_security/login_support/msft_access_users_groups.png){: style="width:100%"} +2. Then, click in “Properties” and enable the **“Assignment required”** +![assigment required](../../../assets/deployment_security/login_support/msft_assignment_required.png){: style="width:50%"} +3. To add users and/or groups, go to *“Users and Groups”* and click *“Add user/group”*. + +With the above steps, only the users and groups listed here can access YData Fabric. For more information check Microsoft's official documentation +for ^^[Microsoft identy platform](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app)^^ and +^^[Microsoft Entra](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/configure-admin-consent-workflow)^^. + + +## AWS Cognito +1. Go to the ^^[Amazon Cognito console](https://console.aws.amazon.com/cognito/home)^^. If prompted, enter your *AWS credentials*. +2. Choose **User Pools**. Create a new *User Pool*. +3. For the *“Configure security requirements”*, *“Configure sign-up experience”* and *“Configure message delivery”* tabs are up to your choices +or leave as the default. +4. In the “**Integrate your app**” please set the attributes as the following: + 1. “**User Pool Name**” - a name of your choice + 2. Tick the “**Use the Cognito Hosted UI**” check box. + 3. “**Domain type**”, you can use a cognito or a custom domain. + 4. “**Initial app client**” choose “**Public client**” and set a “**App client name**” + 5. For “**Client secret**”, choose “**Generate a client secret**” + 6. In the “**Allowed callback URLs**”, set your callback URL with the platform endpoint with a suffix `*/dex/callback*` + For the provided example: + === "If you are using the **DNS Public Endpoint**" + + ![Google custom domain dns](../../../assets/deployment_security/login_support/cognito_public_dns.png){: style="width:60%"} + + === "Or, if you are using the **DNS Custom Domain**" + + ![Public dns](../../../assets/deployment_security/login_support/cognito_private_dns.png){: style="width:60%"} + + 7. In the “**Advanced app client settings**” → “**Authentication flows**” step, choose “**ALLOW_USER_PASSWORD_AUTH**” + 8. For the “**OpenID Connect scopes**” choose: “Email”, “OpenID” and “Profile”. +5. Review your settings, and **“Create User Pool”.** +6. Click your new user pool, go to the “**App integration**” tab and “**App clients and analytics”.** +7. Copy and save the **Client ID** and **Client secret**. +8. For the “**Issuer URL**”, get your URL by going to https://cognito-idp.[region].amazonaws.com/[user_pool_id]/.well-known/openid-configuration +And copy and save the *"issuer* URL. +9. Use these credentials as inputs for *YData Fabric*. + +### Adding new users +1. Go to the Cognito service. +2. Click the YData platform Cognito user pool. +3. Go to the Users tab +4. Click **Create user** +5. Create the users: +![Cognito create users](../../../assets/deployment_security/login_support/cognito_create_users.png){: style="width:60%"} +6. The user will receive an e-mail with the temporary credentials. + +For more information check ^^[Amazon's Cognito official documentation on user pools](https://docs.aws.amazon.com/cognito/latest/developerguide/getting-started-user-pools.html)^ +and ^^[user pool app client](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-settings-client-apps.html)^^. + +## Github +1. Go to the ^^[GitHub OAuth Application](https://github.com/settings/applications/new)^^ page. If prompted, enter your GitHub credentials. +2. For the “**Application Name**”, choose anything. +3. For the “**Homepage URL**” and “**Authorization callback URL**”, fill with the platform endpoint and platform endpoint with a suffix +`*/dex/callback`* correspondingly*.* +For the provided example: +=== "If you are using the **DNS Public Endpoint**" + + ![Google custom domain dns](../../../assets/deployment_security/login_support/github_public_dns.png){: style="width:40%"} + +=== "Or, if you are using the **DNS Custom Domain**" + + ![Public dns](../../../assets/deployment_security/login_support/github_custom_dns.png){: style="width:40%"} +4. Open your new APP and generate a new secret +5. Save the **Client ID** and **Client secret** +6. For the org, use your GitHub organization name. + +Finally, use these credentials as inputs for to login **YData Fabric**. +For more information check ^^[GitHub's official login documentation](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app)^^. diff --git a/docs/deployment_and_security/index.md b/docs/deployment_and_security/index.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/deployment_and_security/security/index.md b/docs/deployment_and_security/security/index.md new file mode 100644 index 00000000..3ce5ef5a --- /dev/null +++ b/docs/deployment_and_security/security/index.md @@ -0,0 +1,147 @@ +# Security + +This section describes YData’s security measures to provide a best-in-class experience for its customers, ensuring not only a good product and +service but also risk management and compliance. + +Visit ^^[YData's Trust page](https://trust.ydata.ai)^^ to check all the Policies, Controls and Monitoring in place. + +## Hosting security +**YData** is not a cloud service provider, however, we use providers which are hosted on their data centers, such as *Google*, *Microsoft* and *Amazon Web Services*, +when the setup is not made on the customer premises. They are leading cloud infrastructure providers with top-class safety standards. +They are able to respond quickly to both operational and security, including well-defined change management policies and procedures to determine when and how +change occurs. + +### Clouds compliance standards + +=== "Google" + + - CSA + - ISO 27018 + - SOC 3 + - ISO 27001 + - SOC 1 + - ISO 27017 + - SOC 2 + +=== "AWS" + + - CSA + - ISO 27017 + - SOC 2 + - ISO 9001 + - ISO 27018 + - SOC 3 + - ISO 27001 + - SOC 1 + +=== "Microsoft Azure" + + - CSA + - ISO 27017 + - ISO 22301 + - SOC + - ISO 9001 + - ISO 27018 + - ISO 20000-1 + - ISO 27001 + - ISO 27701 + - WCAG + +Both physical access perimeters and entry points are strictly controlled by professional security personnel. Authorized personnel must pass a minimum of two-step verification +to gain access to the authorized center floors. + +## Corporate security + +**YData** has applied internal security policies that are in line with the industry's ISO 27001 and SOC 2. We are regularly training our employees in safety +and privacy awareness, which protects technical and non-technical roles. Training materials are developed for individual roles so that employees can fulfill +their responsibilities appropriately. + +- Two-step verification for all services is enforced +- Encrypted hard drives of our devices is enforced +- Hard password requirements and rotation is enforced + +## Verification and Access Management + +Users can log in via a secured Authentication provider, such as Security Assurance Markup Language, Microsoft Active Directory, Google Sign In or OpenID services. +All requests to any of YData’s APIs must be approved. Data writing requests require at least reporting access as well as an API key. Data reading requests +require full user access as well as application keys. These keys act as carrier tokens to allow access to the YData service functionality. We also use Auth0 +in user identification. Auth0 can never save a password because the password is encrypted when the user logs in, and compares with AuthO's encrypted password +to see if they are using the correct password. + +The user can change and save the password as they wish. The user can use all types of characters to strengthen his password. + +## Certificate Management & Communications +All certificates are generated and used inside the Kubernetes cluster, using cert-manager. Exceptions for cloud providers for specific certificates +and described below. +Every component inside the cluster uses its own certificate, sharing the same issuer so all the components exchange encrypted communication between them. + +=== "AWS" + + "During the deployment, a certificate is requested and provisioned by *Let’s Encrypt* to the specified domain." + +=== "Microsoft Azure" + + "The public certificate is generated using the AWS Certificate Manager service." + +## Protection of Customer Data + +User uploaded information or data will be considered confidential, which is stored in encrypted form, separate from other networks, including the public network +if available. Data for a limited time without user request, not allowed to come out. +All data transmitted layer protection (TSL) and HTTP sent by users protected using Strike Transport Security (HSTS). The application is usable if encrypted +communication is compromised. +User uploaded data is not transferred from one data center to another. Encryption is used in many places to protect customer information, such as: +IS-266 with encryption at rest, incomplete encryption (PGP) for system backups, KMS-based protection for privacy protection, and GPG encryption. +Users can use the data stored for business or administrative purposes, but they have to go through many security levels, including multifactor authentication +(MFA). + +## Secure Build Materials (SBOM) +To enhance transparency and facilitate security assessments, we provide access to Secure Build Materials (SBOM) for our products and services. +SBOM files offer detailed insights into the components, dependencies, and associated vulnerabilities within our software stack. These files enable stakeholders, +including customers, auditors, and security researchers, to evaluate the security posture of our offerings comprehensively. +For access to SBOM files and additional security-related information, please visit our Security Resources page at: +[Find more information here.](security_building_materials.md) + +## Certification, Attestation and Framework +YData uses a frontend framework React (originally maintained by Facebook) which combines the use of unique user tokens to protect your users against +common threats such as cross-site scripting (CSS / XSS) and cross-site request fraud (CSRF / XSRF). This makes it impossible for the user to access data +from another user's account. + +## Laws and Regulations +The cloud service providers used by YData are compatible with the General Data Protection Resolution (GDPR). +GDPR is working to expand its products, methods and processes to fulfill its responsibilities as a data processor. +YData's security and privacy teams have established a vendor management program that determines the need for YData to be approved when it involves third parties +or external vendors. Our security team recognizes that the company’s information resources and vendor reliance are critical to our continued activities +and service delivery. These spaces are designed to evaluate technical, physical and administrative controls and ensure that it meets the expectations of it and +its customers. +It is a monitoring service for infrastructure and applications. Our CCPA compliance process may provide additions so that our customers can fulfill their +obligations under the CCPA if there is access to personal data, while we make no plans to transfer, process, use or store personal information. + +## Data Security +- No data ever leaves the costumer client cloud. +- All the data is stored using cloud specific services to ensure security, privacy and compliance with YData’s customers requirements. + +## Data Encryption +The way YData’s customers communicate with the servers is through SSL / TLS connections, which are encrypted. +YData protects the servers where YData Fabric is deployed from DDOS, SQL injection and other fraudulent activities. +If one wants to interrupt the data transfer, one can only see a mixture of some characters, which is not possible to decrypt. +All data in databases is encrypted with industry standard AES-256. + +## API Security +To use the API the user needs to have a *JWT* *token* that is automatically generated by Fabric for a specific user. The *token* is signed and encrypted +using a random key created during the deployment and only known by the service responsible for its provisioning. + +## Availability and disaster recovery +When using one of the cloud providers, the data stored in the bucket and database is distributed and copied to different servers. +If a bucket or database fails, it is usually recovered from a different server without targeting other users.Databases are backed up on a daily basis and +can be restored if the software or server fails significantly. Backups are stored in various European and North American data centers (depending on the customer +location) for extra protection. +It is not possible for YData to recover individual customer information - if you delete something in your account, it will be permanently deleted, and we will +not be able to recover it. + +## Monitoring +The functionality of our applications and databases is monitored 24/7 through in-built monitoring tools provided by Google, Azure and Amazon Web Services. Internal errors or failures of our various integrations trigger logins and notifications. This usually helps us to identify the problem very quickly and remedy the situation. + +## Full disclosure policy +If something serious happens and your data is damaged as required by GDPR, we will disclose in full (such as a data breach). +Transparency is important to us and we will provide you with all the necessary information to properly assess the situation and potential impact. +So far no customer data has been compromised and we aim to keep it that way. diff --git a/docs/deployment_and_security/security/security_building_materials.md b/docs/deployment_and_security/security/security_building_materials.md new file mode 100644 index 00000000..f858f61d --- /dev/null +++ b/docs/deployment_and_security/security/security_building_materials.md @@ -0,0 +1,78 @@ +# Secure Build Materials (SBOM) + +To enhance transparency and facilitate security assessments, we provide access to Secure Build Materials (SBOM) for our products and services. + +SBOM files offer detailed insights into the components, dependencies, and associated vulnerabilities within our software stack. +These files enable stakeholders, including customers, auditors, and security researchers, to evaluate the security posture of our offerings comprehensively. + +### All files + +^^[https://s3.console.aws.amazon.com/s3/buckets/repos-sboms?region=eu-west-1&bucketType=general&tab=objects](https://mock-abronia-a151afe273f2.herokuapp.com/b?y=49q24eh264q64opgckrmaphg60qm6c9ncphjic9gc4p64db174r3iphicch2o8ji48t24q3keho76ehf5tpj6bj3dtn76rrcckn62trj5pgmqobqdtn2sorfdknn6cpfc9qm6qr5ehpiusj5e1nn6bbjc9nmqspve9imeqbfdoumat9detin6t1d64j62rbg7dh7aorbclq58ubgckumepbeclp62r16c5mn0erkc5h3qrr2d9im6t3j48======)^^ + +### Individual raw files + +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/authentication-service/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/authentication-service/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/aws-adapter/metering-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/aws-adapter/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/aws-adapter/quota-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/aws-asg-tags-lambda/command-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/aws-asg-tags-lambda/lambda-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/aws-asg-tags-lambda/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/azure-adapter/metering-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/azure-adapter/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/azure-adapter/quota-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/backoffice-console/command-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/backoffice-console/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/backoffice/api-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/backoffice/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/api-gateway/docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/datasource-controller/api-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/datasource-controller/manager-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/datasource-controller/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dex-theme/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dex-theme/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/dask-gateway-scheduler/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/dask-gateway-worker/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/h2oflow/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/h2oflow/gpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/jupyterlab_python/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/jupyterlab_python_community/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/jupyterlab_python_tensorflow/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/jupyterlab_python_torch/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/jupyterlab_r/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/jupyterlab_r/gpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/pipelines_python_tensorflow/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/pipelines_python_torch/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/pipelines_python_ydata/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/pipelines_ydata/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/visualcode/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/visualcode/gpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/visualcode_tensorflow/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/visualcode_torch/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/visualcode_ydata/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dashboard-app/package-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/gcp-adapter/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/gcp-adapter/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/laboratory-controller/api-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/laboratory-controller/manager-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/laboratory-controller/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/metering-service/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/metering-service/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/profile-controller/api-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/profile-controller/manager-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/profile-controller/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/quota-manager/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/quota-manager/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/static-content-server/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/static-content-server/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/synthesizer-controller/api-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/synthesizer-controller/manager-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/synthesizer-controller/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/uploader-service/docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/uploader-service/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/ydata-lib-platform-integration-tool/cpu-docker-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ +- ^^[https://repos-sboms.s3.eu-west-1.amazonaws.com/ydata-lib-platform-integration-tool/package-sbom.cyclonedx.json](https://repos-sboms.s3.eu-west-1.amazonaws.com/dockerfiles/ydata/cpu-docker-sbom.cyclonedx.json)^^ diff --git a/docs/integrations/index.md b/docs/integrations/index.md new file mode 100644 index 00000000..e839a2a7 --- /dev/null +++ b/docs/integrations/index.md @@ -0,0 +1,41 @@ +# Integrations + +Recognizing the modern enterprise data stack comprises a vast array of services and tools, +**YData Fabric is augmented by a growing ecosystem of partners and integrations**, +acting both upstream and downstream in the lifecycle of an AI project. + +The list below is a non-exhaustive compilation of MLOps, Data and Cloud Providers which smoothly integrate with Fabric: + +- [**DVC**](https://dvc.org/): Enhancing data versioning +- [**Databricks**](https://databricks.com/): Enhancing feature/data engineering before improving with YData + + - ^^[📚 Follow Databricks step-by-step tutorials](databricks)^^ + - ^^[👨‍💻 Check code example in YData Academy](https://github.com/ydataai/academy/tree/master/5%20-%20Integrations/databricks)^^ + +- [**Snowflake**](https://snowflake.com/): Enhancing feature/data engineering before improving with YData + + - ^^[📚 Follow Snowflake step-by-step tutorials](snowflake)^^ + - ^^[👨‍💻 Check code example in YData Academy](https://github.com/ydataai/academy/tree/master/5%20-%20Integrations/snowflake)^^ + +- [**H2O**](https://www.h2o.ai/blog/introducing-flow/): Framework available through code and ^^[Fabric Labs (H2O Flow)](../labs)^^ +- [**Algorithmia**](https://algorithmia.com/): Integration for easy model deployment + + - ^^[👨‍💻 Check code example in YData Academy](https://github.com/ydataai/academy/tree/master/5%20-%20Integrations/algorithmia)^^ + +- [**UbiOps**](https://ubiops.com/): Integration for easy model deployment + + - ^^[👨‍💻 Check code example in YData Academy](https://github.com/ydataai/academy/tree/master/5%20-%20Integrations/ubiops)^^ + +- [**Great Expectations**](https://greatexpectations.io/): Data profiling is integrated with Great Expectations +- [**Azure ML**](https://azure.microsoft.com/pt-pt/services/machine-learning/): Integration for easy model deployment + + - ^^[👨‍💻 Check code example in YData Academy](https://github.com/ydataai/academy/tree/master/5%20-%20Integrations/azure-ml)^^ + +- [**AWS SageMaker**](https://aws.amazon.com/sagemaker/): Integration for easy model deployment + + - ^^[👨‍💻 Check code example in YData Academy](https://github.com/ydataai/academy/tree/master/5%20-%20Integrations/aws-sagemaker)^^ + +- [**Google Vertex AI**](https://cloud.google.com/vertex-ai): Integration for easy model deployment + +!!! Note "Up-to-date examples" + 👉 For the **most up-to-date examples and ready-to-use recipes** of how to integrate with YData Fabric with some services above, check out the ^^[Integrations section of YData’s Academy](https://github.com/ydataai/academy/tree/master/9%20-%20integrations)^^. diff --git a/mkdocs.yml b/mkdocs.yml index 63d0a77e..f2c91ea8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -40,6 +40,7 @@ nav: - 'pipelines/index.md' - Concepts: 'pipelines/concepts.md' - Integrations: + - 'integrations/index.md' - Snowflake: - 'integrations/snowflake/integration_snowflake.md' - Databricks: @@ -74,6 +75,23 @@ nav: - TimeSeries: 'sdk/reference/api/synthesizers/timeseries.md' - MultiTable: 'sdk/reference/api/synthesizers/multitable.md' - Types: 'sdk/reference/api/common/types.md' + - Deployment & Security: + - 'deployment_and_security/index.md' + - Deployment: + - AWS: + - 🕒 Pre-deploy checklist: 'deployment_and_security/deployment/aws/pre_deploy_checklist.md' + - 🔌 Deploy: 'deployment_and_security/deployment/aws/deploy.md' + - 🔌 Updates: 'deployment_and_security/deployment/aws/update.md' + - 🌐 Regions: 'deployment_and_security/deployment/aws/regions.md' + - 💻 Instance types: 'deployment_and_security/deployment/aws/instance_types.md' + - 💻 Bastion host: 'deployment_and_security/deployment/aws/bastion_host.md' + - Azure: 'deployment_and_security/deployment/azure.md' + - Google: 'deployment_and_security/deployment/google.md' + - Login, support & monitoring: + - Login providers: 'deployment_and_security/deployment/login_support/login_providers.md' + - Security: + - 'deployment_and_security/security/index.md' + - Secure Build Materials (SBOM): 'deployment_and_security/security/security_building_materials.md' theme: