docs: add Azure deploy information. (#102)

* docs: add Azure deploy information. * fix(linting): code formatting --------- Co-authored-by: Fabiana Clemente <[email protected]> Co-authored-by: Azory YData Bot <[email protected]>
ydataai · Jun 18, 2024 · 7440d13 · 7440d13
1 parent 340a4f8
commit 7440d13
Show file tree

Hide file tree

Showing 32 changed files with 383 additions and 2 deletions.
diff --git a/docs/assets/deployment_security/azure/azure_activate_jit.png b/docs/assets/deployment_security/azure/azure_activate_jit.png
diff --git a/docs/assets/deployment_security/azure/azure_app_config.png b/docs/assets/deployment_security/azure/azure_app_config.png
diff --git a/docs/assets/deployment_security/azure/azure_authentication.png b/docs/assets/deployment_security/azure/azure_authentication.png
diff --git a/docs/assets/deployment_security/azure/azure_check_permissions.png b/docs/assets/deployment_security/azure/azure_check_permissions.png
diff --git a/docs/assets/deployment_security/azure/azure_check_quotas.png b/docs/assets/deployment_security/azure/azure_check_quotas.png
diff --git a/docs/assets/deployment_security/azure/azure_compute.png b/docs/assets/deployment_security/azure/azure_compute.png
diff --git a/docs/assets/deployment_security/azure/azure_config_route53.png b/docs/assets/deployment_security/azure/azure_config_route53.png
diff --git a/docs/assets/deployment_security/azure/azure_container.png b/docs/assets/deployment_security/azure/azure_container.png
diff --git a/docs/assets/deployment_security/azure/azure_contributor.png b/docs/assets/deployment_security/azure/azure_contributor.png
diff --git a/docs/assets/deployment_security/azure/azure_create.png b/docs/assets/deployment_security/azure/azure_create.png
diff --git a/docs/assets/deployment_security/azure/azure_create_route.png b/docs/assets/deployment_security/azure/azure_create_route.png
diff --git a/docs/assets/deployment_security/azure/azure_define_dns.png b/docs/assets/deployment_security/azure/azure_define_dns.png
diff --git a/docs/assets/deployment_security/azure/azure_delete_resource_group.png b/docs/assets/deployment_security/azure/azure_delete_resource_group.png
diff --git a/docs/assets/deployment_security/azure/azure_existing_ip.png b/docs/assets/deployment_security/azure/azure_existing_ip.png
diff --git a/docs/assets/deployment_security/azure/azure_licenses.png b/docs/assets/deployment_security/azure/azure_licenses.png
diff --git a/docs/assets/deployment_security/azure/azure_network_config.png b/docs/assets/deployment_security/azure/azure_network_config.png
diff --git a/docs/assets/deployment_security/azure/azure_public_ip.png b/docs/assets/deployment_security/azure/azure_public_ip.png
diff --git a/docs/assets/deployment_security/azure/azure_request_quota.png b/docs/assets/deployment_security/azure/azure_request_quota.png
diff --git a/docs/assets/deployment_security/azure/azure_role_assigment_preview.png b/docs/assets/deployment_security/azure/azure_role_assigment_preview.png
diff --git a/docs/assets/deployment_security/azure/azure_role_assignment.png b/docs/assets/deployment_security/azure/azure_role_assignment.png
diff --git a/docs/assets/deployment_security/azure/azure_selected_existing_ip.png b/docs/assets/deployment_security/azure/azure_selected_existing_ip.png
diff --git a/docs/assets/deployment_security/azure/azure_ydata_app.png b/docs/assets/deployment_security/azure/azure_ydata_app.png
diff --git a/docs/assets/deployment_security/azure/azure_ydata_parameters_outputs.png b/docs/assets/deployment_security/azure/azure_ydata_parameters_outputs.png
diff --git a/docs/assets/deployment_security/azure/azure_ydata_resourcegroup.png b/docs/assets/deployment_security/azure/azure_ydata_resourcegroup.png
diff --git a/docs/assets/deployment_security/azure/azure_ydata_resources.png b/docs/assets/deployment_security/azure/azure_ydata_resources.png
diff --git a/docs/deployment_and_security/deployment/aws/billing.md b/docs/deployment_and_security/deployment/aws/billing.md
@@ -0,0 +1,45 @@
+# Billing
+
+After the installation, the client will be billed for all the infrastructure costs plus the usage metrics describe in the offer.
+Using a usage-based pricing model you will only pay for what you use.
+The following metrics are calculated and sent to AWS in order to charge you at the current offer pricing:
+
+- CPU / Hour
+- Memory / Hour
+- GPU / Hour
+
+The following AWS services are mandatory for the platform to work and will be billed:
+
+- VPC
+- ACM
+- Secrets Manager
+- CloudWatch
+- EKS
+- EC2
+- EFS
+- RDS
+- Cognito
+- ECS
+- Lambda
+
+To check the infrastructure costs of the platform, you can use the AWS Cost Explorer and filter by the tag Environment = YData.
+This will aggregate all the resources deployed by the platform.
+
+## Cost Estimations
+
+**YData Fabric** final cost can be estimated following the logic of a usage-based plan since it depends on your users and data. The following table provides
+a guideline of how to compute the total cost for different usage scenarios based on the deployed infrastructure.
+
+| EKS Nodes | Instance Type | vCPUs | Memory (GBi) | GPUs | Number of instances | % Usage/ CPU/Hour | % Usage/ Memory/Hour | % Usage/ GPU/Hour | Cost AWS/Hour | Cost AWS/Day | Cost YData/Hour | Cost YData/Day |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| System | t3a.2xlarge | 8 | 32 | 0 | 2 | 20 | 20 | 0 | $0.30 | $14.44 | $0.38 | $9.22 |
+| CPU Micro (labs) | t3a.large | 2 | 8 | 0 | 1 | 40 | 40 | 0 | $0.08 | $1.80 | $0.10 | $2.30 |
+| CPU Small (labs) | t3a.xlarge | 4 | 16 | 0 | 1 | 20 | 20 | 0 | $0.15 | $3.61 | $0.10 | $2.30 |
+| CPU Medium (labs) | t3a.2xlarge | 8 | 32 | 0 | 0 | 0 | 0 | 0 | $0.30 | $0.00 | $0.00 | $0.00 |
+| CPU Large (labs) | m5a.4xlarge | 16 | 64 | 0 | 0 | 0 | 0 | 0 | $0.69 | $0.00 | $0.00 | $0.00 |
+| CPU Compute Micro (computing) | r5a.4xlarge | 16 | 128 | 0 | 1 | 20 | 20 | 0 | $0.90 | $21.70 | $0.64 | $15.36 |
+| GPU Micro (labs) | g4dn.xlarge | 4 | 16 | 1 | 0 | 0 | 0 | 0 | $0.53 | $0.00 | $0.00 | $0.00 |
+| GPU Compute Micro (computing) | g3.4xlarge | 16 | 122 | 1 | 0 | 0 | 0 | 0 | $1.14 | $0.00 | $0.00 | $0.00 |
+
+The example above illustrates a scenario where the Micro and Small instances are used. It is also illustrated that despite the Nodes being available,
+they're not necessarily being used, hence billed - only when the infrastructure is required and actually used, it is measured and billed accordingly.
diff --git a/docs/deployment_and_security/deployment/aws/deploy.md b/docs/deployment_and_security/deployment/aws/deploy.md
@@ -6,7 +6,7 @@ to the platform after the installation. The full procedure takes around 45m to 1
 In order to install the platform in your account, the user must have basic knowledge with the used tools, such as CloudFormation,
 Route53 and Cognito.
 
-### 1. Configure the product
+### Configure the product
 
 !!! Note "Make sure that you comply with the pre-flight checks"
 

diff --git a/docs/deployment_and_security/deployment/azure/billing.md b/docs/deployment_and_security/deployment/azure/billing.md
@@ -0,0 +1,46 @@
+# Billing
+After the installation, the client will be billed for all the infrastructure costs plus the usage metrics describe in the offer.
+
+Using a usage-based pricing model you will only pay for what you use.
+
+The following metrics are calculated and sent to Azure in order to charge you at the current offer pricing:
+
+- CPU / Hour
+- Memory / Hour
+- GPU / Hour
+
+The following *Azure services* are mandatory for the platform to work and will be billed:
+
+- Virtual networks
+- IP Address
+- Private DNS Zones
+- Container Registry
+- Storage Account
+- MySQL Server
+- Deployment Scripts
+- Kubernetes Services
+- Key Vault
+- Container Instances
+
+To check the infrastructure costs of the platform, you can use the Azure Cost analysis (under the Cost Management + Billing service) and filter by the
+created resource groups during the deployment. This will aggregate all the resources deployed by the platform.
+
+## Cost Estimations
+
+**YData Fabric** final cost can be estimated following the logic of a usage-based plan since it depends on your users and data. The following table provides
+a guideline of how to compute the total cost for different usage scenarios based on the deployed infrastructure.
+
+| AKS Nodes | Instance Type | vCPUs | Memory (GBi) | GPUs | Number of instances | % Usage/ CPU/Hour | % Usage/ Memory/Hour | % Usage/ GPU/Hour | Cost Azure/Hour | Cost Azure/Day | Cost YData/Hour | Cost YData/Day |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| System | Standard_D8s_v3 | 8 | 32 | 0 | 2 | 30 | 30 | 0 | 0.4800 | 23.04 | 0.288 | 6.912 |
+| CPU Micro (labs) | Standard_D2s_v3 | 2 | 8 | 0 | 1 | 50 | 50 | 0 | 0.1200 | 2.88 | 0.06 | 1.44 |
+| CPU Small (labs) | Standard_D4s_v3 | 4 | 16 | 0 | 1 | 50 | 50 | 0 | 0.2400 | 5.76 | 0.12 | 2.88 |
+| CPU Medium (labs) | Standard_D8s_v3 | 8 | 32 | 0 | 0 | 0 | 0 | 0 | 0.4800 | 0 | 0 | 0 |
+| CPU Large (labs) | Standard_D16s_v3 | 16 | 64 | 0 | 0 | 0 | 0 | 0 | 0.9600 | 0 | 0 | 0 |
+| CPU Compute Micro (computing) | Standard_D32s_v3 | 32 | 128 | 0 | 1 | 80 | 80 | 0 | 1.9200 | 46.08 | 1.536 | 36.864 |
+| GPU Micro (labs) | Standard_NC6s_v3 | 6 | 112 | 1 | 0 | 0 | 0 | 0 | 3.8230 | 0 | 0 | 0 |
+| GPU Compute Micro (computing) | Standard_NC6s_v3 | 6 | 112 | 1 | 0 | 0 | 0 | 0 | 3.8230 | 0 | 0 | 0 |
+
+The example above illustrates a scenario where the Micro and Small instances are used.
+It is also illustrated that despite the Nodes being available, they're not necessarily being used, hence billed - only when the infrastructure is required and actually used,
+it is measured and billed accordingly.
diff --git a/docs/deployment_and_security/deployment/azure/clean.md b/docs/deployment_and_security/deployment/azure/clean.md
@@ -0,0 +1,13 @@
+# Clean
+
+The following procedure explains how to delete the platform. The full procedure takes around 45m to 1h to be completed.
+To clean up **YData Fabric**, you will need to delete the managed app.
+
+Please take in consideration that this will delete **everything associated with the installation**.
+
+- Start by opening the resource group where the managed app is installed, select the **Managed Application** and click *"Delete"*.
+
+![azure Fabric resource group](../../../assets/deployment_security/azure/azure_ydata_resourcegroup.png){: style="width:75%"}
+![azure Fabric delete group](../../../assets/deployment_security/azure/azure_delete_resource_group.png){: style="width:75%"}
+
+This will delete the managed app and the managed resource group where all the components are installed.
diff --git a/docs/deployment_and_security/deployment/azure/deploy.md b/docs/deployment_and_security/deployment/azure/deploy.md
@@ -0,0 +1,103 @@
+# Deploy
+
+## Installation process
+
+!!! note "Ensure that you have completed the pre-deploy checklist"
+
+    Validate if you have checked all the ^^[deploy requirements](deploy.md)^^ before moving forward with the deploy.
+
+## Basic configuration
+
+- Start by defining the basic configuration for the app installation.
+
+#### JIT Access
+![azure app config](../../../assets/deployment_security/azure/azure_activate_jit.png){: style="width:40%"}
+
+- Enable the Just in Time (JIT) access for the app installation as shown in the image below. You can see ^^[more about JIT access in the pre-deploy checklist](pre_deploy_checklist.md)^^.
+
+#### Network configuration
+- Define your network configuration to access *YData Fabric*.
+
+![azure network configuration](../../../assets/deployment_security/azure/azure_network_config.png){: style="width:55%"}
+
+=== "New Public IP"
+
+    - If you choose a new Public IP, you can choose the name or leave it as *(new) default*, but the remaining properties are ignored since the SKU standard
+    and assignment static is the recommended by Azure.
+
+    ![azure public ip](../../../assets/deployment_security/azure/azure_public_ip.png){: style="width:30%"}
+
+    - After that, choose a DNS label for the domain as shown below.
+
+    ![azure define dns](../../../assets/deployment_security/azure/azure_define_dns.png){: style="width:55%"}
+
+=== "Existing Public IP"
+
+    - If you opt for an existing Public IP, you can choose that IP from the dropdown. The DNS Public Endpoint is automatically filled since this is configured on the IP Address level.
+    If your IP is  disabled, please ensure you have the DNS name label defined and the IP is not allocated to any other resource.
+
+    ![azure existing ip](../../../assets/deployment_security/azure/azure_existing_ip.png){: style="width:55%"}
+    ![azure existing ip](../../../assets/deployment_security/azure/azure_selected_existing_ip.png){: style="width:55%"}
+
+    For the DNS Custom Domain, you can use a custom domain, such as, for example platform.ydata.ai.
+    After the installation process you will need to create a CNAME or an A record in your DNS provider. More information in the *Post installation step*.
+
+#### OAuth
+- Define how you will authenticate to the app after the deployment is completed.
+
+![azure authentication](../../../assets/deployment_security/azure/azure_authentication.png){: style="width:55%"}
+
+#### Analytics
+- You can opt for allowing or not the collection of metrics in order to help us understand how users interact with the product. No user data is collected at any point.
+Read more about ^^[YData privacy policy](https://ydata.ai/privacy)^^.
+
+#### Create
+- Click *“Next”*. Check the provided parameters.
+- Insert the contact information
+- Read and accept the terms and conditions. Finally click in *"Create"*.
+
+![azure create](../../../assets/deployment_security/azure/azure_create.png){: style="width:40%"}
+
+## Following the installation process
+- After click the “Create” button, the installation of the managed app will start, as shown in the image below.
+
+![azure ydata app](../../../assets/deployment_security/azure/azure_ydata_app.png){: style="width:75%"}
+
+The process will take approximately 45-60 minutes.
+
+- If any error occur during installation, please open a support case at ^^[support.ydata.ai](https://support.ydata.ai)^^.
+
+## Post installation configuration
+
+### IP configuration
+If you choose to use **one existing IP** for the platform, you will need to create a role assignment to the resource group where the IP is located.
+To do this, open your managed resource group (where the resources are created) and open the `ydata-cluster-managed-identity` Managed Identity.
+
+- Click “**Azure Role Assignments**”
+
+![azure role assigment](../../../assets/deployment_security/azure/azure_role_assignment.png){: style="width:55%"}
+
+- Click “Add role assignment” as shown in the image below.
+
+![azure role assigment](../../../assets/deployment_security/azure/azure_role_assignment_preview.png){: style="width:60%"}
+
+- Choose the Scope *“Resource group”*.
+- Choose the subscription where the resource group is located.
+- Select the resource group where the IP is located.
+- Add the role *“Network Contributor”* and *"Save"*.
+
+### DNS Configuration
+If you opt for the **DNS Custom Domain**, you will need to create a CNAME record pointing to the DNS Public Endpoint or an A record pointing to the IP.
+Example in Route53:
+
+![azure route53](../../../assets/deployment_security/azure/azure_config_route53.png){: style="width:65%"}
+![azure create route53](../../../assets/deployment_security/azure/azure_create_route.png){: style="width:65%"}
+
+## Connecting to YData Fabric
+
+You can get the full URL in the **Managed APP → “Parameters and Outputs” tab → Outputs**
+
+![azure resources YData Fabric](../../../assets/deployment_security/azure/azure_ydata_resources.png){: style="width:75%"}
+![azure parameters YData Fabric](../../../assets/deployment_security/azure/azure_ydata_parameters_outputs.png){: style="width:75%"}
+
+🚀 Congratulations you are now ready to start exploring your data with **YData Fabric**!
diff --git a/docs/deployment_and_security/deployment/azure/pre_deploy_checklist.md b/docs/deployment_and_security/deployment/azure/pre_deploy_checklist.md
@@ -0,0 +1,170 @@
+# Checklist and Prerequisites
+
+Deploying [YData Fabric in the Microsoft Azure](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/ydatalda1622051287515.ydata1?tab=overview) offers a scalable and efficient solution for managing and generating synthetic data. AWS provides a robust
+infrastructure that ensures high availability, security, and performance, making it an ideal platform for **YData Fabric**.
+
+This cloud deployment allows for rapid scaling of resources to meet varying workloads, ensuring optimal performance and cost-efficiency.
+
+With Microsoft's comprehensive security features, including data encryption, network firewalls, and identity management,
+your synthetic data and models are protected against unauthorized access and threats.
+Additionally, Azure's global infrastructure allows for the deployment of YData Fabric in multiple regions,
+ensuring low latency and high availability for users worldwide.
+
+!!! Note "Prerequisites"
+
+    If you don't have an Azure account, create a ^^[free account](https://azure.microsoft.com/en-us/free/)^^ before you begin.
+
+## Basic Configuration
+
+- **Subscription**: where the platform will be installed
+- **Resource group**: where the managed app will be installed:
+    - A new one is recommended and can be created automatically during the deployment.
+
+- **Location**: where to install the Managed APP and create the resource groups. The available location for now are:
+    - West Europe - Netherlands [westeurope]
+    - West US - California [westus]
+    - West US - Washington [westus2]
+    - Canada Central [canadacentral]
+    - Sweden Central [swedencentral]*
+
+    If you need another region, please fill up a support case at ^^[support.ydata.ai](http://support.ydata.ai)^^.
+
+    **Regions without available GPU’s machine types at the time*
+
+- **Application Name**: the Managed APP name
+- **Managed Resource Group**: the resource group created by the Managed APP and where all the infrastructure services will be created
+(this is created automatically).
+
+## Permissions
+Check and add (if needed) the necessary permissions to the subscription where the platform will be installed.
+
+- Go to **Subscriptions**.
+- Select the subscription where *YData Fabric* will be installed.
+- Click *“View my access”* as shown in the image below.
+
+![azure check permissions](../../../assets/deployment_security/azure/azure_check_permissions.png){: style="width:35%"}
+
+- Check if you have at least the following configurations:
+
+
+  - Contributor
+  ![azure contributor](../../../assets/deployment_security/azure/azure_contributor.png){: style="width:70%"}
+
+  And the following **permissions**:
+
+  - Microsoft.Authorization/roleAssignments/read
+
+  - Microsoft.Authorization/roleAssignments/write
+
+  ![azure role assignment](../../../assets/deployment_security/azure/azure_role_assignment.png){: style="width:30%"}
+
+- If not, please create a custom role with this two permissions and create the role assignment to the user in the subscription.
+
+For more information check Azure's official documentation on ^^[Azure custom roles](https://learn.microsoft.com/en-us/azure/role-based-access-control/custom-roles)^^
+and [Azure built-in roles](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#contributor).
+
+## Resource Providers
+Check and activate (if needed) resource providers for the subscription where the YData platform will be installed following the next steps.
+
+- Go to **Subscriptions**
+- Select the subscription where *YData Fabric* will be installed
+- Go to Resource Providers
+- Using the filter, check if you have the following resource providers registered. If not, please click the resource provider and click *“Register”*.
+
+    - **Microsoft.Compute**
+    - **Microsoft.ContainerInstance**
+
+    ![azure compute](../../../assets/deployment_security/azure/azure_contributor.png){: style="width:75%"}
+    ![azure container](../../../assets/deployment_security/azure/azure_container.png){: style="width:75%"}
+
+For more information check ^^[Azure's official documentation on resource providers](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-providers-and-types)^^
+and ^^[Azure Resource Manager](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-services-resource-providers)^^.
+
+## Register features
+Check and register (if needed) the required features.
+- Install and update the **aks-preview extension**:
+
+``` shell
+    az extension add --name aks-preview
+    az extension update --name aks-preview
+```
+
+- Register the **'EnableWorkloadIdentityPreview'** feature flag
+
+``` shell
+    az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"
+```
+
+- Wait until feature to be registered:
+
+``` shell
+    az feature show --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"
+```
+``` json
+    {
+        "id": "/subscriptions/xxxxx/providers/Microsoft.Features/providers/Microsoft.ContainerService/features/EnableWorkloadIdentityPreview",
+        "name": "Microsoft.ContainerService/EnableWorkloadIdentityPreview",
+        "properties": {
+            "state": "Registered"
+        },
+        "type": "Microsoft.Features/providers/features"
+    }
+```
+
+- After the feature status is “Registered”, refresh the registration of the container service resource provider:
+
+``` shell
+    az provider register --namespace Microsoft.ContainerService
+```
+
+Read more in Azure's official documentation on ^^[Azure Kubernetes Services (AKS)](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster#install-the-aks-preview-azure-cli-extension)^^.
+
+## Resource compute quotas
+Check and set (if needed) new quotas for the region where the managed app will be installed.
+
+- Go to **Subscriptions**.
+- Select the subscription where *YData Fabric* will be installed
+- Click _“Usage + quotas”_
+- Filter by the region where _YData Fabric_ will be installed
+
+![azure check quotas](../../../assets/deployment_security/azure/azure_check_quotas.png){: style="width:60%"}
+
+- Check for the following quota limits:
+
+| Quota | Minimum | Recommended |
+| --- | --- | --- |
+| Total Regional vCPUs | 16* | 100** |
+| Standard DSv3 Family vCPUs | 16* | 100** |
+| Standard NCSv3 Family vCPUs*** | 6* | 20** |
+| Standard DDSv4 Family vCPUs | 10 | 10 |
+
+*These limits are the required only for the installation of the platform. Usage is limited.
+
+** *Each limit will depend on the platform usage and each client requirements.*
+
+*** Not available in Sweden region
+
+- If needed, request for a new limit to the azure support team as per the image below.
+
+![azure check quotas](../../../assets/deployment_security/azure/azure_request_quota.png){: style="width:60%"}
+
+Check ^^[Azure's official documentation on quotas](https://learn.microsoft.com/en-us/azure/quotas/view-quotas)^^,
+^^[increase regional vCPU quotas](https://learn.microsoft.com/en-us/azure/quotas/regional-quota-requests)^^ and
+^^[increase VM-family quotas](https://learn.microsoft.com/en-us/azure/quotas/per-vm-quota-requests)^^.
+
+## JIT Access
+The JIT Access feature will prevent YData Fabric from having write access to the managed app at any time.
+
+- To use the just-in-time access, you must have an ^^[Azure Active Directory P2 license](https://docs.microsoft.com/en-us/azure/active-directory/privileged-identity-management/subscription-requirements)^^.
+- Without this license and with the JIT enable, YData will not be able to give any closer support or make updates to the solution.
+
+To check your current license, go to the **Azure Portal → Azure Active Directory → Licenses** and check your license.
+To activate the P2 license, click the **“Try/Buy”** button.
+
+![azure licenses](../../../assets/deployment_security/azure/azure_licenses.png){: style="width:50%"}
+
+For more information check Azure's official documentation on ^^[assigning and removing licenses to Azure Active directory](https://learn.microsoft.com/en-us/entra/fundamentals/license-users-groups)^^.
+^^[To learn how to enable JIT access and approve requests](https://learn.microsoft.com/en-us/azure/azure-resource-manager/managed-applications/approve-just-in-time-access)^^.
+
+After accepting the request, the YData team will have access in order to make updates and give you closer support.
+Any other requests open a support case at ^^[support.ydata.ai](https://support.ydata.ai)^^.