diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index 1e065aa1..92a9312e 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -90,6 +90,9 @@ You can extend UDX Worker to create custom images tailored to your applications ```shell FROM usabilitydynamics/udx-worker:latest +# Copy the worker +COPY src/worker.yml /home/${USER}/.cd/configs/worker.yml + # Add your dependencies or scripts COPY my-app /usr/src/app ``` diff --git a/README.md b/README.md index dda23393..0ddbcc76 100644 --- a/README.md +++ b/README.md @@ -49,11 +49,11 @@ For more details on available commands make ``` -## Detailed Documentation +## Detailed Modules Documentation -- [Secure Environment](src/configs/readme.md) -- [Auth Modules](lib/auth/readme.md) -- [Configuration and Environment Management](lib/secrets/readme.md) +- [Secure Environment Configuration](src/configs/readme.md) +- [Authentication](lib/auth/readme.md) +- [Secrets Management](lib/secrets/readme.md) ## Contributing diff --git a/lib/auth/readme.md b/lib/auth/readme.md index f3e90e2b..9f9cb476 100644 --- a/lib/auth/readme.md +++ b/lib/auth/readme.md @@ -1,50 +1,27 @@ -# Authentication Modules +# Authentication -This directory contains scripts for setting up authentication modules for various cloud providers. These modules allow the UDX Worker to securely authenticate and perform tasks on behalf of the user. +This directory contains scripts for setting up authentication modules for various cloud providers. These modules allow the worker container to securely authenticate and perform tasks on behalf of the user. -## Prerequisites - -- Azure CLI: [Installation Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) -- AWS CLI: [Installation Guide](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) -- GCP SDK: [Installation Guide](https://cloud.google.com/sdk/docs/install) - -## Setup - -### Azure Service Principal - -#### Create - -To create an Azure Service Principal, run the following command: +## Overview -```shell -az ad sp create-for-rbac --name "udx-worker-sp" --role contributor --scopes /subscriptions/YOUR_SUBSCRIPTION_ID -``` +Authentication is a critical component for securely managing interactions with various cloud services. The provided authentication modules support cloud providers like `Azure`, `AWS`, `GCP`, and `Bitwarden`. By utilizing these modules, the worker container can securely access and perform tasks with the necessary permissions. -Note the appId, password, and tenant. +These authentication modules not only enable the worker container to authenticate with different cloud providers but also allow it to fetch task-specific or application-specific secrets. This ensures that sensitive information is securely retrieved and used only when necessary. -#### Use +Ensuring proper authentication helps maintain the integrity and security of automated operations in a DevSecOps environment. By adhering to best practices, such as using least privilege and regularly auditing access logs, you can further enhance the security of your workflows. -Update your `worker.yml` configuration file to include the Azure Service Principal credentials: - -```yaml -actors: - - type: azure - creds: "${AZURE_CREDS}" -``` - -### AWS IAM Role (TBD) - -Instructions for setting up AWS IAM Role will be provided here. - -### GCP Service Account (TBD) +## Prerequisites -Instructions for setting up GCP Service Account will be provided here. +- **Azure CLI**: [Installation Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) +- **AWS CLI**: [Installation Guide](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) +- **GCP SDK**: [Installation Guide](https://cloud.google.com/sdk/docs/install) +- **Bitwarden CLI**: [Installation Guide](https://bitwarden.com/help/cli/) -### Bitwarden Service Account (TBD) +## How to Use -Instructions for setting up GCP Service Account will be provided here. +Please check [Secure Environment Configuration](src/configs/readme.md) for config details. ## Best Practices - **Use least privilege**: Assign the minimum required permissions to service accounts and roles. -- **Audit access regularly**: Periodically review access logs and permissions to ensure compliance with security policies. +- **Audit access regularly**: Periodically review access logs and permissions to ensure compliance with security policies. \ No newline at end of file diff --git a/lib/secrets/readme.md b/lib/secrets/readme.md index 12ef5768..882531bb 100644 --- a/lib/secrets/readme.md +++ b/lib/secrets/readme.md @@ -1,49 +1,31 @@ -# Secrets Management Modules +# Secrets Management -This directory contains scripts for setting up and managing secrets for various cloud providers. These modules enable the UDX Worker to securely fetch and use secrets during its operations. +## Overview -## Prerequisites - -- Azure CLI: [Installation Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) -- AWS CLI: [Installation Guide](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) -- GCP SDK: [Installation Guide](https://cloud.google.com/sdk/docs/install) - -## Setup - -### Azure Key Vault +Secrets management is a vital component for securely handling sensitive information required by applications and tasks within a containerized environment. The provided secrets management scripts support cloud providers like `Azure`, `AWS`, `GCP`, and `Bitwarden`. These scripts enable the worker container to securely retrieve secrets from different providers and make them available for application logic and task execution. -#### Configure +The scripts ensure that secrets are fetched dynamically at runtime, reducing the risk of exposure and ensuring that applications always have access to the latest sensitive information. By integrating with secure storage solutions and following encryption best practices, these scripts help maintain the confidentiality and integrity of the data used by automated tasks. -Ensure you have set up an Azure Key Vault and added your secrets. You can add secrets using the Azure CLI: +### Importance in DevSecOps -```shell -az keyvault secret set --vault-name "your-vault-name" --name "your-secret-name" --value "your-secret-value" -``` +In a DevSecOps environment, proper secrets management is crucial for maintaining the security and integrity of automated workflows. By adhering to best practices, such as using least privilege, regularly auditing access, and monitoring for leaks, organizations can significantly reduce the risk of sensitive information exposure. -#### Use - -Update your worker.yml configuration file to include the Azure Key Vault secrets: - -```yaml -secrets: - NEW_RELIC_API_KEY: "azure/kv-udx-worker/new-relic-api-key" - HEALTHCHECK_IO_API_KEY: "azure/kv-udx-worker/healthcheck-io-api-key" -``` - -### AWS IAM Role (TBD) +## Prerequisites -Instructions for setting up AWS IAM Role will be provided here. +- Azure CLI: [Installation Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) +- AWS CLI: [Installation Guide](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) +- GCP SDK: [Installation Guide](https://cloud.google.com/sdk/docs/install) -### GCP Service Account (TBD) +## How to Use -Instructions for setting up GCP Service Account will be provided here. +Please check [Secure Environment Configuration](src/configs/readme.md) for config details. -### Bitwarden Service Account (TBD) +## Secrets Configurations -Instructions for setting up GCP Service Account will be provided here. +You can find details of secrets configurations for each supported service provider here - [Secrets Configurations](lib/secrets/secrets_configurations.md) ## Best Practices - **Encrypt secrets**: Ensure secrets are encrypted at rest and in transit. - **Secure storage**: Use secure storage solutions like AWS Secrets Manager, Azure Key Vault, or Google Secret Manager for storing sensitive information. -- **Monitor for leaks**: Use monitoring tools to detect if secrets are accidentally exposed. +- **Monitor for leaks**: Use monitoring tools to detect if secrets are accidentally exposed. \ No newline at end of file diff --git a/lib/secrets/secrets_configurations.md b/lib/secrets/secrets_configurations.md new file mode 100644 index 00000000..abed9b1a --- /dev/null +++ b/lib/secrets/secrets_configurations.md @@ -0,0 +1,147 @@ +# Secrets Configurations + +## Azure Key Vault + +### Prerequisites +- Access to an Azure subscription and an Azure Key Vault. +- Azure CLI installed for setup and authentication. + +### Setup +1. **Create Key Vault**: + - In the Azure portal, navigate to Key Vaults and select Create. + - Set up the resource group, key vault name, and region. + - Choose Review + Create and then Create. + +### Add Secrets +1. Go to Secrets within your Key Vault. +2. Select Generate/Import to add a new secret. +3. Name your secret descriptively and set the secret value. + +### Grant Access +1. Go to Access policies in your Key Vault. +2. Add an access policy to grant permissions for Get and List for secrets. +3. Select the Azure AD Application that represents your UDX Worker or create a new one. +4. Save the changes. + +### Set Credentials For Environment Authorization +Set Azure credentials in your `.env` file (or set GitHub Action secret), referencing a service principal with access to the Key Vault. + +```txt +AZURE_CREDS='{"clientId": "your-client-id", "clientSecret": "your-client-secret", "tenantId": "your-tenant-id"}' +``` + +### Reference in `worker.yml` + +```yml +secrets: + MY_SECRET: "azure/your-vault-name/your-secret-name" +``` + +- **Vault Reference**: Use "azure/{Vault Name}/{Secret Name}" in the secrets section of `worker.yml`. +- **Secret Reference**: Use the secret's name as defined in Azure Key Vault. + +## Google Cloud Secret Manager + +### Prerequisites +- Access to a Google Cloud project with Secret Manager API enabled. +- Google Cloud SDK installed for setup and authentication. + +### Setup +1. **Enable Secret Manager API**: + - In the Google Cloud Console, go to APIs & Services > Dashboard. + - Select Enable APIs and Services and search for Secret Manager API. + - Enable the API for your project. + +### Create Secrets +1. In the Google Cloud Console, go to Secret Manager. +2. Select Create Secret. +3. Provide a descriptive name for the secret and enter the secret value. +4. Click Create Secret to save. + +### Grant Access +1. Go to IAM & Admin > IAM in the Google Cloud Console. +2. Add a role to the service account that will be used by UDX Worker to access the secrets. +3. Assign the Secret Manager Secret Accessor role to this service account for access to specific secrets. + +### Set Credentials For Environment Authorization +Set Google Cloud credentials in your `.env` file (or set GitHub Action secret), referencing a service account with access to Secret Manager. + +```txt +GCP_CREDS='{"type": "service_account", "project_id": "your-project-id", "private_key_id": "your-private-key-id", "private_key": "your-private-key", "client_email": "your-client-email", "client_id": "your-client-id", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "your-cert-url"}' +``` + +### Reference in `worker.yml` + +```yml +secrets: + MY_SECRET: "gcp/your-project-id/your-secret-name" +``` + +- **Project Reference**: Use "gcp/{Project ID}/{Secret Name}" in the secrets section of `worker.yml`. +- **Secret Reference**: Use the secret's name as defined in Google Cloud Secret Manager. + +## AWS Secrets Manager + +### Prerequisites +- Access to an AWS account with Secrets Manager enabled. +- AWS CLI installed for setup and configuration. + +### Setup +1. **Create Secrets**: + - Go to AWS Secrets Manager in the AWS Management Console. + - Choose Store a new secret and select the secret type (e.g., credentials or plaintext). + - Enter the secret value and give it a descriptive name. + - Complete the setup and store the secret. + +### Grant Access +1. Go to IAM in the AWS Management Console and create (or select) a service role that UDX Worker will use. +2. Attach the SecretsManagerReadWrite permission or a custom policy that grants read access to specific secrets. + +### Set Credentials For Environment Authorization +Set AWS credentials in your `.env` file (or set GitHub Action secret), referencing an IAM user or role with the necessary permissions to access Secrets Manager. + +```txt +AWS_CREDS='{"aws_access_key_id": "your-access-key-id", "aws_secret_access_key": "your-secret-access-key", "region": "your-region"}' +``` + +### Reference in `worker.yml` + +```yml +secrets: + MY_SECRET: "aws/secrets-manager/your-secret-name" +``` + +- **Path Reference**: Use "aws/secrets-manager/{Secret Name}" in the secrets section of `worker.yml`. +- **Secret Name**: Use the name of the secret as it appears in AWS Secrets Manager. + +## Bitwarden + +### Prerequisites +- Access to a Bitwarden organization and Bitwarden CLI. + +### Enable Secret Manager (Optional for Advanced Features) +1. Go to Settings > Secret Manager in your Bitwarden organization admin portal (requires an enterprise plan). + +### Add Secrets +1. Create a Collection in Collections. +2. Add secrets as Secure Notes in this collection, using descriptive names for easy identification. + +### Grant Access +1. Ensure the correct users or groups have read access to the collection. + +### Set Credentials For Environment Authorization +Add Bitwarden credentials in your `.env` file (or set GitHub Action secret). + +```txt +BITWARDEN_CREDS='{"clientId": "your-client-id", "clientSecret": "your-client-secret"}' +``` + +### Reference in `worker.yml` + +```yml +secrets: + MY_SECRET: "bitwarden/your-collection-name/your-secret-name" +``` + +- **Collection**: Use the Collection Name (or ID if there’s ambiguity). +- **Secret Reference**: Use the format "bitwarden/{Collection Name}/{Secret Name}". \ No newline at end of file diff --git a/src/configs/readme.md b/src/configs/readme.md index e0e5dd55..c22dcc8d 100644 --- a/src/configs/readme.md +++ b/src/configs/readme.md @@ -1,6 +1,6 @@ # Secure Environment Configuration -This directory contains configuration files for setting up a secure UDX Worker environment. +This directory contains configuration files for setting up a secure worker environment. The configurations are designed to ensure that the environment adheres to zero-trust principles and provides maximum security for handling secrets and running automation tasks. @@ -8,56 +8,105 @@ The configurations are designed to ensure that the environment adheres to zero-t - `worker.yml`: Main configuration file for environment variables, secrets, and authentication. +Your user configuration in `worker.yml` can extend or replace values from the built-in configuration. Below is an example of how you can specify your own environment variables and secrets. + +**Example `worker.yml`** + +```yaml +--- +kind: workerConfig +version: udx.io/worker-v1/config +config: + env: + AZURE_CLIENT_ID: "12345678-1234-1234-1234-1234567890ab" + AZURE_TENANT_ID: "abcdef12-3456-7890-abcd-ef1234567890" + AZURE_SUBSCRIPTION_ID: "1234abcd-5678-90ef-abcd-12345678abcd" + AZURE_RESOURCE_GROUP: "rg-example" + APIM_SERVICE_NAME: "example-apim" + ACR_REPO_NAME: "exampleacr" + STORAGE_ACCOUNT_NAME: "examplestorage" + KEY_VAULT_NAME: "examplekv" + MANAGED_IDENTITY_NAME: "exampleidentity" + + secrets: + APP_CLIENT_SECRET: "azure/kv-example/clientSecret" +``` + ## Usage To use these configuration files, ensure that the `worker.yml` file is correctly configured and placed in the appropriate directory (`/home/$USER/.cd/configs/`) within the container. -### Example Configuration +### Volume Mount -**worker.yml** +If you have a worker configuration outside of the worker image, you can mount it as a volume into the container: -```yaml -config: - variables: - DOCKER_IMAGE_NAME: "udx-worker" - secrets: - NEW_RELIC_API_KEY: "azure/kv-udx-worker/new-relic-api-key" - HEALTHCHECK_IO_API_KEY: "azure/kv-udx-worker/healthcheck-io-api-key" - actors: - - type: azure - creds: "${AZURE_CREDS}" +```shell +docker run -d --name udx-worker \ + --env-file .env \ + -v $(pwd)/my-tasks:/usr/src/app \ + -v $(pwd)/.cd/configs/worker.yml:/home/udx/.cd/configs/worker.yml \ + usabilitydynamics/udx-worker:latest ``` -## Local Environment Configuration +### Github Action Integration + +In a GitHub Actions workflow, you can mount a configuration file from outside the worker image as a volume into the container: -The `.udx` file is used to store local environment variables required by the UDX Worker. This file should be placed in the root directory of your project. +```yml +name: Deploy with UDX Worker -### Purpose +on: + workflow_dispatch: -The `.udx` file contains sensitive environment variables that are referenced in the `worker.yml` configuration file. This allows you to keep secrets out of your configuration files and manage them securely. +permissions: + contents: read -### Usage +jobs: + deploy: + runs-on: ubuntu-latest -1. Create a `.udx` file in the root directory of your project. -2. Add the necessary environment variables to the `.udx` file. + steps: + - name: Checkout Code + uses: actions/checkout@v4 -### Example + - name: Deploy Using UDX Worker + env: + AZURE_CREDS: ${{ secrets.AZURE_CREDS }} + run: | + echo "Starting deployment with UDX Worker..." + docker run --rm \ + -e AZURE_CREDS \ + -v $(pwd)/src/configs/worker.yml:/home/udx/.cd/configs/worker.yml:ro \ + -v $(pwd)/.cd/bin:/home/udx/.cd/bin:ro \ + usabilitydynamics/udx-worker:latest \ + sh -c " + echo 'Step 1: Show Deployment Variables'; + /home/udx/.cd/bin/10_show_variables.sh; -**.udx** + echo 'Step 2: Deploy Infrastructure'; + /home/udx/.cd/bin/20_deploy_infra.sh; + + echo 'Step 3: Deploy Application'; + /home/udx/.cd/bin/30_deploy_service.sh; + " +``` -```txt -AZURE_SUBSCRIPTION_ID="b83b62a9-286f-426c-be8a-fc71300f92d2" -AZURE_TENANT_ID="2a8330a4-138c-4c93-977b-cee1faadb2dc" -AZURE_APPLICATION_ID="44f11324-81a9-4573-8853-21c1f44f0ed0" -AZURE_APPLICATION_PASSWORD="*************" +### Child Image Integration + +To configure the child worker, you can integrate the configuration into a Docker image. + +If you want to include the worker configuration directly in your Docker image, you can use the `COPY` command in your Dockerfile. Assuming your configuration file is located at `src/configs/worker.yml` in your repository, you can add the following line to your Dockerfile: + +``` +COPY src/worker.yml /home/${USER}/.cd/configs/worker.yml ``` -### How It's Working +## Configuration Loading and Merging -The `.udx` file is loaded by the UDX Worker to populate the environment variables referenced in `worker.yml`. This ensures that sensitive information is managed securely and not hard-coded in configuration files. +The [lib/worker_config.sh](../../lib/worker_config.sh) script handles the loading and merging of the configurations. It combines the built-in configuration with user-provided configurations if they exist. -### Best Practices +### Process -- **Do not hard-code secrets**: Use environment variables or a secrets management tool. -- **Regularly rotate secrets**: Change your secrets periodically to reduce the risk of compromise. -- **Limit access**: Ensure that only authorized personnel have access to the configuration files. +1. **Built-in Configuration**: The built-in configuration file is located at `/etc/worker/worker.yml`. +2. **User Configuration**: If a user-provided configuration file exists at `/home/$USER/.cd/configs/worker.yml` in the container, it will be merged with the built-in configuration. User configurations can extend or replace values from the built-in configuration. +3. **Merged Configuration**: The merged configuration is stored at `/home/$USER/.cd/configs/merged_worker.yml` and parsed by worker config module logic.