v1.83.0

Update `atmos describe affected` command @aknysh (#635)

## what

• Update atmos describe affected command
• If --upload=true flag is passed, include dependents for all dependents (even an empty list), and include the settings section for all the dependent components

why

• Make the API schema consistent on the server that processes the result of atmos describe affected --upload=true command

v1.82.0

Add `--upload` flag to `atmos describe affected` command @aknysh (#631)

## what

• Add --upload flag to atmos describe affected command
• Update docs
  • https://atmos.tools/cli/commands/describe/affected/

why

If the --upload=true command-line flag is passed, Atmos will upload the affected components and stacks to a specified HTTP endpoint.

The endpoint can process the affected components and their dependencies in a CI/CD pipeline (e.g. execute terraform apply on all the affected components in the stacks and all the dependencies).

Atmos will perform an HTTP POST request to the URL ${ATMOS_PRO_BASE_URL}/${ATMOS_PRO_ENDPOINT}, where the base URL is defined by the ATMOS_PRO_BASE_URL environment variable, and the URL path is defined by the ATMOS_PRO_ENDPOINT environment variable.

An Authorization header Authorization: Bearer $ATMOS_PRO_TOKEN will be added to the HTTP request (if the ATMOS_PRO_TOKEN environment variable is set) to provide credentials to authenticate with the server.

NOTE: If the --upload=true command-line flag is passed, the --include-dependencies and --include-settings flags are automatically set to true, so the affected components will be uploaded with their dependencies and settings (if they are configured in Atmos stack manifests).

The payload of the HTTP POST request will be a JSON object with the following schema:

  {
     "base_sha": "6746ba4df9e87690c33297fe740011e5ccefc1f9",
     "head_sha": "5360d911d9bac669095eee1ca1888c3ef5291084",
     "repo_url": "https://github.com/cloudposse/atmos",
     "repo_host": "github.com",
     "repo_name": "atmos",
     "repo_owner": "cloudposse",
     "stacks": [
        {
          "component": "vpc",
          "component_type": "terraform",
          "component_path": "examples/quick-start/components/terraform/vpc",
          "stack": "plat-ue2-dev",
          "stack_slug": "plat-ue2-dev-vpc",
          "affected": "stack.vars",
          "included_in_dependents": false,
          "dependents": [],
          "settings": {}
        }
    ]
 }

where:

• base_sha - the Git commit SHA of the base branch against which the changes in the current commit are compared

• head_sha - the SHA of the current Git commit

• repo_url - the URL of the current repository

• repo_host - the host of the current repository

• repo_name - the name of the current repository

• repo_owner - the owner of the current repository

• stacks - a list of affected components and stacks with their dependencies

v1.81.0

Introduce Atmos `Go` template functions. Add `atmos.Component` function to read values from other Atmos components including outputs (remote state) @aknysh (#628)

## what

• Introduce Atmos Go template functions
• Add atmos.Component template function to read values from other Atmos components including outputs (remote state)
• Update docs
  • Atmos Template Functions
  • atmos.Component function

why

• Atmos now supports custom Go template functions similar to Sprig and Gomplate

  In Go templates in Atmos stack manifests, you can use the following functions and datasources:

  • Go text/template functions
  • Sprig Functions
  • Gomplate Functions
  • Gomplate Datasources
  • Atmos Template Functions

description

The atmos.Component template function allows you to read any Atmos section or any attribute from a section for an Atmos component in a stack, and use it in Go templates in Atmos component configurations.

Usage

  {{ (atmos.Component "<component>" "<stack>").<section>.<attribute> }}

Arguments

• component - Atmos component name

• stack - Atmos stack name

• section - Atmos section name. Any section returned by the CLI command atmos describe component can be used. A special outputs section is also supported to get the outputs (remote state) of Terraform/OpenTofu components.

  NOTE: Using the outputs section in the atmos.Component command is an alternative way to read the outputs (remote state) of a component in a stack directly in Atmos stack manifests instead of using the remote-state module and configuring Terraform/OpenTofu components to use the remote-state module as described in Component Remote State

• attribute - attribute name (field) from the section. attribute is optional, you can use the section itself if it's a simple type (e.g. string). Any number of attributes can be chained using the dot (.) notation. For example, if the first two attributes are maps, you can chain them and get a field from the last map:

    {{ (atmos.Component "<component>" "<stack>").<section>.<attribute1>.<attribute2>.<field1> }}

Specifying Atmos stack

stack is the second argument of the atmos.Component function, and it can be specified in a few different ways:

• Hardcoded stack name. Use it if you want to get an output from a component from a different (well-known and static) stack. For example, you have a tgw component in a stack plat-ue2-dev that requires the vpc_id output from the vpc component from the stack plat-ue2-prod:

    components:
      terraform:
        tgw:
          vars:
            vpc_id: '{{ (atmos.Component "vpc" "plat-ue2-prod").outputs.vpc_id }}'

• Use the .stack (or .atmos_stack) template identifier to specify the same stack as the current component (for which the atmos.Component function is executed):

    {{ (atmos.Component "<component>" .stack).<section>.<attribute> }}
    {{ (atmos.Component "<component>" .atmos_stack).<section>.<attribute> }}

  For example, you have a tgw component that requires the vpc_id output from the vpc component in the same stack:

    components:
      terraform:
        tgw:
          vars:
            vpc_id: '{{ (atmos.Component "vpc" .stack).outputs.vpc_id }}'

• Use the printf template function to construct stack names using static strings and dynamic identifiers:

    {{ (atmos.Component "<component>" (printf "%s-%s-%s" .vars.tenant .vars.environment .vars.stage)).<section>.<attribute> }}
  
    {{ (atmos.Component "<component>" (printf "plat-%s-prod" .vars.environment)).<section>.<attribute> }}
  
    {{ (atmos.Component "<component>" (printf "%s-%s-%s" .settings.context.tenant .settings.context.region .settings.context.account)).<section>.<attribute> }}

  For example, you have a tgw component deployed in the stack plat-ue2-dev. The tgw component requires the vpc_id output from the vpc component from the same environment (ue2) and same stage (dev), but from a different tenant net (instead of plat):

    components:
      terraform:
        tgw:
          vars:
            vpc_id: '{{ (atmos.Component "vpc" (printf "net-%s-%s" .vars.environment .vars.stage)).outputs.vpc_id }}'

  
  

  NOTE: By using the printf "%s-%s-%s" function, you are constructing stack names using the stack context variables/identifiers. For more information on Atmos stack names and how to define them, refer to stacks.name_pattern and stacks.name_template sections in atmos.yaml CLI config file

Examples

The following configurations show different ways of using the atmos.Component template function to read values from different Atmos sections directly in Atmos stack manifests, including the outputs of other (already provisioned) components.

# Global `settings` section
# It will be added and deep-merged to the `settings` section of all components
settings:
  test: true

components:
  terraform:
    test:
      metadata:
        # Point to the Terraform/OpenTofu component
        component: "test"
      vars:
        name: "test"

    test1:
      metadata:
        # Point to the Terraform/OpenTofu component
        component: "test1"
      vars:
        name: "test1"
    
    test2:
      metadata:
        # Point to the Terraform/OpenTofu component
        component: "test2"
      vars:
        name: "test2"
        # Use the `atmos.Component` function to get the outputs of the Atmos component `test1`
        # The `test1` component must be already provisioned and its outputs stored in the Terraform/OpenTofu state
        # Atmos will execute `terraform output` on the `test1` component in the same stack to read its outputs
        test1_id: '{{ (atmos.Component "test1" .stack).outputs.test1_id }}'
        tags:
          # Get the `settings.test` field from the `test` component in the same stack
          test: '{{ (atmos.Component "test" .stack).settings.test }}'
          # Get the `metadata.component` field from the `test` component in the same stack
          test_terraform_component: '{{ (atmos.Component "test" .stack).metadata.component }}'
          # Get the `vars.name` field from the `test1` component in the same stack
          test1_name: '{{ (atmos.Component "test1" .stack).vars.name }}'

[docs] Integration GHA fix version compatibility table @goruha (#626)

## what * [docs] Integration GHA fix version compatibility table

why

• Table in tip box looks ugly

v1.80.0

Add `--include-settings` flag to `atmos describe affected` command @aknysh (#624)

## what

• Add --include-settings flag to atmos describe affected command
• Update docs

why

• If the --include-settings=true flag is passed, atmos describe affected will include the settings section for each affected component in the stack. The settings sections is a free-form map used to pass configuration information to Atmos Integrations. Having the settings section in the output will allow the integrations to parse it and detect settings for the corresponding integration

Fix goreleaser @goruha (#623)

## what * Added `atmos` specific goreleaser

why

• Set atmos version on building

v1.79.0

Fix an issue with the `component_info` output from the `atmos describe component` command. Add `assume_role` property to Atmos JSON Schema S3 backend @aknysh (#621)

## what

• Fix an issue with the component_info output from the atmos describe component command
• Add assume_role property to Atmos JSON Schema S3 backend

why

• The issue with the component_info output from the atmos describe component command was introduced in the previous PRs (different order of execution when evaluation Go templates in Atmos stack manifests)

• Support the recommended assume_role property in S3 backends. Assuming an IAM Role can be configured in two ways. The preferred way is to use the argument assume_role, the other, which is deprecated, is with arguments at the top level (e.g. role_arn)

references

• https://developer.hashicorp.com/terraform/language/settings/backends/s3#assume-role-configuration

Update github actions documentation @goruha (#606)

## what * Update github actions documentation

why

• Document the latest gitops

references

• DEV-491: Update Atmos.tools documentation for GitHub Actions to use atmos.yaml

Fix tests @goruha (#619)

## what * Fix tests * Fix documentation

why

• Because branch master renamed to main

Go auto release workflows @goruha (#586)

# What * Use go auto-release workflow `cloudposse/.github/.github/workflows/shared-go-auto-release.yml@main` * Remove `.goreleaser.yml`. Now will use https://github.com/cloudposse/.github/blob/main/.github/goreleaser.yml * Drop `auto-release.yaml`. Now will use https://github.com/cloudposse/.github/blob/main/.github/auto-release.yml and https://github.com/cloudposse/.github/blob/main/.github/auto-release-hotfix.yml

Why

• Consolidate go releases workflow pattern
• Closes #579

v1.78.0

Update `atmos validate stacks` command @aknysh (#611)

what

• Update atmos vendor pull command
• Update atmos validate stacks command
• Add --include-dependents flag to atmos describe affected command
• Update docs
  • https://atmos.tools/cli/commands/describe/affected/

why

• When executing atmos vendor pull, Atmos creates a temp directory to clone the remote repo into.
  Atmos uses go-getter to download the sources into the temp directory. When cloning from the root of a repo w/o using modules (sub-paths), go-getter does the following:

  • If the destination directory does not exist, it creates it and runs git init
  • If the destination directory exists, it should be an already initialized Git repository (otherwise an error will be thrown)

  For more details, refer to

  • hashicorp/go-getter#114
  • https://github.com/hashicorp/go-getter?tab=readme-ov-file#subdirectories

• Don't check for duplicate abstract components in the same stack from different stack manifests. Abstract components are never provisioned and serve as blueprints for real components. This is an update (follow up) to the previous PRs:

  • #608
  • #611

• The --include-dependents flag allows including dependencies for the affected components

If the command-line flag --include-dependents=true is passed to the atmos describe affected command, and there are other components that depend on the affected components in the stack, the command will include a dependents property (list) for each affected component. The dependents property is hierarchical - each component in the list will also contain a dependents property if that component has dependent components as well.

For example, suppose that we have the following configuration for the Atmos components component-1, component-2 and component-3 in the stack plat-ue2-dev:

  components:
    terraform:
      component-1:
        metadata:
          component: "terraform-component-1"
        vars: {}

      component-2:
        metadata:
          component: "terraform-component-2"
        vars: {}
        settings:
          depends_on:
            1:
              component: "component-1"

      component-3:
        metadata:
          component: "terraform-component-3"
        vars: {}
        settings:
          depends_on:
            1:
              component: "component-2"

In the above configuration, component-3 depends on component-2, whereas component-2 depends on component-1.

If all the components are affected (modified) in the current working branch, the atmos describe affected --include-dependents=true command will produce the following result:

 [
   {
     "component": "component-1",
     "stack": "plat-ue2-dev",
     "stack_slug": "plat-ue2-dev-component-1",
     "included_in_dependents": false,
     "dependents": [
       {
         "component": "component-2",
         "stack": "plat-ue2-dev",
         "stack_slug": "plat-ue2-dev-component-2",
         "dependents": [
           {
             "component": "component-3",
             "stack": "plat-ue2-dev",
             "stack_slug": "plat-ue2-dev-component-3"
           }
         ]
       }
     ]
   },
   {
     "component": "component-2",
     "stack": "plat-ue2-dev",
     "stack_slug": "plat-ue2-dev-component-2",
     "included_in_dependents": true,
     "dependents": [
       {
         "component": "component-3",
         "stack": "plat-ue2-dev",
         "stack_slug": "plat-ue2-dev-component-3"
       }
     ]
   },
   {
     "component": "component-3",
     "stack": "plat-ue2-dev",
     "stack_slug": "plat-ue2-dev-component-3",
     "included_in_dependents": true
   }
 ]

The component-1 component does not depend on any other component, and therefore it has the included_in_dependents attribute set to false. The component-2 and component-3 components depend on other components and are included in the dependents property of the other components, and hence the included_in_dependents attribute is set to true.

When processing the above output, you might decide to not plan/apply the component-2 and component-3 components since they are in the dependents property of the component-1 component. Instead, you might just trigger component-1 and then component-2 and component-3 in the order of dependencies.

v1.77.0

Update `atmos validate stacks` command @aknysh (#611)

what

• Update atmos validate stacks command
• Improve stack validation error messages

why

• When checking for misconfiguration and duplication of components in stacks, throw errors only if the duplicate component configurations in the same stack are different (this will allow importing the base default/abstract components into many stack manifest files)

• The atmos validate stacks command check the following

  • All YAML manifest files for YAML errors and inconsistencies

  • All imports: if they are configured correctly, have valid data types, and point to existing manifest files

  • Schema: if all sections in all YAML manifest files are correctly configured and have valid data types

  • Misconfiguration and duplication of components in stacks. If the same Atmos component in the same Atmos stack is defined in more than one stack manifest file, and the component configurations are different, an error message will be displayed similar to the following:

    The Atmos component 'vpc' in the stack 'plat-ue2-dev' is defined in more than one
    top-level stack manifest file: orgs/acme/plat/dev/us-east-2-extras, orgs/acme/plat/dev/us-east-2.
    
    The component configurations in the stack manifests are different.
    
    To check and compare the component configurations in the stack manifests, run the following commands:
    - atmos describe component vpc -s orgs/acme/plat/dev/us-east-2-extras
    - atmos describe component vpc -s orgs/acme/plat/dev/us-east-2
    
    You can use the '--file' flag to write the results of the above commands to files
    (refer to https://atmos.tools/cli/commands/describe/component).
    
    You can then use the Linux 'diff' command to compare the files line by line and show the differences
    (refer to https://man7.org/linux/man-pages/man1/diff.1.html)
    
    When searching for the component 'vpc' in the stack 'plat-ue2-dev', Atmos can't decide which
    stack manifest file to use to get configuration for the component. This is a stack misconfiguration.
    
    Consider the following solutions to fix the issue:
    
      - Ensure that the same instance of the Atmos 'vpc' component in the stack 'plat-ue2-dev'
        is only defined once (in one YAML stack manifest file)
      
      - When defining multiple instances of the same component in the stack,
        ensure each has a unique name
      
      - Use multiple-inheritance to combine multiple configurations together
        (refer to https://atmos.tools/core-concepts/components/inheritance)   

notes

• This is an improvement of the previous release https://github.com/cloudposse/atmos/releases/tag/v1.75.0. The previous release introduced too strict checking and disabled the case where the same component in the same stack was just imported into two or more stack manifest files (this type of configuration is acceptable since the component config is always the same in the stack manifests since it's just imported and not modified)

v1.76.0

Add Atmos manifest lists merge strategies @aknysh (#609)

what

• Add Atmos manifest lists merge strategies
• Update docs
  • Settings section in CLI Configuration

why

• Allow using the following list merge strategies in Atmos stack manifests:

  • replace - Most recent list imported wins (the default behavior).

  • append - The sequence of lists is appended in the same order as imports.

  • merge - The items in the destination list are deep-merged with the items in the source list. The items in the source list take precedence. The items are processed starting from the first up to the length of the source list (the remaining items are not processed). If the source and destination lists have the same length, all items in the destination lists are deep-merged with all items in the source list.

The list merging strategies are configured in atmos.yaml CLI config file in the settings.list_merge_strategy section

settings:
  # `list_merge_strategy` specifies how lists are merged in Atmos stack manifests.
  # Can also be set using 'ATMOS_SETTINGS_LIST_MERGE_STRATEGY' environment variable, or '--settings-list-merge-strategy' command-line argument
  # The following strategies are supported:
  # `replace`: Most recent list imported wins (the default behavior).
  # `append`:  The sequence of lists is appended in the same order as imports.
  # `merge`:   The items in the destination list are deep-merged with the items in the source list.
  #            The items in the source list take precedence.
  #            The items are processed starting from the first up to the length of the source list (the remaining items are not processed).
  #            If the source and destination lists have the same length, all items in the destination lists are
  #            deep-merged with all items in the source list.
  list_merge_strategy: replace

v1.75.0

Improve `atmos validate stacks` and `atmos describe affected` commands @aknysh (#608)

what

• Improve atmos validate stacks and atmos describe affected commands
• Update docs
  • atmos validate stacks
  • alias: Multiple Provider Configuration in Atmos Manifests

why

• atmos validate stacks now detects if the same component in the same stack are defined in more than one Atmos stack manifest files. If such a misconfiguration is detected, the following error is shown:

  the Atmos component 'vpc' in the stack 'plat-ue2-dev' is defined in more than one top-level 
  stack manifest file: orgs/acme/plat/dev/us-east-2, orgs/acme/plat/dev/us-east-2-extras.
  Atmos can't decide which stack manifest to use to get configuration for the component in the stack.
  This is a stack misconfiguration.

• atmos describe affected now has better error messages and also executes atmos validate stacks before detecting the affected components and stacks. This prevents the issue when the same component in the same stack is defined in more than one stack manifest file, and the command used one file from the current local branch and the other file from the target branch, resulting in drift in affected components and stacks. Since this is a stack misconfiguration and is not permitted, atmos describe affected will display the error and exit

• alias: Multiple Provider Configuration in Atmos Manifests

  Atmos allows you to define multiple configurations for the same provider using a list of provider blocks and the alias meta-argument.

  The generated providers_override.tf.json file will have a list of provider configurations, and Terraform will use and override the providers as long as the aliased providers are defined in the Terraform component.

  For example:

  components: 
    terraform:
      vpc:
        providers:
          aws:
            - region: us-west-2
              assume_role:
                role_arn: "role-1"
            - region: us-west-2
              alias: "account-2"
              assume_role:
                role_arn: "role-2"

Use Darker Theme, Add File Component @osterman (#607)

what

• Add a component to render files distinct from console commands
• Darken theme

why

• Previous background felt too washed out
• It's much easier to understand files and commands separately, when they are clearly distinguished.

v1.74.0

Update Atmos logs. Update docs @aknysh (#605)

what

• Update Atmos logs
• Make the logs respect the standard file descriptors like /dev/stderr
• Update docs
  • https://atmos.tools/cli/configuration/#logs

why

Atmos logs are configured in the logs section in the atmos.yaml CLI config file:

logs:
  # Can also be set using 'ATMOS_LOGS_FILE' ENV var, or '--logs-file' command-line argument
  # File or standard file descriptor to write logs to
  # Logs can be written to any file or any standard file descriptor, including `/dev/stdout`, `/dev/stderr` and `/dev/null`
  file: "/dev/stderr"
  # Supported log levels: Trace, Debug, Info, Warning, Off
  # Can also be set using 'ATMOS_LOGS_LEVEL' ENV var, or '--logs-level' command-line argument
  level: Info

• logs.file - the file to write Atmos logs to. Logs can be written to any file or any standard file descriptor, including /dev/stdout, /dev/stderr and /dev/null. If omitted, /dev/stdout will be used. The environment variable ATMOS_LOGS_FILE can also be used to specify the log file

• logs.level - Log level. Supported log levels are Trace, Debug, Info, Warning, Off. If the log level is set to Off, Atmos will not log any messages (note that this does not prevent other tools like Terraform from logging). The environment variable ATMOS_LOGS_LEVEL can also be used to specify the log level

To prevent Atmos from logging any messages (except for the outputs of the executed commands), you can do one of the following:

• Set logs.file or the ENV variable ATMOS_LOGS_FILE to /dev/null

• Set logs.level or the ENV variable ATMOS_LOGS_LEVEL to Off

Note that when you set the log level to Debug or Trace, Atmos will log additional messages before printing the output of an executed command. For example, let's consider the atmos describe affected command:

logs:
  file: "/dev/stdout"
  level: Trace

> atmos describe affected

Checking out Git ref 'refs/remotes/origin/HEAD' ...
Checked out Git ref 'refs/remotes/origin/HEAD'

Current working repo HEAD: ffd2154e1daa32357b75460b9f45d268922b51e1 refs/heads/update-logs
Remote repo HEAD: f7aa382aa8b3d48be8f06cfdb27aad344b89aff4 HEAD

Changed files:

examples/quick-start/Dockerfile
examples/quick-start/atmos.yaml

Affected components and stacks:

[
   {
      "component": "vpc",
      "component_type": "terraform",
      "component_path": "examples/quick-start/components/terraform/vpc",
      "stack": "plat-uw2-prod",
      "stack_slug": "plat-uw2-prod-vpc",
      "affected": "stack.vars"
   },
   {
      "component": "vpc",
      "component_type": "terraform",
      "component_path": "examples/quick-start/components/terraform/vpc",
      "stack": "plat-ue2-prod",
      "stack_slug": "plat-ue2-prod-vpc",
      "affected": "stack.vars"
   }
]

With logs.level: Trace, and logs.file: "/dev/stdout", all the messages and the command's JSON output will be printed to the console to the /dev/stdout standard output.

This behavior might be undesirable when you execute the command atmos describe affected in CI/CD (e.g. GitHub Actions).

For example, you might want to log all the Atmos messages (by setting logs.level: Trace) for debugging purposes, and also want to parse the JSON output of the command (e.g. by using jq) for further processing. In this case, jq will not be able to parse the JSON output because all the other messages make the output an invalid JSON document.

To deal with that, you can set logs.file to /dev/stderr in atmos.yaml:

logs:
  file: "/dev/stderr"
  level: Trace

Now when the atmos describe affected command is executed, the additional messages are printed to /dev/stderr, but the command's JSON output is printed to /dev/stdout, allowing jq to parse it without errors.

> atmos describe affected

# NOTE: These messages are printed to `/dev/stderr`

Checking out Git ref 'refs/remotes/origin/HEAD' ...
Checked out Git ref 'refs/remotes/origin/HEAD'
Current working repo HEAD: ffd2154e1daa32357b75460b9f45d268922b51e1 refs/heads/update-logs
Remote repo HEAD: f7aa382aa8b3d48be8f06cfdb27aad344b89aff4 HEAD


# NOTE: This JSON output is printed to `/dev/stdout`

[
   {
      "component": "vpc",
      "component_type": "terraform",
      "component_path": "examples/quick-start/components/terraform/vpc",
      "stack": "plat-uw2-prod",
      "stack_slug": "plat-uw2-prod-vpc",
      "affected": "stack.vars"
   },
   {
      "component": "vpc",
      "component_type": "terraform",
      "component_path": "examples/quick-start/components/terraform/vpc",
      "stack": "plat-ue2-prod",
      "stack_slug": "plat-ue2-prod-vpc",
      "affected": "stack.vars"
   }
]