A more expressive way to define assertions for multiple input variables or resources

[JonathonAnderson]

Terraform Version

Terraform v1.9.8

Use Cases

I am doing some complex validation on inputs that don't fit neatly into the module declarations that are depending on the input. I'm essentially creating nested "for" loops iterating over resources of different types that will be fed into different modules to determine if the combined set is suitable for deployment. The validation doesn't belong to either set individually, and applying it to both doesn't seem like an elegant solution because "Do Not Repeat Yourself". The "terraform_data" resource works nicely for validation, but I don't want to add data to my state file. When I try to change the type from "resource" to "ephemeral" I receive an error:

│ Error: Unsupported block type
│ 
│   on path\to\module\main.tf line 6:
│    6: ephemeral terraform_data validation {
│ 
│ Blocks of type "ephemeral" are not expected here.

Attempted Solutions

Attempted to change resource type for "terraform_data" from "resource" to "ephemeral"

ephemeral terraform_data validation {
  for_each = { complex nested for loops }

  lifecycle {
    precondition {
      condition     = upper(each.value.one) != upper(each.value.two)
      error_message = "Input values must be unique"
    }
  }
}

Proposal

Enable "terraform_data" to be an "ephemeral" resource

References

No response

[jbardin]

Hi @JonathonAnderson,

The purpose of terraform_data is to provide a generic resource which implements a full managed resource lifecycle. That itself is contradictory with ephemeral, since an ephemeral resource cannot store any data. Furthermore, overloading managed resource preconditions or postconditions for unrelated variable validations doesn't seem any more consistent than using variable validation blocks which refer to multiple variables. In fact cross-variable references were added for this use case.

It sounds like you are asking for a different way to express variable validations, aside from the existing check assertions and variable validation. I don't know if another way to express the validations is a good idea or not, but maybe something like mandatory check assertions which can cause the plan or apply to fail?

[JonathonAnderson]

Thanks @jbardin,

I was unaware of the check block and validation block.

check looks like it almost does what I need, except for not supporting the for_each meta argument, and I'd like a failed condition to block plan and apply. It may be interesting if the check block could be referenced in depends_on arguments in other resources

validation looks interesting, but again not sure how to iterate over lists

Since my goal is to check for duplicates in two separate lists, I would need the ability to iterate

[jbardin]

Thanks @JonathonAnderson,

Anything that can be expressed through a resource's for_each can also be generated through another expression. Usually for duplicates is common to take advantage of distinct for single lists, or setintersection for multiple sets. More complex expressions can be built up from multiple for expressions in other local values.

I don't think there's any reason to tie this to terraform_data though, as it seems you are searching for a more expressive way to generate assertions rather needing an ephemeral resource.

[JonathonAnderson]

@jbardin I noticed setintersection before and I think it's the closest to what I'm looking for, but it would need the ability to match on a specific key/attribute in a map/object.

Also, postcondition and precondition on modules would help a lot

[JonathonAnderson]

@jbardin

This is a bit tangential to the original problem because I'm only comparing a single resource type, and my original problem was comparing two different resource types. To give you a better picture of the direction I'm going, here's the code I worked out for the simpler problem

locals {
  groups = concat(local.default-groups, var.entra.groups)
}

resource terraform_data validate-unique-group-names {
  for_each =  { for names in 
                flatten([ for index, group1 in local.groups : 
                  [ for group2 in slice(local.groups, index + 1, length(local.groups)) :
                    {
                      group1-name = group1.display-name
                      group2-name = group2.display-name
                    }
                  ]
                ]) : format("%s :: %s", names.group1-name, names.group2-name) => names
              }

  lifecycle {
    precondition {
      condition     = upper(each.value.group1-name) != upper(each.value.group2-name)
      error_message = "Groups must have unique display names"
    }
  }
}

This produces an informative error message that directs the user to exactly which group is violating the constraint

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Resource precondition failed
│ 
│   on company\azure-tenant\entra\main.tf line 45, in resource "terraform_data" "validate-unique-group-names":
│   45:       condition     = upper(each.value.group1-name) != upper(each.value.group2-name)
│     ├────────────────
│     │ each.value.group1-name is "Test22"
│     │ each.value.group2-name is "Test22"
│ 
│ Groups must have unique display names

By contrast, when using the variable validation

variable entra {
  type = object({
    groups = list(object({
      display-name = string
    }))

  validation {
    condition     = length(distinct(concat(local.default-groups.*.display-name, var.entra.groups.*.display-name))) == length(concat(local.default-groups.*.display-name, var.entra.groups.*.display-name))
    error_message = "All groups must have a unique display name"
  }

The error message doesn't contain any specific values that point the user to the group that needs to be corrected

╷
│ Error: Invalid value for variable
│ 
│   on company\azure-tenant\main.tf line 5, in module "entra":
│    5:   entra = var.azure-tenant.entra
│     ├────────────────
│     │ local.default-groups is empty tuple
│     │ var.entra.groups is list of object with 2 elements
│ 
│ All groups must have a unique display name
│ 
│ This was checked by the validation rule at company\azure-tenant\entra\parameters.tf:13,3-13.

The drawback to the first method, using terraform_data, is polluting the state file when all precondition checks pass

The drawback to the second method, using validation, is loss of information in the error message

[jbardin]

Thanks @JonathonAnderson, that's what I had in mind here. You are essentially using the terraform_data resource to dynamically generate preconditions, rather than having to manually write out individual validations. Granted this might not be the best example; it looks like it's going to be pretty fragile and highly dependent on order and alignment of the input data, but I think the idea for easier bulk validations is worth exploring.

[JonathonAnderson]

@jbardin could you explain fragile and dependent on order?

[jbardin]

@JonathonAnderson, OK it's not as much of a problem as I thought at first glance, I originally read that as matching items between lists by index, but rather it's just creating all the combinations of items. It's still not going to reach the validation if there is a duplicate though, since you can't create a map from a for expression with duplicate keys, so you will get an error like Two different items produced the key .... It's not going to let your invalid values through which is OK, but you're not going to get the error_message. The expression itself though is probably as good a validation as you can expect right now, since it will at least output the names.group1-name and names.group2-name values.

[JonathonAnderson]

@jbardin I was running the Two different items produced the same key... originally but taking a slice that represents everything after the current index of the outer loop solved it, but I realized later that only solves the problem for the case of only 2 of the same items. You are right, though, for a case of 3 or more of the same items. I think I can solve that like this

resource terraform_data validate-unique-group-names {
  for_each =  { for names in 
                flatten([ for index0, group1 in local.groups : 
                  [ for index1, group2 in slice(local.groups, index0 + 1, length(local.groups)) :
                    {
                      group1-name = group1.display-name
                      group2-name = group2.display-name
                    }
                  ]
                ]) : format("%s :: %s :: %s", names.group1-name, names.group2-name, index1) => names
              }

  lifecycle {
    precondition {
      condition     = upper(each.value.group1-name) != upper(each.value.group2-name)
      error_message = "Groups must have unique display names"
    }
  }
}

Basically, add an index to the inner loop and append it to the format string for the key. That way if the same item is present at position 3, 7, and 11, they key will have a unique value for each combination of items

[JonathonAnderson]

@jbardin I'm beating a dead horse a bit here, so I apologize, but the above solution also only works for identical lists. A more generic solution for any two lists would be more like this

resource terraform_data validate-unique-group-names {
  for_each =  { for names in 
                flatten([ for index0, group0 in local.groups0 : 
                  [ for index1, group1 in local.groups1 :
                    {
                      group0-name = group0.display-name
                      group1-name = group1.display-name
                    }
                  ]
                ]) : format("%s :: %s :: %s :: %s", names.group0-name, index0, names.group1-name, index1) => names
              }

  lifecycle {
    precondition {
      condition     = upper(each.value.group0-name) != upper(each.value.group1-name)
      error_message = "Groups must have unique display names"
    }
  }
}