support "variables" key to fetch TF_VAR_* from env or envFrom

[dan1el-k]

Proposal

It would be nice to have a separate key "variables" to fetch TF_VAR_ either inline using env: for via envFrom:. There exists already a similar options for secrets, the "credentials" key, which allows to fetch "TF_VAR_" from secrets.
Environment variables from "variables" key should just be added to the terraform tasks: "init, plan, apply, init-delete, plan-delete, apply-delete" without removing the prefix.

  # proposal
  variables:
    - env:
      - name: TF_VAR_s3_bucket
        value: test-bucket
    - envFrom:
      - configMapRef:
          name: terraform-variables
  # existing
  credentials:
    - secretNameRef:
        name: aws-credentials
        key: TF_VAR_AWS_ACCESS_KEY_ID
    - secretNameRef:
        name: aws-credentials
        key: TF_VAR_AWS_SECRET_ACCESS_KEY

Motiviation

I know, that "env" and "envFrom" options already exists as part of the "taskOptions" key, but here is the challenge of it:

I am applying this terraform operator for multiple teams on multiple clusters behind a company proxy and air-gapped. It means we need a lot of settings to be set default (e.g. different image, task scripts read from configmap, SCM credentials,..). In order to not duplicate this information in every "terraform CR", I added all default settings directly in the CRD as a default.

However, the taskOption key can't be defaulted, if we wanted to also use it for defining TF_VAR variables for just the terraform tasks, right.
So I would need to always define this full block in the terraform CR, for just defining the "s3_bucket terraform variable"

taskOptions:
  - for:
    - '*' # The following config affects all task pods
    env:
    - name: TF_VAR_s3_bucket
      value: test-bucket

  # Read more about the available tasks and default scripts here: https://github.com/GalleyBytes/terraform-operator-tasks
  - for:
    - 'setup'
    - 'setup-delete'
    script:
      configMapSelector:
        name: terraform-setup-script
        key: setup.sh
  - for:
    - 'init'
    - 'plan'
    - 'apply'
    - 'init-delete'
    - 'plan-delete'
    - 'apply-delete'
    script:
      configMapSelector:
        name: terraform-setup-script
        key: tf.sh

With the proposal above however, we could still default the scripts used in the tasks and additionally "TF_VAR variables" could be defined separately by the end user.
This

Version used

I am using the release "0.9.0-pre3" with CRD "v1alpha2".

[isaaguilar]

Hi @dan1el-k Great analysis of the variables constructs. I honestly hadn't even considered how credentials' secretNameRef could be used as a general env since my tunnel vision focused on the credentials problem lol. Everything you said is absolutely right, especially the pain of needing to redefine all of taskOptions if even one item is different. (For that, a bit of helm templating voodoo is required... I know that because I've had to do it in the past for similar complex arrays.)

So regarding adding variables, it's a good idea. The only change I'd make to the proposal is to add the env to all the task pods instead of targeting the terraform tasks; mostly for simplicity. What do you think?

[dan1el-k]

To follow up on your question.
Yes, I also wouldn't see any problem adding the envs to all task pods and not only the terraform ones.
So agree with your proposal.