Account Customizations and Shared Modules Issue (Terraform Clould Only)

[jpablos11]

Terraform Version & Prov:

AFT Version:
1.13.1

Bug Description
The examples describe a particular scenario where a "modules" folder can be defined at the root of the "aft-account-customizations" repository to define reusable pieces across accounts:
https://github.com/aws-ia/terraform-aws-control_tower_account_factory/blob/main/examples/multiple-account-customizations/account-customization-dev/terraform/main.tf

This seems to work OK for Terraform OSS but not for Terraform Cloud, and this is because the customizations pipeline (CodePipeline pipeline generated for the concrete account customizations) zips the contents of the "terraform" folder only, so the "modules" level does not make it to Terraform Cloud causing an error; relevant section of the pipeline code, Terraform Cloud part:

cd $DEFAULT_PATH/$CUSTOMIZATION
tar -czf temp_configuration_file.tar.gz -C terraform --exclude .git --exclude venv .
python3 $DEFAULT_PATH/aws-aft-core-framework/sources/scripts/workspace_manager.py --operation "deploy ..."

vs. Terraform OSS:

if [ $TF_DISTRIBUTION = "oss" ]; then
   ...
    cd $DEFAULT_PATH/$CUSTOMIZATION/terraform
    export AWS_PROFILE=aft-management-admin
    /opt/aft/bin/terraform init -no-color
    /opt/aft/bin/terraform apply -no-color --auto-approve

To Reproduce
Steps to reproduce the behavior:

1. Deploy AFT 1.13.1 and configure it to use and keep state in Terraform Cloud (provide token, etc.)
2. Define an account customization that references a shared module in "modules" (root of its repo like in the provided example: https://github.com/aws-ia/terraform-aws-control_tower_account_factory/blob/main/examples/multiple-account-customizations/account-customization-dev/terraform/main.tf)

Expected behavior
Account customization should succeed, instead, see Terraform output below

Related Logs
Output in Terraform Cloud for the [ACCOUNT_ID]-aft-account-customizations workspace:

Initializing Terraform Cloud...
Initializing modules...

• iam_role_terraform_web_identity in
  ╷
  │ Error: Unreadable module directory
  │
  │ Unable to evaluate directory symlink: lstat ../../modules: no such file or
  │ directory
  ╵

╷
│ Error: Unreadable module directory
│
│ The directory could not be read for module
│ "my-module" at main.tf:1.

Additional context
For Terraform Cloud to reference modules in directories higher in the hierarchy where Terraform is executed, the "Terraform Working Directory" Workspace Setting also needs to be set to the subdirectory where the Terraform code is, this was correctly set to "account-customization-accountx/terraform" but it failed (both with and without the value).

Reviewing the code of the CodePipeline pipeline generated can be observed that just the direct "terraform" folder is being zipped and sent to Terraform Cloud, in which case, the fix would be to zip the whole repository, or at least the "modules" folder on the root level + the account customization folder

[sk-at-amazon]

Thanks for reaching out. We’ll add this to our issues backlog.

[wellsiau-aws]

@jpablos11 , good point, yes with TFC we have option to specify the working directory.

that said, have you consider to use the private registry ?

[jpablos11]

@sk-at-amazon , thank you for the acknowledgement.

@wellsiau-aws , thank you for the suggestion, I did thought about putting the modules in our private registry, but most reusable pieces are too simple to justify having their own repo/lifecycle outside of AFT.

A good example is a GitHub actions role we only need in Workloads accounts but nowhere else, it only has a couple of resources but enough for us to try to keep things DRY and consistent with minimal effort. e.g.

modules/role-for-workloads-accts 
account-workloads-a/[import here]
account-workloads-b/[import here]
account-other-c
account-other-d

We have other little modules/pieces like the one above; for us it made sense to make a quick-and-dirty patch while a fix is released, we forked the repo and added a line to copy the "modules" folder into the customization folder before the tar step:
main...labinhood:forks_terraform-aws-control_tower_account_factory:main

Our temp fix does not have the same directory structure within the tar file as the source files, but it allowed us to get things working without having to set Working Directory and deeper changes (which the right fix might) - it does the trick for now and it will be easy enough to update once a fix is released.

Thank you guys!