Provider cache server was unable to process the received request, code=400

[mwos-sl]

Describe the bug

When running terragrunt 0.69.0 and some older terraform version (I saw this for tf 0.13.5, 0.14.3, 1.1.2, 1.1.9) I'm getting the error on terragrunt plan:

21:17:41.807 INFO   Terragrunt Cache server is listening on 127.0.0.1:36757
21:17:41.809 INFO   Start Terragrunt Cache server
21:18:21.400 INFO   Downloading Terraform configurations from git::ssh://[email protected]/<anonymised> into ./.terragrunt-cache/gK-KBwohpR10ufYxkwT9yO68-HI/Oc7GmR7V1D-J0aE1XreS8HCGi24
21:18:23.319 INFO   Caching terraform providers for ./.terragrunt-cache/gK-KBwohpR10ufYxkwT9yO68-HI/Oc7GmR7V1D-J0aE1XreS8HCGi24/modules/base
21:18:24.638 ERROR  Cache server was unable to process the received request, code=400, message=missing key in request header

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: 400
Bad Request

21:18:24.647 ERROR  terraform invocation failed in ./.terragrunt-cache/gK-KBwohpR10ufYxkwT9yO68-HI/Oc7GmR7V1D-J0aE1XreS8HCGi24/modules/base
21:18:24.647 INFO   Shutting down Terragrunt Cache server...
21:18:24.647 INFO   Terragrunt Cache server stopped
21:18:24.647 ERROR  error occurred:

* Failed to execute "terraform init -backend-config=region=us-west-2 -backend-config=bucket=<anonymised> -backend-config=dynamodb_table=infrastructure-live-infrastructure -backend-config=encrypt=true -backend-config=key=<anonymised>/global/terraform.tfstate -no-color -no-color" in ./.terragrunt-cache/gK-KBwohpR10ufYxkwT9yO68-HI/Oc7GmR7V1D-J0aE1XreS8HCGi24/modules/base
  
  Error: Failed to query available provider packages
  
  Could not retrieve the list of available versions for provider hashicorp/aws:
  could not query provider registry for registry.terraform.io/hashicorp/aws: 400
  Bad Request
  
  
  exit status 1

The problem is NOT there when using terraform versions: 1.2.0, 1.2.9, 1.3.0, 1.3.10, 1.5.7.

I haven't tested other versions of terraform, but based on these, it seems that breaking point is between 1.1.9 and 1.2.0 terraform releases, so maybe something along the lines: https://github.com/hashicorp/terraform/blob/v1.2/CHANGELOG.md#120-may-18-2022

Steps To Reproduce

Steps to reproduce the behavior:

1. Enable terragrunt provider cache.
2. run terragrunt plan on whatever project with terraform 1.1.2 (other terraform might be also affected - the list of OK and not OK that I found so far is in the previous point)

Expected behavior

Terragrunt does not fail when running some older terraform.

Versions

• Terragrunt version: 0.69.0
• Terraform version: <= 1.1.9
• Environment details: Ubuntu 24.04

Additional context

I'm getting this error both from laptop and from container running atlantis.

[mwos-sl]

FYI @levkohimins

[mwos-sl]

EDIT: Updated original message with tf version breaking point.

[yhakbar]

Hey @mwos-sl ,

I'm sorry you're running into this. This seems to be a pretty rare problem, so we would have trouble providing a fix without a good way to reproduce the issue.

Do you have a fixture that can reliably surface the problem on a more modern version of OpenTofu/Terraform? If the fix for users is to update their version of OpenTofu/Terraform, then we really don't have anything we need to do in Terragrunt. We would need either a public repo, or a fork of the Terragrunt repo with a new fixture added, and a test that reliably fails.

We have other work that we're prioritizing now, so we might not be able to get to this problem promptly, even after being provided that, just so that expectations are communicated correctly.

[mwos-sl]

@yhakbar I'm not sure if I understand correctly what do you mean by:

Do you have a fixture that can reliably surface the problem

Can you elaborate?
In the meantime - not sure if that answers, but the last terraform version where the described problem is always there is terraform 1.1.9. You can create whatever simple terragrunt project using terraform 1.1.9 with aws provider and cache enabled, and the problem is there.

[yhakbar]

Hey @mwos-sl ,

As you can see here:
https://terragrunt.gruntwork.io/docs/reference/supported-versions/

We don't aim to have every version of Terragrunt be perfectly compatible with every version of OpenTofu/Terraform.

Given that we're currently testing against 1.9.x in the latest versions of Terragrunt, we would need to see reproducible issues there to create new tests and fixes to address the issue.

Given that the version of Terraform you're running into issues with is pretty old, and you know it's resolved in a later version, why aren't you considering upgrading to resolve the issue?