Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(e2e): Define docker attributes to prevent forced replacement on second launch #4918

Merged
merged 2 commits into from
Jul 8, 2024

Conversation

moduli
Copy link
Collaborator

@moduli moduli commented Jun 28, 2024

This PR modifies some enos modules to define some docker container attributes to prevent the containers from being regenerated on a second launch/terraform apply.

Currently, after launching a docker-based scenario, a subsequent launch would result all of the docker containers in being regenerated, which is not expected if they were generated successfully the first time around. This is caused because of...

❯ enos scenario exec --cmd plan e2e_ui_docker builder:local

 # module.create_boundary.docker_container.boundary must be replaced
-/+ resource "docker_container" "boundary" {
      + bridge                                      = (known after apply)
      + container_logs                              = (known after apply)
      - cpu_shares                                  = 0 -> null
      - dns                                         = [] -> null
      - dns_opts                                    = [] -> null
      - dns_search                                  = [] -> null
      ~ entrypoint                                  = [
          - "docker-entrypoint.sh",
        ] -> (known after apply)
      + exit_code                                   = (known after apply)
      - group_add                                   = [] -> null
      ~ hostname                                    = "c5f40572b57f" -> (known after apply)
      ~ id                                          = "c5f40572b57f46c5238e1cffcd9c113568e4f8806efa2160aeb9b0c2159427c2" -> (known after apply)
      ~ init                                        = false -> (known after apply)
      ~ ipc_mode                                    = "private" -> (known after apply)
      ~ log_driver                                  = "json-file" -> (known after apply)
      - log_opts                                    = {} -> null
      - max_retry_count                             = 0 -> null
      - memory                                      = 0 -> null
      - memory_swap                                 = 0 -> null
        name                                        = "boundary"
      ~ network_data                                = [
          - {
              - gateway                   = "192.168.0.1"
              - global_ipv6_prefix_length = 0
              - ip_address                = "192.168.0.6"
              - ip_prefix_length          = 20
              - network_name              = "e2e_cluster"
                # (2 unchanged attributes hidden)
            },
        ] -> (known after apply)
      - network_mode                                = "bridge" -> null # forces replacement
      - privileged                                  = false -> null
      - publish_all_ports                           = false -> null
      ~ runtime                                     = "runc" -> (known after apply)
      ~ security_opts                               = [] -> (known after apply)
      ~ shm_size                                    = 64 -> (known after apply)
      + stop_signal                                 = (known after apply)
      ~ stop_timeout                                = 0 -> (known after apply)
      - storage_opts                                = {} -> null
      - sysctls                                     = {} -> null
      - tmpfs                                       = {} -> null

Note how the network_mode attribute going back to null forces a replacement. This is potentially an issue with the provider for trying to set all of these values to null, but the workaround here is to define network_mode = "bridge" in our modules to prevent this replacement from occurring.

This fixes an issue when an e2e test tries to retry. On the first run, we do a launch. If the test fails, the workflow tries to do a second launch, but it keeps failing because it the boundary docker container doesn't start up correctly. This is because the docker container was replaced/regenerated, but the other initialization steps of the module weren't executed (i.e. initializing the database).

moduli added 2 commits June 28, 2024 10:50
On a second `terraform apply`, there were a few attributes forcing the replacement (i.e. regeneration) of the resource. This commit defines some attributes explicitly to prevent the regeneration from occurring.
@moduli moduli added the pr/no-milestone Ignores the Milestone Check label Jun 28, 2024
@moduli moduli changed the title chore(e2e): Define docker settings to prevent forced replacement on second launch chore(e2e): Define docker attributes to prevent forced replacement on second launch Jun 28, 2024
@@ -105,7 +106,7 @@ resource "enos_local_exec" "wait" {
docker_container.openssh_server
]

inline = ["timeout 30s bash -c 'until ssh -t -t -i ${var.private_key_file_path} -p 2222 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ${var.target_user}@localhost hostname; do sleep 2; done'"]
inline = ["timeout 60s bash -c 'until ssh -t -t -i ${var.private_key_file_path} -p 2222 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ${var.target_user}@localhost hostname; do sleep 2; done'"]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also mode this modification as the startup for this openssh container can take a little longer. Longer-term, I want to find a better solution for this, but this is the quickest way for now.

@moduli moduli requested review from stasryzhov June 28, 2024 15:23
@moduli moduli marked this pull request as ready for review June 28, 2024 15:23
@moduli moduli requested a review from johanbrandhorst June 28, 2024 17:08
@moduli moduli merged commit 1909889 into main Jul 8, 2024
66 checks passed
@moduli moduli deleted the moduli-e2e-launch branch July 8, 2024 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr/no-milestone Ignores the Milestone Check
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants