HPC tool kit demo failed in second attempt!

[bethejulia]

Hi List

I have a basic question: I tried HPC Toolkit and followed 'guide me' option to build a HPC as it is defined in the tutorial page.
My First attempt was successful but after a week when I tried again the same procedure it gave ERROR messages after the commands:

./ghpc create community/examples/slurm-gcp-v5-hpc-centos7.yaml -l IGNORE --vars project_id=gcpbillcheking1 (it was OK!)
terraform -chdir=slurm-gcp-v5/primary init (OK!)
terraform -chdir=slurm-gcp-v5/primary validate (OK!)
terraform -chdir=slurm-gcp-v5/primary apply (Eneterd: yes during its execution, but it gave error)

The part of ERROR message is:

EOT
+ name = (known after apply)
+ name_pref = "slurmgcpv5-compute-debug-ghpc-"
+ project = "gcpbillcheking1"
+ region = (known after apply)
+ self_link = (known after apply)
+ tags = [
+ "slurmgcpv5",
]
+ tags_fingerprint = (known after apply)

  + advanced_machine_features {
      + enable_nested_virtualization = false
      + threads_per_core             = 1
    }

  + confidential_instance_config {
      + enable_confidential_compute = false
    }

  + disk {
      + auto_delete  = true
      + boot         = true
      + device_name  = (known after apply)
      + disk_size_gb = 50
      + disk_type    = "pd-standard"
      + interface    = (known after apply)
      + labels       = {
          + "ghpc_blueprint"      = "slurm-gcp-v5-hpc-centos7"
          + "ghpc_deployment"     = "slurm-gcp-v5"
          + "ghpc_role"           = "compute"
          + "slurm_cluster_name"  = "slurmgcpv5"
          + "slurm_instance_role" = "compute"
        }
      + mode         = (known after apply)
      + source_image = "projects/schedmd-slurm-public/global/images/family/schedmd-v5-slurm-22-05-8-hpc-centos-7"
      + type         = "PERSISTENT"
    }

  + network_interface {
      + ipv6_access_type   = (known after apply)
      + name               = (known after apply)
      + network            = (known after apply)
      + stack_type         = (known after apply)
      + subnetwork         = (known after apply)
      + subnetwork_project = (known after apply)
    }

  + scheduling {
      + automatic_restart   = true
      + on_host_maintenance = "TERMINATE"
      + preemptible         = false
      + provisioning_model  = (known after apply)
    }

  + service_account {
      + email  = "[email protected]"
      + scopes = [
          + "https://www.googleapis.com/auth/cloud-platform",
        ]
    }
}

Plan: 30 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

module.slurm_controller.module.slurm_controller_instance.random_uuid.cluster_id: Creating...
module.slurm_controller.module.slurm_controller_instance.random_uuid.cluster_id: Creation complete after 0s [id=4100620d-13f1-f573-6f1c-295827941298]
module.slurm_controller.module.slurm_controller_instance.random_string.topic_suffix: Creating...
module.slurm_login.module.slurm_login_instance.random_string.suffix: Creating...
module.homefs.random_id.resource_name_suffix: Creating...
module.homefs.random_id.resource_name_suffix: Creation complete after 0s [id=Bj989g]
module.slurm_controller.module.slurm_controller_instance.random_string.topic_suffix: Creation complete after 0s [id=qQrAi9Mu]
module.slurm_login.module.slurm_login_instance.random_string.suffix: Creation complete after 0s [id=ugd5x6pt]
module.network1.module.vpc.module.vpc.google_compute_network.network: Creating...
module.compute_partition.module.slurm_partition.google_compute_project_metadata_item.partition_startup_scripts["ghpc_startup_sh"]: Creating...
module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.slurmdbd_conf: Creating...
module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.controller_startup_scripts["ghpc_startup_sh"]: Creating...
module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.slurm_conf: Creating...
module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.cgroup_conf: Creating...
module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.compute_startup_scripts["ghpc_startup_sh"]: Creating...
module.network1.module.nat_ip_addresses["us-central1"].google_compute_address.ip[1]: Creating...
module.network1.module.nat_ip_addresses["us-central1"].google_compute_address.ip[0]: Creating...
module.debug_partition.module.slurm_partition.google_compute_project_metadata_item.partition_startup_scripts["ghpc_startup_sh"]: Creating...
module.slurm_login.module.slurm_login_instance.google_compute_project_metadata_item.login_startup_scripts["ghpc_startup_sh"]: Creating...
module.network1.module.vpc.module.vpc.google_compute_network.network: Still creating... [10s elapsed]
module.slurm_login.module.slurm_login_instance.google_compute_project_metadata_item.login_startup_scripts["ghpc_startup_sh"]: Still creating... [10s elapsed]
module.network1.module.vpc.module.vpc.google_compute_network.network: Still creating... [20s elapsed]
module.slurm_login.module.slurm_login_instance.google_compute_project_metadata_item.login_startup_scripts["ghpc_startup_sh"]: Still creating... [20s elapsed]
module.network1.module.vpc.module.vpc.google_compute_network.network: Creation complete after 23s [id=projects/gcpbillcheking1/global/networks/slurm-gcp-v5-net]
module.network1.module.cloud_router["us-central1"].google_compute_router.router: Creating...
module.network1.module.vpc.module.subnets.google_compute_subnetwork.subnetwork["us-central1/slurm-gcp-v5-primary-subnet"]: Creating...
module.network1.module.vpc.module.firewall_rules.google_compute_firewall.rules["slurm-gcp-v5-net-fw-allow-iap-ssh-ingress"]: Creating...
module.network1.module.vpc.module.firewall_rules.google_compute_firewall.rules["slurm-gcp-v5-net-fw-allow-internal-traffic"]: Creating...
module.slurm_login.module.slurm_login_instance.google_compute_project_metadata_item.login_startup_scripts["ghpc_startup_sh"]: Creation complete after 24s [id=slurmgcpv5-slurm-login_ugd5x6pt-script-ghpc_startup_sh]
module.slurm_login.module.slurm_login_instance.module.slurm_login_instance.data.local_file.startup: Reading...
module.slurm_login.module.slurm_login_instance.module.slurm_login_instance.data.local_file.startup: Read complete after 0s [id=de68d872e4df054209706dbeee9bfec9dca89970]
module.network1.module.cloud_router["us-central1"].google_compute_router.router: Creation complete after 3s [id=projects/gcpbillcheking1/regions/us-central1/routers/slurm-gcp-v5-net-router]
module.network1.module.vpc.module.subnets.google_compute_subnetwork.subnetwork["us-central1/slurm-gcp-v5-primary-subnet"]: Still creating... [10s elapsed]
module.network1.module.vpc.module.firewall_rules.google_compute_firewall.rules["slurm-gcp-v5-net-fw-allow-internal-traffic"]: Still creating... [10s elapsed]
module.network1.module.vpc.module.firewall_rules.google_compute_firewall.rules["slurm-gcp-v5-net-fw-allow-iap-ssh-ingress"]: Still creating... [10s elapsed]
module.network1.module.vpc.module.firewall_rules.google_compute_firewall.rules["slurm-gcp-v5-net-fw-allow-internal-traffic"]: Creation complete after 12s [id=projects/gcpbillcheking1/global/firewalls/slurm-gcp-v5-net-fw-allow-internal-traffic]
module.network1.module.vpc.module.firewall_rules.google_compute_firewall.rules["slurm-gcp-v5-net-fw-allow-iap-ssh-ingress"]: Creation complete after 12s [id=projects/gcpbillcheking1/global/firewalls/slurm-gcp-v5-net-fw-allow-iap-ssh-ingress]
module.network1.module.vpc.module.subnets.google_compute_subnetwork.subnetwork["us-central1/slurm-gcp-v5-primary-subnet"]: Creation complete after 15s [id=projects/gcpbillcheking1/regions/us-central1/subnetworks/slurm-gcp-v5-primary-subnet]
╷
│ Error: Error creating Address: googleapi: Error 409: The resource 'projects/gcpbillcheking1/regions/us-central1/addresses/slurm-gcp-v5-net-nat-ips-us-central1-0' already exists, alreadyExists
│
│ with module.network1.module.nat_ip_addresses["us-central1"].google_compute_address.ip[0],
│ on .terraform/modules/network1.nat_ip_addresses/main.tf line 52, in resource "google_compute_address" "ip":
│ 52: resource "google_compute_address" "ip" {
│
╵
╷
│ Error: Error creating Address: googleapi: Error 409: The resource 'projects/gcpbillcheking1/regions/us-central1/addresses/slurm-gcp-v5-net-nat-ips-us-central1-1' already exists, alreadyExists
│
│ with module.network1.module.nat_ip_addresses["us-central1"].google_compute_address.ip[1],
│ on .terraform/modules/network1.nat_ip_addresses/main.tf line 52, in resource "google_compute_address" "ip":
│ 52: resource "google_compute_address" "ip" {
│
╵
╷
│ Error: key "slurmgcpv5-slurm-partition-compute-script-ghpc_startup_sh" already present in metadata for project "gcpbillcheking1". Use terraform import to manage it with Terraform
│
│ with module.compute_partition.module.slurm_partition.google_compute_project_metadata_item.partition_startup_scripts["ghpc_startup_sh"],
│ on .terraform/modules/compute_partition.slurm_partition/terraform/slurm_cluster/modules/slurm_partition/main.tf line 168, in resource "google_compute_project_metadata_item" "partition_startup_scripts":
│ 168: resource "google_compute_project_metadata_item" "partition_startup_scripts" {
│
╵
╷
│ Error: key "slurmgcpv5-slurm-partition-debug-script-ghpc_startup_sh" already present in metadata for project "gcpbillcheking1". Use terraform import to manage it with Terraform
│
│ with module.debug_partition.module.slurm_partition.google_compute_project_metadata_item.partition_startup_scripts["ghpc_startup_sh"],
│ on .terraform/modules/debug_partition.slurm_partition/terraform/slurm_cluster/modules/slurm_partition/main.tf line 168, in resource "google_compute_project_metadata_item" "partition_startup_scripts":
│ 168: resource "google_compute_project_metadata_item" "partition_startup_scripts" {
│
╵
╷
│ Error: key "slurmgcpv5-slurm-tpl-slurm-conf" already present in metadata for project "gcpbillcheking1". Use terraform import to manage it with Terraform
│
│ with module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.slurm_conf,
│ on .terraform/modules/slurm_controller.slurm_controller_instance/terraform/slurm_cluster/modules/slurm_controller_instance/main.tf line 193, in resource "google_compute_project_metadata_item" "slurm_conf":
│ 193: resource "google_compute_project_metadata_item" "slurm_conf" {
│
╵
╷
│ Error: key "slurmgcpv5-slurm-tpl-cgroup-conf" already present in metadata for project "gcpbillcheking1". Use terraform import to manage it with Terraform
│
│ with module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.cgroup_conf,
│ on .terraform/modules/slurm_controller.slurm_controller_instance/terraform/slurm_cluster/modules/slurm_controller_instance/main.tf line 200, in resource "google_compute_project_metadata_item" "cgroup_conf":
│ 200: resource "google_compute_project_metadata_item" "cgroup_conf" {
│
╵
╷
│ Error: key "slurmgcpv5-slurm-tpl-slurmdbd-conf" already present in metadata for project "gcpbillcheking1". Use terraform import to manage it with Terraform
│
│ with module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.slurmdbd_conf,
│ on .terraform/modules/slurm_controller.slurm_controller_instance/terraform/slurm_cluster/modules/slurm_controller_instance/main.tf line 207, in resource "google_compute_project_metadata_item" "slurmdbd_conf":
│ 207: resource "google_compute_project_metadata_item" "slurmdbd_conf" {
│
╵
╷
│ Error: key "slurmgcpv5-slurm-controller-script-ghpc_startup_sh" already present in metadata for project "gcpbillcheking1". Use terraform import to manage it with Terraform
│
│ with module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.controller_startup_scripts["ghpc_startup_sh"],
│ on .terraform/modules/slurm_controller.slurm_controller_instance/terraform/slurm_cluster/modules/slurm_controller_instance/main.tf line 231, in resource "google_compute_project_metadata_item" "controller_startup_scripts":
│ 231: resource "google_compute_project_metadata_item" "controller_startup_scripts" {
│
╵
╷
│ Error: key "slurmgcpv5-slurm-compute-script-ghpc_startup_sh" already present in metadata for project "gcpbillcheking1". Use terraform import to manage it with Terraform
│
│ with module.slurm_controller.module.slurm_controller_instance.google_compute_project_metadata_item.compute_startup_scripts["ghpc_startup_sh"],
│ on .terraform/modules/slurm_controller.slurm_controller_instance/terraform/slurm_cluster/modules/slurm_controller_instance/main.tf line 243, in resource "google_compute_project_metadata_item" "compute_startup_scripts":
│ 243: resource "google_compute_project_metadata_item" "compute_startup_scripts" {

---

Please also note that I deleted the previous instances manually and ran the following command to clean it:
terraform -chdir=slurm-gcp-v5/primary destroy -auto-approve

Also before this second attempt I searched the 'slurm*' to get back any previous traces - BUT IT WAS ABSENT !

So I guess that some terraform data is missing and please help us to resolve this!
Thanks in advance

Krishna

[nick-stroud]

In general this is a sign that something was not destroyed properly from your original deployment. Calling terraform -chdir=slurm-gcp-v5/primary destroy -auto-approve is the right way to do this but it is possible that something went wrong when you called terraform destroy. For example:

• If some autoscaled compute nodes were still running when you tried to destroy the cluster they may have persisted and prevented other infrastructure from being destroyed.
• You may have called terraform destroy without the original terraform state file that was created during terraform apply. In other words, you must call terraform destroy with the same directory from which you called terraform apply. If you were to delete the deployment folder (in this case slurm-gcp-v5/primary/) and then regenerate it later, and call destroy on the regenerated folder, it would not destroy your infrastructure because the terraform state file would be missing. Note that this sort of issue can be avoided by setting up a terraform_backend.

To condense the error message, the leftover infrastructure is:

1. projects/gcpbillcheking1/regions/us-central1/addresses/slurm-gcp-v5-net-nat-ips-us-central1-0
2. projects/gcpbillcheking1/regions/us-central1/addresses/slurm-gcp-v5-net-nat-ips-us-central1-1
3. metadata key slurmgcpv5-slurm-partition-compute-script-ghpc_startup_sh
4. metadata key slurmgcpv5-slurm-partition-debug-script-ghpc_startup_sh
5. metadata key slurmgcpv5-slurm-tpl-slurm-conf
6. metadata key slurmgcpv5-slurm-tpl-cgroup-conf
7. metadata key slurmgcpv5-slurm-tpl-slurmdbd-conf
8. metadata key slurmgcpv5-slurm-controller-script-ghpc_startup_sh
9. metadata key slurmgcpv5-slurm-compute-script-ghpc_startup_sh

Items 1 & 2 can be viewed and deleted (RELEASE STATIC ADDRESS) at https://console.cloud.google.com/networking/addresses/list. I suspect you may also still have a Cloud NAT enabled that could also be deleted. You can check at https://console.cloud.google.com/net-services/nat/list.

Items 3-9 can be viewed and deleted at https://console.cloud.google.com/compute/metadata.

Unfortunately, none of the items 1-9 show up when you search cloud console, so it is understandable that you did not find them in search.

There may be more items that need to be cleaned up once these are deleted. I would suggest cleaning up the items above and then try to deploy the cluster again, if there are additional conflicts they will be listed.

Alternatively, if you are able, it can sometimes be a good practice to create a new project for test deployments so when you are done you can simply delete the project and make sure nothing is left behind. This is by no means required. When used properly, the terraform destroy command you used should clean up all of these artifacts.

[bethejulia]

Dear Nick, Indeed it is a great Help. And thanks a lot for your detailed answer...
I just deleted the project and try a new set up and it is OK!
K