cbdinocluster remove-all fails on namespaces

[ingenthr]

These were all created with dinocluster, but it cannot delete them. I'll look further into it later, but I thought it might be useful to at least file an issue.

➜  cbdinocluster git:(master) ✗ ./cbdinocluster remove-all
2024-10-17T14:06:52.904-0700	INFO	logger initialized
2024-10-17T14:06:52.906-0700	INFO	identified available deployers	{"deployers": ["docker", "cao"]}
2024-10-17T14:06:52.906-0700	INFO	removing all clusters	{"deployer": "docker"}
2024-10-17T14:06:52.911-0700	INFO	removing all clusters	{"deployer": "cao"}
2024-10-17T14:06:52.920-0700	INFO	deleting namespace	{"namespace": "cbdc2-d772038a112a4add9ca2daa6000261cf"}
2024-10-17T14:06:52.921-0700	INFO	deleting namespace	{"namespace": "cbdc2-eecc8d6cffb24cefaf60f8d15a282633"}
2024-10-17T14:06:52.923-0700	INFO	waiting for namespaces to disappear
2024-10-17T14:07:53.156-0700	FATAL	failed to remove all clusters	{"error": "failed delete namespaces: failed to wait for namespace deletion: timed out waiting for condition", "errorVerbose": "timed out waiting for condition\ngithub.com/couchbaselabs/cbdinocluster/utils/caocontrol.waitForFunc\n\t/Users/ingenthr/src/cbdinocluster/utils/caocontrol/controller.go:112\ngithub.com/couchbaselabs/cbdinocluster/utils/caocontrol.(*Controller).DeleteNamespaces\n\t/Users/ingenthr/src/cbdinocluster/utils/caocontrol/controller.go:435\ngithub.com/couchbaselabs/cbdinocluster/deployment/caodeploy.(*Deployer).RemoveAll\n\t/Users/ingenthr/src/cbdinocluster/deployment/caodeploy/deployer.go:387\ngithub.com/couchbaselabs/cbdinocluster/cmd.init.func53\n\t/Users/ingenthr/src/cbdinocluster/cmd/removeall.go:32\ngithub.com/spf13/cobra.(*Command).execute\n\t/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:944\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:992\ngithub.com/couchbaselabs/cbdinocluster/cmd.Execute\n\t/Users/ingenthr/src/cbdinocluster/cmd/root.go:16\nmain.main\n\t/Users/ingenthr/src/cbdinocluster/main.go:8\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.1/libexec/src/runtime/proc.go:272\nruntime.goexit\n\t/opt/homebrew/Cellar/go/1.23.1/libexec/src/runtime/asm_arm64.s:1223\nfailed to wait for namespace deletion\ngithub.com/couchbaselabs/cbdinocluster/utils/caocontrol.(*Controller).DeleteNamespaces\n\t/Users/ingenthr/src/cbdinocluster/utils/caocontrol/controller.go:455\ngithub.com/couchbaselabs/cbdinocluster/deployment/caodeploy.(*Deployer).RemoveAll\n\t/Users/ingenthr/src/cbdinocluster/deployment/caodeploy/deployer.go:387\ngithub.com/couchbaselabs/cbdinocluster/cmd.init.func53\n\t/Users/ingenthr/src/cbdinocluster/cmd/removeall.go:32\ngithub.com/spf13/cobra.(*Command).execute\n\t/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:944\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:992\ngithub.com/couchbaselabs/cbdinocluster/cmd.Execute\n\t/Users/ingenthr/src/cbdinocluster/cmd/root.go:16\nmain.main\n\t/Users/ingenthr/src/cbdinocluster/main.go:8\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.1/libexec/src/runtime/proc.go:272\nruntime.goexit\n\t/opt/homebrew/Cellar/go/1.23.1/libexec/src/runtime/asm_arm64.s:1223\nfailed delete namespaces\ngithub.com/couchbaselabs/cbdinocluster/deployment/caodeploy.(*Deployer).RemoveAll\n\t/Users/ingenthr/src/cbdinocluster/deployment/caodeploy/deployer.go:389\ngithub.com/couchbaselabs/cbdinocluster/cmd.init.func53\n\t/Users/ingenthr/src/cbdinocluster/cmd/removeall.go:32\ngithub.com/spf13/cobra.(*Command).execute\n\t/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:944\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:992\ngithub.com/couchbaselabs/cbdinocluster/cmd.Execute\n\t/Users/ingenthr/src/cbdinocluster/cmd/root.go:16\nmain.main\n\t/Users/ingenthr/src/cbdinocluster/main.go:8\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.1/libexec/src/runtime/proc.go:272\nruntime.goexit\n\t/opt/homebrew/Cellar/go/1.23.1/libexec/src/runtime/asm_arm64.s:1223"}
github.com/couchbaselabs/cbdinocluster/cmd.init.func53
	/Users/ingenthr/src/cbdinocluster/cmd/removeall.go:34
github.com/spf13/cobra.(*Command).execute
	/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:944
github.com/spf13/cobra.(*Command).ExecuteC
	/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068
github.com/spf13/cobra.(*Command).Execute
	/Users/ingenthr/go/pkg/mod/github.com/spf13/[email protected]/command.go:992
github.com/couchbaselabs/cbdinocluster/cmd.Execute
	/Users/ingenthr/src/cbdinocluster/cmd/root.go:16
main.main
	/Users/ingenthr/src/cbdinocluster/main.go:8
runtime.main
	/opt/homebrew/Cellar/go/1.23.1/libexec/src/runtime/proc.go:272
➜  cbdinocluster git:(master) ✗ git describe
v0.0.43-24-g31316aa

[brett19]

It would be helpful to see the kubernetes logs to understand why it's failing to destroy the namespace. Perhaps that namespace has some form of resource that has a 'destructor' that is getting stuck?