API Server Erroring

[dacox]

@coderanger 👋 I know it's been a while, but I've been playing with this project this afternoon.

I've got it most of the way there, but the API server seems to always return 400 which is causing the waiter containers to crash.

Any ideas? I'm reading through the code trying to figure it out - using the main image as well.

EDIT

Ok, its obviously because my curl code has no payload.

However, the waiter container is having

"2024/08/13 23:17:14 Error while polling: Post "http://migrations-operator-api-service.migrations-operator-system.svc:5000/api/ready": dial tcp 10.28.9.169:5000: connect: connection refused"

I presume this is related to the fact that we also inject the istio proxy in our pods, but am not sure

[coderanger]

If it's redirecting the network traffic through Istio then it does sound likely that is causing the issue. I know Istio has it's own internal ACL layer for authorizing inter-service traffic. I don't use it myself so I can't really help more than pointing you to their docs.

[dacox]

I have partially resolved my problems. Istio injects an initContainer itself which does iptables stuff.

I was able to use an annotation to allow waiter traffic on port 5000.

There are still some problems with the migration Pod itself, which seems to never Terminate and instead stay in a NotReady state after completing - likely also related to the istio proxy sidecar but I'm not 100% sure how.

This can also be mitigated in the short-term by adding an annotation to disable injection.

[coderanger]

Daemon sidecars on Jobs is a long-standing problem, kubernetes/enhancements#753 should address at least this kind of simple case with it but is still in beta AFAIK.

[dacox]

thanks! yep, been waiting for that spec to be finalized for many years