AWS ingress

[bprashanth]

Currently people just stick a Service of Type=LB in front of the nginx ingress on AWS. Apparently, this causes some issues, and an ALB works better (sky-uk/feed#111). Maybe we should actually write an AWS ingress controller that spins up an ALB so people can tier it over the nginx ingress key-ing off ingress.class=aws/nginx ?

ref kubernetes-retired/contrib#346
@jsravn @justinsb

[pieterlange]

ALB while very scalable is also still quite limited.

Plain old TCP ELB with proxyprotocol (full handoff to nginx) seems to work fine for me in AWS.

[justinsb]

I'm sure we'll end up supporting ALB eventually. But ... what are the issues that ALB solves over ELB + ingress?

[jsravn]

ELBs will return 504s occasionally when nginx does a reload. ALBs don't have this problem.

Number of 504s correlates to the load on the ELB - it isn't many (probably less than 1% even under heavy load), but it happens. Even with TCP ELB it happens - you get a reset connection.

Obviously its worse the more often a reload happens, so busy environments will see this happen more often than very static ones which probably won't see it at all or extremely rarely.

[pieterlange]

ELBs will return 504s occasionally when nginx does a reload. ALBs don't have this problem.

When running in tcp proxyprotocol mode this doesn't happen. A more generic problem with using nginx specifically is it resets the loadbalancing counters on config reloads, causing more load on the "first" pod in the backend for a service. That means you can run into cascading errors on busy sites as the "first" pod dies due to the load, causing a new config loads, etc.

[justinsb]

Do we know why nginx reload is being surfaced as a 504 as well? I was under the impression that nginx didn't drop connections during a HUP. Is it a matter of decreasing the ELB timeout so that it is less than some nginx timeout?

[aledbf]

@jsravn @pieterlange could you provide an example/guide that reproduces the described error?

[donaldguy]

ALBs now have host-based routing https://aws.amazon.com/blogs/aws/new-host-based-routing-support-for-aws-application-load-balancers/

so they seem a yet better fit for Ingress; not sure whether they support multi-cert-TLS termination

EDIT: They definitely support at least a cert, so a star-cert might be good enough (it would for my company's major use case)

[donaldguy]

ALBs also (same blog post) now support 75 routes per balancer, rather than just 10

[aledbf]

Closing. Current catalog contains a link to an ALB ingress controller.