WIP chore(aws-provider): aws tags batching functionality

[ivankatliarchuk]

Description

No issue to day

Multi tenant platform, with dozens of customers, clients of random size and different requirements.

Some of them have route53 setups with 100+, 200-300 zones in accounts. We have no control over things like split zones or similar stuff.

So what is going on. Filter by tag is a great feature, as it helps to attach metadata to workloads of different types and etc. At the moment the logic

external-dns/provider/aws/aws.go

Line 948 in 8eb8ea3

response, err := client.ListTagsForResource(ctx, &route53.ListTagsForResourceInput{

does one request per zone, it only works for external-dns when the size is ~ 20-40 zones. More important, as we have some controls and platform customers build other tooling that rely on Route53 API, they all throttled as well. So we have outages, as a result the external-dns need to be stopped time-to-time.

At the moment with current implementation, the external-dns not even able to fetch this zones data, cachind does not help either, as zones are not in the cache. external-dns do not crash, but zones list is empty, hence no records to update

Rate limit error on UI when external-dns refreshing a zone list

Current implementation

With batching API in use

Why there is a difference from ~30 seconds FAIL and occasionally SUCCESS vs ~ 6 seconds SUCCESS

• Current implementation makes one-to-one call per zone
• aws-sdk-go-v2 has default retry 3x times with back-off, so before to fail, it tries 3x times before to return a soft error. This ads-up to API rate limits and timings
• Forthe size of 160 zones, current implementation does 160 calls, while with batching there is only 17 requests

** Follow up **

• I'll create a proposal to implement a global rate limiter for providers as well. Not sure if I'm going to have time to implement, but at least we could get a consensus around required/nice-to-have API rate limiting for providers as abusing 3rd Party API is not the great idea, instead this should be respected

Checklist

• Unit tests updated
• End user documentation updated

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign raffo for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment