test(e2e): add in chaos E2E test suite

[charliedmcb]

Fixes #216

Description

How was this change tested?
9/10 successful runs required as the benchmark for adding a new test suite

Last 10 runs (10/10 successful):

• https://github.com/Azure/karpenter-provider-azure/actions/runs/8365250389
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8362568356
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8362566552
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8362565033
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8351464977
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8351464441
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8351463743
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8350864118
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8350862656
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8350860961

Previous 10 runs (8/10 successful):

• https://github.com/Azure/karpenter-provider-azure/actions/runs/8237295256 (Passing)
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8237293774 (Passing)
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8237291970 (Passing)
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8239311793/job/22532260269 (Failed)
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8239310691 (Passing)
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8239309385 (Passing)
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8239308109 (Passing)
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8240306812/job/22535476433 (Passing)
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8240305331/job/22535478179 (Failed)
• https://github.com/Azure/karpenter-provider-azure/actions/runs/8240304174/job/22535468040 (Passing)

Older passing E2E test:
https://github.com/Azure/karpenter/actions/runs/7202151191

Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: #
• No

Release Note

NONE

[charliedmcb]

/test

[coveralls]

Pull Request Test Coverage Report for Build 8427608579

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 97.716%

---

Totals
Change from base Build 8397094800:	0.0%
Covered Lines:	35591
Relevant Lines:	36423

---

💛 - Coveralls

[charliedmcb]

/test

[Bryce-Soghigian]

Can we do 5 runs showing its not flakey

[charliedmcb]

/test

[charliedmcb]

Can we do 5 runs showing its not flakey

My current benchmark is 9/10

[rakechill]

Have a few questions to understand the goals of these tests

[Bryce-Soghigian]

Have a few questions to understand the goals of these tests

Look at the original pr for the chaos suite maybe? This might give us better insights as to the why behind these tests.

@jonathan-innis any insights as I see you authored the original pr

[rakechill]

The extra information from the original PR:

constantly tainting a node after creation

makes sense why we'd get > 1 node for a 1 pod deployment.

35 still feels like a magic number to me though :)

[rakechill]

Also found this comment interesting for our consideration:

It's worth noting that without enabling either ttlSecondsAfterEmpty or consolidation.enabled, this issue persists for nodes that would receive taints that Karpenter is unaware of after node launch

[charliedmcb]

35 still feels like a magic number to me though :)

Yea, I'm not sure on the 35. Agree seems magic number-ish

[rakechill]

Checked with the AWS folks, and the < 35 nodes check was added due to testing as a typical upper limit with disruption. This is good enough for me right now, so I'll approve for now. Moreso for posterity's sake in case we want to adjust this limit in the future and how we might reason about doing so.