[DevX-183] Add rebalancing logic to kafka consumers

[ish-bindra]

Node-rdKafka won't handle rebalancing state and leaves it to lib-rdkafka to do it.

There are issues we are observing with this approach: Link to logs where consumers are left in a Zombie state and they don't process anything even though they are connected to Kafka

Other solutions rebalance on specific error codes: tulios/kafkajs#1474

This PR:

• Adds rebalancing callback to assign and unassign partitions on the consumer
• Adds incremental assign and unassign rebalance protocol support. This replaces EAGER rebalancing approach which is STOP THE WORLD rebalancing
• Makes incremental rebalancing default for consumers
• Upgrades node-rdkafka to 3.2.1

More reading:

• Cooperative Rebalance Blizzard/node-rdkafka#1081
• https://github.com/apache/kafka/blob/2.6.1/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java
• consumer stream hangs and node process must be restarted Blizzard/node-rdkafka#887

[changeset-bot]

🦋 Changeset detected

Latest commit: 0ad457e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 6 packages

Name	Type
@steveojs/storage-postgres	Major
steveo	Major
@steveojs/datadog	Major
@steveojs/newrelic	Major
@steveojs/sentry	Major
@steveojs/statsd	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR