Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore retries to default, since idempotency is guaranteed #1227

Open
wants to merge 2 commits into
base: series/3.x
Choose a base branch
from

Conversation

soujiro32167
Copy link

@soujiro32167 soujiro32167 commented Jul 18, 2023

The goal is to guarantee ordering, while preserving defaults as much as possible.

From fs2 docs:

max.retries is set to 0, to avoid the risk of records being produced out-of-order. If we don't need to produce records in-order, then this can be set to some positive integer value. An alternative is to enable retries and use withMaxInFlightRequestsPerConnection(1) or withEnableIdempotence(true). The blog post Does Kafka really guarantee the order of messages? provides more detail on this topic.

Consider the following defaults:

  1. retries: Int.MaxValue
  2. enable.idempotence: true (if retries > 0 and max.in.flight.requests.per.connection <= 5)
  3. max.in.flight.requests.per.connection: 5

Since idempotance is enabled by default, the scenario where records arrive out of order is prevented.
Therefore, we can eliminate setting retries to 0

@aartigao
Copy link
Contributor

I'm concerned that this could break some existing codebases.

Imagine an scenario where users are aware of fs2-kafka not retrying on errors, and it's fine for them. Most probably the error handler and potential retry logic is set in user land. If we merge this change, we might broke that code.

TBH, I'm not sure how to proceed with this, because even if nothing changes at the API level, it does at runtime. I'm not sure if it's fine to accept this change in a minor version and just mention it in the release notes, or if this deserves a major release.

I'll let the more experienced people to decide on that @mpilquist

@aartigao aartigao added this to the v4.0.0 milestone Oct 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants