Why constant UNKNOWN_TOPIC_CACHE_EXPIRE_SECONDS (module brod_client) hard set 120 seconds?

[dbegunkov]

I observe a strange behavior when there is no topic in Kafka. Topic status is cached. So when it’s created there is a delay up to 120 seconds. This is because of the constant -define(UNKNOWN_TOPIC_CACHE_EXPIRE_SECONDS, 120) in brod_client module.

Is there any reason why this delay is so high? And could we make it configurable?

[zmstone]

Hi @dbegunkov
Topic re-creation was not taken into account when it was implemented.
The cache was to avoid refreshing metadata too often, also to survive temporary errors like leader unavailable errors etc.

We should probably inspect Reason at this line

brod/src/brod_client.erl

Line 861 in 6966f02

[{_, {error, Reason}, Ts}] ->

and expire immediately if it's unknown_topic_or_partiton and expire immediately.

[dbegunkov]

Yes - this is also a good option, it seems to me