brod doesn't handle new partitions added to a topic via kafka-topics --alter command

[ananthakumaran]

Producer

ok = :brod.produce_sync(:brod_client, "dummy-data", partition_fun(key), key, value)

The partition_fun callback doesn't get the updated partition count. Calling :brod_client.get_metadata(:brod_client, :undefined) seems to update the internal cache which results in the callback getting the updated partition count, but the relevant producer for the new partitions are not started. Any attempts to write to the new partition results in {:error, {:producer_not_found, topic, partition}}. Note: we run with auto_start_producers: true

Consumer Group

I was expecting a reassignment of partitions within the group subscribers, but the addition of new partition doesn't seem to have any effect on the consumer group. The lag keeps on increasing on the new partitions.

Both of the above issues go away when we restart the service.

[abejoe]

I noticed the same behavior. Any updates on this?

[k32]

Hi! Sorry for silence. We are aware of this problem (or rather two separate problems), but we haven't got any good solutions yet, other than client restart workaround.
It's definitely something to be fixed.