Allow setting keys in outbox messages

[mgibowski]

The problem

At the moment, published messages have always an empty key:

kafkaesque/lib/kafkaesque/clients/brod_client.ex

Line 42 in 2c43100

case :brod.produce_sync(client_id, topic, partition, "", values) do

This prevents key-based retention via topic compaction:
https://developer.confluent.io/courses/architecture/compaction/#topic-compaction-key-based-retention

The solution

That would be a breaking change, but I think the MyApp.Outbox.publish/2 function should accept three arguments and become:

MyApp.Outbox.publish("topic", "key", %{hello: :kafka})

Similarly, the Kafkaesque.publish/4 function would become Kafkaesque.publish/5 and accept key argument just before the payload.

And the structure of Kafkaesque.Message schema would be extended with the key field.

---

@v0idpwn What do you think?

[v0idpwn]

Hi, @mgibowski. I agree we should support key, I'm not 100% sure we should make it required, though, as its very common to use empty keys in Kafka.

:brod has the following interface:

• produce_sync(Pid, Value)
• produce_sync(Pid, Key, Value)

Perhaps we could follow along and have publish(topic, key, payload) and publish(topic, payload)? (i.e.: this would be publish(topic, key \\ "", payload)

[mgibowski]

Perhaps we could follow along and have publish(topic, key, payload) and publish(topic, payload)? (i.e.: this would be publish(topic, key \ "", payload)

Yes, that makes sense!

[v0idpwn]

I started #2 adding this feature. It will be a breaking change, but I'll try to make it smooth.
Do you intend to use Kafkaesque in production? I can try to cut 1.0 for that :)

[mgibowski]

I started #2 adding this feature. It will be a breaking change, but I'll try to make it smooth.

So far looks great!

Do you intend to use Kafkaesque in production? I can try to cut 1.0 for that :)

Yes, we plan to release code that uses Kafkaesque to production in middle November. 1.0 by that time would be awesome :)