feat: Added kafka service

[Alexnortung]

Here is a rough implementation of a kafka service for devenv.

I think there are definetely place for configuration options. but I am not sure how we can implement this properly. So feedback would be appreciated :)

closes #1563

[Alexnortung]

Update: I have added initial connectors to kafka connect, but they don't seem to persist when i restart kafka connect, not sure why.

[Alexnortung]

Okay, I figured it out, you could just pass the connectors to the start command

[sandydoo]

Take a look at https://github.com/NixOS/nixpkgs/blob/nixos-unstable/nixos/modules/services/misc/apache-kafka.nix. You should be able to reuse large parts of this, particularly the config/settings handling, and ideally keep a similar interface.

[Alexnortung]

Hey @sandydoo thanks for the feedback!
I have now used many of the options provided in the NixOS module. However I have tweaked some defaults, so that it just works without having the user configure anything but enable = true.

I probably also need to create settings for kafka connect.

I would also like to have some kind of readiness probe for kafka.

TODO:

• settings for kafka connect
• readiness probe for kafka
• make kafka connect start after kafka

[sandydoo]

However I have tweaked some defaults, so that it just works without having the user configure anything but enable = true.

That's the right approach.

@Alexnortung, is there a basic test we could add for this to make sure it launches? I'm not familiar with Kafka, but maybe a ping/healthcheck/queue-unqueue type of thing. See the examples and tests directories.

[Alexnortung]

Hi @sandydoo I have Added settings to kafka connect and added a test for kafka.
I also wanted to add readiness probe to kafka, but it seems like something weird is going on when you add readiness probe to the service. Do you have any suggestions?

[domenkozar]

Where does kafka-topics.sh come from?

[Alexnortung]

It's part of the apacheKafka package

[Alexnortung]

If you prefer we could probably also do a curl since the script is probably just calling an endpoint in kafka

[domenkozar]

yeah let's do just curl

[Alexnortung]

I'm sorry, after looking at the docs, it looks like it is only kafka connect which got a REST API, not Kafka.
Is there any issues in using their own client?

[Alexnortung]

Am I missing something since the github actions are failing?

[domenkozar]

Am I missing something since the github actions are failing?

some of them fail, but you'll see kafka being run in the test action

[Alexnortung]

I have now also added readiness probes for kafka and kafka connect. I have also added kafka as a dependency to kafka connect, so that kafka connect doesn't fail to start if kafka is not started correctly or is using more time than usual.