Can we offer a migration story for Kafka topics? #1195
Labels
enhancement
New feature or request
p3
Nice-to-have features
size/M
Medium effort
spike/research
Requires more research before implementation
Comments
nscuro
added
enhancement
nscuro
added a commit
to DependencyTrack/helm-charts
that referenced
this issue
Jun 12, 2024
This is still a temporary solution for until the services are able to set up topics on their own (DependencyTrack/hyades#1195). For the moment we just want to reduce duplication. Signed-off-by: nscuro <[email protected]>
nscuro
added a commit
to DependencyTrack/helm-charts
that referenced
this issue
Jun 12, 2024
This is still a temporary solution for until the services are able to set up topics on their own (DependencyTrack/hyades#1195). For the moment we just want to reduce duplication. Signed-off-by: nscuro <[email protected]>
nscuro
added a commit
to DependencyTrack/hyades-apiserver
that referenced
this issue
Oct 3, 2024
Adds a new init task that creates all required topics using their respective default configuration during startup. This replaces the need for a separate init container (i.e. `redpanda-init` in our `docker-compose.yml`: https://github.com/DependencyTrack/hyades/blob/008c3a1b024969a159de085d5ca6b464147faa9a/docker-compose.yml#L251-L273). The creation of topics is also faster because it can be done in a single batched request, vs. 10s of repetitive CLI invocations. This init task is disabled by default for now, since it's not meant to be used for production deployments. The idea is to build on this work, and allow users to customize the default topic configurations (e.g. via env vars or config files). Relates to DependencyTrack/hyades#1195 Signed-off-by: nscuro <[email protected]>
2 tasks
nscuro
added a commit
to DependencyTrack/hyades-apiserver
that referenced
this issue
Oct 17, 2024
Adds a new init task that creates all required topics using their respective default configuration during startup. This replaces the need for a separate init container (i.e. `redpanda-init` in our `docker-compose.yml`: https://github.com/DependencyTrack/hyades/blob/008c3a1b024969a159de085d5ca6b464147faa9a/docker-compose.yml#L251-L273). The creation of topics is also faster because it can be done in a single batched request, vs. 10s of repetitive CLI invocations. This init task is disabled by default for now, since it's not meant to be used for production deployments. The idea is to build on this work, and allow users to customize the default topic configurations (e.g. via env vars or config files). Relates to DependencyTrack/hyades#1195 Signed-off-by: nscuro <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Current Behavior
For database migrations, we are in a very good spot due to the usage of Liquibase.
However, it's a different story for Kafka topics. We currently create them using a bash script, which we mount into the
redpanda-initcontainer.This of course only works for self-hosted setups, and currently only works for Redpanda, but not Apache Kafka.
Proposed Behavior
I am wondering if we can offer a functionality that is similar to how we do database migrations.
Is it viable to implement a migration functionality that:
And if so, can / should this be done in only a single component (i.e. API server), or should each component take care of its own topics?
Checklist
The text was updated successfully, but these errors were encountered: