Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use custom streams implementation to simplify error handling, writing to topics and serde configuration #265

Open
wants to merge 75 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
ab53ded
Simplify writing to output topics
philipp94831 Jan 6, 2025
59591d5
Simplify writing to output topics
philipp94831 Jan 6, 2025
322eb93
Add error handling
philipp94831 Jan 27, 2025
8f1b481
Merge remote-tracking branch 'origin/master' into feature/improved-ks…
philipp94831 Jan 27, 2025
d321b8c
Add error handling
philipp94831 Jan 27, 2025
5377422
Add error handling
philipp94831 Jan 27, 2025
a5f272e
Add error handling
philipp94831 Jan 27, 2025
0895122
Add error handling
philipp94831 Jan 27, 2025
103a20e
Add error handling
philipp94831 Jan 27, 2025
5780aef
Add error handling
philipp94831 Jan 27, 2025
bac7fe4
Merge remote-tracking branch 'origin/master' into feature/improved-ks…
philipp94831 Feb 13, 2025
4fe684f
Add JavaDoc
philipp94831 Feb 13, 2025
c0cdd70
Add docs
philipp94831 Feb 14, 2025
bb0ca14
Add docs
philipp94831 Feb 14, 2025
b89c2ac
Add docs
philipp94831 Feb 14, 2025
eb012c9
Add tests
philipp94831 Feb 14, 2025
2452aec
Add tests
philipp94831 Feb 14, 2025
dcdbe90
Add tests
philipp94831 Feb 14, 2025
bec9805
Add tests
philipp94831 Feb 14, 2025
21b013c
Add ConfiguredStreamJoined
philipp94831 Feb 14, 2025
0f401eb
Add tests
philipp94831 Feb 14, 2025
e22d8b2
Add tests
philipp94831 Feb 14, 2025
e529a2a
Add ConfiguredJoined
philipp94831 Feb 14, 2025
000360d
Add docs
philipp94831 Feb 14, 2025
efc1a69
Add docs
philipp94831 Feb 14, 2025
a3e2417
Add tests
philipp94831 Feb 14, 2025
0a55285
Add tests
philipp94831 Feb 14, 2025
8d7c664
Add docs
philipp94831 Feb 14, 2025
e2629d9
Rename
philipp94831 Feb 14, 2025
b6903cc
Rename
philipp94831 Feb 14, 2025
0088a57
Rename
philipp94831 Feb 14, 2025
f1d9198
Add docs
philipp94831 Feb 14, 2025
210c6d7
Rename
philipp94831 Feb 14, 2025
d53721a
Add docs
philipp94831 Feb 14, 2025
e687c3e
Add tests
philipp94831 Feb 14, 2025
10adc0c
Add tests
philipp94831 Feb 14, 2025
baefa30
Add tests
philipp94831 Feb 14, 2025
86935e4
Add methods to TopologyBuilder
philipp94831 Feb 14, 2025
2e0f96d
Extend Branched
philipp94831 Feb 14, 2025
b5b9f6d
Extend Branched
philipp94831 Feb 14, 2025
207c6b5
Extend Branched
philipp94831 Feb 14, 2025
104354d
Extend Branched
philipp94831 Feb 14, 2025
e696490
Rename and test
philipp94831 Feb 17, 2025
6bbbcc2
Update tests
philipp94831 Feb 17, 2025
3201fce
Add tests
philipp94831 Feb 17, 2025
8aaeac4
Add tests
philipp94831 Feb 17, 2025
67e2390
Add tests
philipp94831 Feb 17, 2025
23c382c
Add tests
philipp94831 Feb 17, 2025
0e4e9df
Add tests
philipp94831 Feb 17, 2025
d71cf6a
Add tests
philipp94831 Feb 17, 2025
6355003
Add tests
philipp94831 Feb 17, 2025
144f474
Add tests
philipp94831 Feb 17, 2025
34a9634
Add tests
philipp94831 Feb 17, 2025
e0bd7d9
Add tests
philipp94831 Feb 17, 2025
9ddefc8
Add tests
philipp94831 Feb 17, 2025
24b2b36
Add tests
philipp94831 Feb 17, 2025
8a7ab88
Add tests
philipp94831 Feb 17, 2025
62adf8b
Add tests
philipp94831 Feb 17, 2025
d66d250
Add tests
philipp94831 Feb 17, 2025
1251b54
Add tests
philipp94831 Feb 19, 2025
a1849d2
Add tests
philipp94831 Feb 19, 2025
9fc61bc
Add tests
philipp94831 Feb 19, 2025
5761ed6
Add tests
philipp94831 Feb 19, 2025
0b6dae5
Add tests
philipp94831 Feb 19, 2025
3d9ba36
Add tests
philipp94831 Feb 19, 2025
7a368a7
Add tests
philipp94831 Feb 19, 2025
4b45b59
Add tests
philipp94831 Feb 19, 2025
37f3344
Add tests
philipp94831 Feb 19, 2025
e8e71ae
Clean up
philipp94831 Feb 19, 2025
682fc4b
Update
philipp94831 Feb 19, 2025
848bec0
Remove refresh dependencies
philipp94831 Feb 19, 2025
dc85aa3
Clean up
philipp94831 Feb 19, 2025
817f2d6
Update dependencies
philipp94831 Feb 20, 2025
2ee879a
Merge remote-tracking branch 'origin/master' into feature/improved-ks…
philipp94831 Feb 20, 2025
63ce55a
Add topics to TestTopology
philipp94831 Feb 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add error handling
philipp94831 committed Jan 27, 2025

Unverified

This user has not yet uploaded their public signing key.
commit a5f272e7cf0e92c3269f1a8b0acd2fb94b3a9329
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
/*
* MIT License
*
* Copyright (c) 2025 bakdata
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/

package com.bakdata.kafka;

import java.time.Duration;
import java.util.HashMap;
import java.util.Map;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import lombok.With;
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.processor.StateStore;
import org.apache.kafka.streams.state.DslStoreSuppliers;

@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class ConfiguredMaterialized<K, V, S extends StateStore> {

@With
private final Preconfigured<Serde<K>> keySerde;
@With
private final Preconfigured<Serde<V>> valueSerde;
private final String storeName;
@With
private final Duration retention;
@With
private final DslStoreSuppliers storeType;
private final Map<String, String> topicConfig;
private final boolean loggingEnabled;
private final boolean cachingEnabled;

public static <K, V, S extends StateStore> ConfiguredMaterialized<K, V, S> keySerde(
final Preconfigured<Serde<K>> keySerde) {
return with(keySerde, null);
}

public static <K, V, S extends StateStore> ConfiguredMaterialized<K, V, S> valueSerde(
final Preconfigured<Serde<V>> valueSerde) {
return with(null, valueSerde);
}

public static <K, V, S extends StateStore> ConfiguredMaterialized<K, V, S> with(
final Preconfigured<Serde<K>> keySerde,
final Preconfigured<Serde<V>> valueSerde) {
return new ConfiguredMaterialized<>(keySerde, valueSerde, null, null, null, new HashMap<>(), true, true);
}

public static <K, V, S extends StateStore> ConfiguredMaterialized<K, V, S> as(final String storeName) {
return new ConfiguredMaterialized<>(null, null, storeName, null, null, new HashMap<>(), true, true);
}

public static <K, V, S extends StateStore> ConfiguredMaterialized<K, V, S> as(
final DslStoreSuppliers storeSuppliers) {
return new ConfiguredMaterialized<>(null, null, null, null, storeSuppliers, new HashMap<>(), true, true);
}

Materialized<K, V, S> configure(final Configurator configurator) {
final Materialized<K, V, S> materialized = Materialized.<K, V, S>as(this.storeName)
.withKeySerde(this.configureKeySerde(configurator))
.withValueSerde(this.configuredValueSerde(configurator));
if (this.retention != null) {
materialized.withRetention(this.retention);
}
if (this.storeType != null) {
materialized.withStoreType(this.storeType);
}
if (this.loggingEnabled) {
materialized.withLoggingEnabled(this.topicConfig);
} else {
materialized.withLoggingDisabled();
}
if (this.cachingEnabled) {
materialized.withCachingEnabled();
} else {
materialized.withCachingDisabled();
}
return materialized;
}

private Serde<V> configuredValueSerde(final Configurator configurator) {
return this.valueSerde == null ? null : configurator.configureForValues(this.valueSerde);
}

private Serde<K> configureKeySerde(final Configurator configurator) {
return this.keySerde == null ? null : configurator.configureForKeys(this.keySerde);
}
}
Original file line number Diff line number Diff line change
@@ -53,10 +53,16 @@ <VIn> ImprovedCogroupedKStream<K, VOut> cogroup(KGroupedStream<K, VIn> groupedSt
ImprovedKTable<K, VOut> aggregate(Initializer<VOut> initializer,
Materialized<K, VOut, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, VOut> aggregate(Initializer<VOut> initializer,
ConfiguredMaterialized<K, VOut, KeyValueStore<Bytes, byte[]>> materialized);

@Override
ImprovedKTable<K, VOut> aggregate(Initializer<VOut> initializer, Named named,
Materialized<K, VOut, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, VOut> aggregate(Initializer<VOut> initializer, Named named,
ConfiguredMaterialized<K, VOut, KeyValueStore<Bytes, byte[]>> materialized);

@Override
<W extends Window> ImprovedTimeWindowedCogroupedKStream<K, VOut> windowedBy(Windows<W> windows);

Original file line number Diff line number Diff line change
@@ -68,12 +68,24 @@ public ImprovedKTable<K, V> aggregate(final Initializer<V> initializer,
return this.context.wrap(this.wrapped.aggregate(initializer, materialized));
}

@Override
public ImprovedKTable<K, V> aggregate(final Initializer<V> initializer,
final ConfiguredMaterialized<K, V, KeyValueStore<Bytes, byte[]>> materialized) {
return this.aggregate(initializer, materialized.configure(this.context.getConfigurator()));
}

@Override
public ImprovedKTable<K, V> aggregate(final Initializer<V> initializer, final Named named,
final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized) {
return this.context.wrap(this.wrapped.aggregate(initializer, named, materialized));
}

@Override
public ImprovedKTable<K, V> aggregate(final Initializer<V> initializer, final Named named,
final ConfiguredMaterialized<K, V, KeyValueStore<Bytes, byte[]>> materialized) {
return this.aggregate(initializer, named, materialized.configure(this.context.getConfigurator()));
}

@Override
public <W extends Window> ImprovedTimeWindowedCogroupedKStream<K, V> windowedBy(final Windows<W> windows) {
return this.context.wrap(this.wrapped.windowedBy(windows));
Original file line number Diff line number Diff line change
@@ -48,30 +48,47 @@ public interface ImprovedKGroupedStream<K, V> extends KGroupedStream<K, V> {
@Override
ImprovedKTable<K, Long> count(Materialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, Long> count(ConfiguredMaterialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized);

@Override
ImprovedKTable<K, Long> count(Named named, Materialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, Long> count(Named named,
ConfiguredMaterialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized);

@Override
ImprovedKTable<K, V> reduce(Reducer<V> reducer);

@Override
ImprovedKTable<K, V> reduce(Reducer<V> reducer, Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, V> reduce(Reducer<V> reducer,
ConfiguredMaterialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);

@Override
ImprovedKTable<K, V> reduce(Reducer<V> reducer, Named named,
Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, V> reduce(Reducer<V> reducer, Named named,
ConfiguredMaterialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);

@Override
<VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? super K, ? super V, VR> aggregator);

@Override
<VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? super K, ? super V, VR> aggregator,
Materialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized);

<VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? super K, ? super V, VR> aggregator,
ConfiguredMaterialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized);

@Override
<VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? super K, ? super V, VR> aggregator,
Named named, Materialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized);

<VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? super K, ? super V, VR> aggregator,
Named named, ConfiguredMaterialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized);

@Override
<W extends Window> ImprovedTimeWindowedKStream<K, V> windowedBy(Windows<W> windows);

Original file line number Diff line number Diff line change
@@ -63,12 +63,23 @@ public ImprovedKTable<K, Long> count(final Materialized<K, Long, KeyValueStore<B
return this.context.wrap(this.wrapped.count(materialized));
}

@Override
public ImprovedKTable<K, Long> count(final ConfiguredMaterialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized) {
return this.count(materialized.configure(this.context.getConfigurator()));
}

@Override
public ImprovedKTable<K, Long> count(final Named named,
final Materialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized) {
return this.context.wrap(this.wrapped.count(named, materialized));
}

@Override
public ImprovedKTable<K, Long> count(final Named named,
final ConfiguredMaterialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized) {
return this.count(named, materialized.configure(this.context.getConfigurator()));
}

@Override
public ImprovedKTable<K, V> reduce(final Reducer<V> reducer) {
return this.context.wrap(this.wrapped.reduce(reducer));
@@ -80,12 +91,24 @@ public ImprovedKTable<K, V> reduce(final Reducer<V> reducer,
return this.context.wrap(this.wrapped.reduce(reducer, materialized));
}

@Override
public ImprovedKTable<K, V> reduce(final Reducer<V> reducer,
final ConfiguredMaterialized<K, V, KeyValueStore<Bytes, byte[]>> materialized) {
return this.reduce(reducer, materialized.configure(this.context.getConfigurator()));
}

@Override
public ImprovedKTable<K, V> reduce(final Reducer<V> reducer, final Named named,
final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized) {
return this.context.wrap(this.wrapped.reduce(reducer, named, materialized));
}

@Override
public ImprovedKTable<K, V> reduce(final Reducer<V> reducer, final Named named,
final ConfiguredMaterialized<K, V, KeyValueStore<Bytes, byte[]>> materialized) {
return this.reduce(reducer, named, materialized.configure(this.context.getConfigurator()));
}

@Override
public <VR> ImprovedKTable<K, VR> aggregate(final Initializer<VR> initializer,
final Aggregator<? super K, ? super V, VR> aggregator) {
@@ -99,13 +122,27 @@ public <VR> ImprovedKTable<K, VR> aggregate(final Initializer<VR> initializer,
return this.context.wrap(this.wrapped.aggregate(initializer, aggregator, materialized));
}

@Override
public <VR> ImprovedKTable<K, VR> aggregate(final Initializer<VR> initializer,
final Aggregator<? super K, ? super V, VR> aggregator,
final ConfiguredMaterialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized) {
return this.aggregate(initializer, aggregator, materialized.configure(this.context.getConfigurator()));
}

@Override
public <VR> ImprovedKTable<K, VR> aggregate(final Initializer<VR> initializer,
final Aggregator<? super K, ? super V, VR> aggregator, final Named named,
final Materialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized) {
return this.context.wrap(this.wrapped.aggregate(initializer, aggregator, named, materialized));
}

@Override
public <VR> ImprovedKTable<K, VR> aggregate(final Initializer<VR> initializer,
final Aggregator<? super K, ? super V, VR> aggregator, final Named named,
final ConfiguredMaterialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized) {
return this.aggregate(initializer, aggregator, named, materialized.configure(this.context.getConfigurator()));
}

@Override
public <W extends Window> ImprovedTimeWindowedKStream<K, V> windowedBy(final Windows<W> windows) {
return this.context.wrap(this.wrapped.windowedBy(windows));
Original file line number Diff line number Diff line change
@@ -38,9 +38,14 @@ public interface ImprovedKGroupedTable<K, V> extends KGroupedTable<K, V> {
@Override
ImprovedKTable<K, Long> count(Materialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, Long> count(ConfiguredMaterialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized);

@Override
ImprovedKTable<K, Long> count(Named named, Materialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, Long> count(Named named,
ConfiguredMaterialized<K, Long, KeyValueStore<Bytes, byte[]>> materialized);

@Override
ImprovedKTable<K, Long> count();

@@ -51,10 +56,16 @@ public interface ImprovedKGroupedTable<K, V> extends KGroupedTable<K, V> {
ImprovedKTable<K, V> reduce(Reducer<V> adder, Reducer<V> subtractor,
Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, V> reduce(Reducer<V> adder, Reducer<V> subtractor,
ConfiguredMaterialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);

@Override
ImprovedKTable<K, V> reduce(Reducer<V> adder, Reducer<V> subtractor, Named named,
Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);

ImprovedKTable<K, V> reduce(Reducer<V> adder, Reducer<V> subtractor, Named named,
ConfiguredMaterialized<K, V, KeyValueStore<Bytes, byte[]>> materialized);

@Override
ImprovedKTable<K, V> reduce(Reducer<V> adder, Reducer<V> subtractor);

@@ -63,11 +74,19 @@ <VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? s
Aggregator<? super K, ? super V, VR> subtractor,
Materialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized);

<VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? super K, ? super V, VR> adder,
Aggregator<? super K, ? super V, VR> subtractor,
ConfiguredMaterialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized);

@Override
<VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? super K, ? super V, VR> adder,
Aggregator<? super K, ? super V, VR> subtractor, Named named,
Materialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized);

<VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? super K, ? super V, VR> adder,
Aggregator<? super K, ? super V, VR> subtractor, Named named,
ConfiguredMaterialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized);

@Override
<VR> ImprovedKTable<K, VR> aggregate(Initializer<VR> initializer, Aggregator<? super K, ? super V, VR> adder,
Aggregator<? super K, ? super V, VR> subtractor);
Loading