@@ -121,16 +121,10 @@ A few notes about the `Process` method:
121
121
records, Conduit will acknowledge the original record only once all split records
122
122
are acknowledged).
123
123
- If an error happens in ` Process ` it should be stored in the ` ProcessedRecord ` that
124
- was being processed when the error occurred. In that case the processor has two
125
- options on how to continue processing the remaining records in the slice.
126
- 1 . If the processor has no side effects and cheap it can process the remaining
127
- records and return them normally in the slice. Conduit will figure out what to
128
- do with the remaining ones based on the dead-letter queue configuration.
129
- 2 . If the processor has side effects or requires a larger amount of time to process
130
- a record, we would like to avoid processing records after an error. The remaining
131
- records can be filled with a specific ` ProcessedRecord ` that marks them as not
132
- processed. In that case Conduit will determine if it's required to process those
133
- records and pass them to the processor in the next call to ` Process ` .
124
+ was being processed when the error occurred. In that case the remaining records
125
+ can be filled with a specific ` ProcessedRecord ` that marks them as not processed.
126
+ In that case Conduit will determine if it's required to process those records and
127
+ pass them to the processor in the next call to ` Process ` .
134
128
135
129
### Listing processor plugins
136
130
@@ -274,15 +268,14 @@ conditions need to evaluate to a boolean value. We propose to simply use the pac
274
268
` strconv` , specifically the function
275
269
[`strconv.ParseBool`](https://pkg.go.dev/strconv#ParseBool) to convert the result to
276
270
a boolean value. This means that values 1, t, T, TRUE, true, True are converted to
277
- ` true` . In addition, any value that is not 0, f, F, FALSE, false, False will return
278
- an error, which we should also regard as `false`, but additionally log a warning.
271
+ ` true` , while 0, f, F, FALSE, false and False are converted to `false`. Any value
272
+ that can not be converted to `true` or `false` will return an error which means the
273
+ record gets passed to the DLQ handler.
279
274
280
- Note that we still need to evaluate if `text/template` is actually the implementation
281
- we want to use to evaluate these expressions. If the performance is too slow we might
282
- need to manually implement a compatible evaluator, which should be able to take
283
- shortcuts and reach better performance, since we know that we are always evaluating
284
- the expressions on an OpenCDC record (no need for reflection if we are not diving into
285
- the key or payload).
275
+ In our evaluations of template engines `text/template` seems to be performant enough
276
+ for our use case. Evaluating a simple template is currently faster than the measured
277
+ maximum throughput of Conduit itself, to be precise, it is 10x faster, so we don't
278
+ expect it to be a bottleneck.
286
279
287
280
# ## Documenting processors
288
281
@@ -490,7 +483,7 @@ processors:
490
483
id: example
491
484
type: field.rename
492
485
settings:
493
- fields : ".Key.foo:bar,.Payload.Before.id:key,.Metadata.topic:subject"
486
+ mapping : ".Key.foo:bar,.Payload.Before.id:key,.Metadata.topic:subject"
494
487
` ` `
495
488
496
489
# ## `field.flatten`
0 commit comments