Skip to content

Commit 6a04840

Browse files
committed
clarify evaluation of conditions, failures and field.rename
1 parent 72c55f1 commit 6a04840

File tree

1 file changed

+12
-19
lines changed

1 file changed

+12
-19
lines changed

docs/design-documents/20231117-better-processors.md

Lines changed: 12 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -121,16 +121,10 @@ A few notes about the `Process` method:
121121
records, Conduit will acknowledge the original record only once all split records
122122
are acknowledged).
123123
- If an error happens in `Process` it should be stored in the `ProcessedRecord` that
124-
was being processed when the error occurred. In that case the processor has two
125-
options on how to continue processing the remaining records in the slice.
126-
1. If the processor has no side effects and cheap it can process the remaining
127-
records and return them normally in the slice. Conduit will figure out what to
128-
do with the remaining ones based on the dead-letter queue configuration.
129-
2. If the processor has side effects or requires a larger amount of time to process
130-
a record, we would like to avoid processing records after an error. The remaining
131-
records can be filled with a specific `ProcessedRecord` that marks them as not
132-
processed. In that case Conduit will determine if it's required to process those
133-
records and pass them to the processor in the next call to `Process`.
124+
was being processed when the error occurred. In that case the remaining records
125+
can be filled with a specific `ProcessedRecord` that marks them as not processed.
126+
In that case Conduit will determine if it's required to process those records and
127+
pass them to the processor in the next call to `Process`.
134128

135129
### Listing processor plugins
136130

@@ -274,15 +268,14 @@ conditions need to evaluate to a boolean value. We propose to simply use the pac
274268
`strconv`, specifically the function
275269
[`strconv.ParseBool`](https://pkg.go.dev/strconv#ParseBool) to convert the result to
276270
a boolean value. This means that values 1, t, T, TRUE, true, True are converted to
277-
`true`. In addition, any value that is not 0, f, F, FALSE, false, False will return
278-
an error, which we should also regard as `false`, but additionally log a warning.
271+
`true`, while 0, f, F, FALSE, false and False are converted to `false`. Any value
272+
that can not be converted to `true` or `false` will return an error which means the
273+
record gets passed to the DLQ handler.
279274

280-
Note that we still need to evaluate if `text/template` is actually the implementation
281-
we want to use to evaluate these expressions. If the performance is too slow we might
282-
need to manually implement a compatible evaluator, which should be able to take
283-
shortcuts and reach better performance, since we know that we are always evaluating
284-
the expressions on an OpenCDC record (no need for reflection if we are not diving into
285-
the key or payload).
275+
In our evaluations of template engines `text/template` seems to be performant enough
276+
for our use case. Evaluating a simple template is currently faster than the measured
277+
maximum throughput of Conduit itself, to be precise, it is 10x faster, so we don't
278+
expect it to be a bottleneck.
286279

287280
### Documenting processors
288281

@@ -490,7 +483,7 @@ processors:
490483
id: example
491484
type: field.rename
492485
settings:
493-
fields: ".Key.foo:bar,.Payload.Before.id:key,.Metadata.topic:subject"
486+
mapping: ".Key.foo:bar,.Payload.Before.id:key,.Metadata.topic:subject"
494487
```
495488

496489
### `field.flatten`

0 commit comments

Comments
 (0)