Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run processor examples, gather specifications #1384

Merged
merged 8 commits into from
Feb 16, 2024

Conversation

lovromazgon
Copy link
Member

@lovromazgon lovromazgon commented Feb 15, 2024

Description

This PR adds the utility test function RunExample for builtin processors. The idea is that we can write runnable examples for builtin processors, which will be tested for their correctness and gathered together with the processors' specifications in a JSON file called processors.json. We can use that JSON file in our documentation to display documentation about our processors.

The function will compare the record before and after the processing and output the diff in the unified format.

Adding an example looks like this:

// Simulates a processor that will be tested.
func newTestProcessor() sdk.Processor {
	return sdk.NewProcessorFunc(sdk.Specification{
		Name:        "test",
		Version:     "v0.1.0",
		Description: "Test processor",
	}, func(ctx context.Context, rec opencdc.Record) (opencdc.Record, error) {
		rec.Operation = opencdc.OperationUpdate
		return rec, nil
	})
}

func ExampleTestProcessor_1() {
	p := newTestProcessor()

	RunExample(p, example{
		Description: `When the processor receives a record with operation "create", ` +
			`it will transform the operation into "update".`,
		Config: map[string]string{},
		Have:   opencdc.Record{Operation: opencdc.OperationCreate},
		Want:   sdk.SingleRecord{Operation: opencdc.OperationUpdate},
	})

	// Output:
	// processor transformed record:
	// --- before
	// +++ after
	// @@ -1,10 +1,10 @@
	//  {
	//    "position": null,
	// -  "operation": "create",
	// +  "operation": "update",
	//    "metadata": null,
	//    "key": null,
	//    "payload": {
	//      "before": null,
	//      "after": null
	//    }
	//  }
}

Closes #1289

Quick checks:

  • I have followed the Code Guidelines.
  • There is no other pull request for the same update/change.
  • I have written unit tests.
  • I have made sure that the PR is of reasonable size and can be easily reviewed.

@lovromazgon lovromazgon requested a review from a team as a code owner February 15, 2024 20:22
@@ -0,0 +1,16 @@
# Diff
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This package doesn't really need to be reviewed, see readme.

@lovromazgon lovromazgon merged commit 97b0d1a into feature/better-processors Feb 16, 2024
4 checks passed
@lovromazgon lovromazgon deleted the lovro/processor-gather-specs branch February 16, 2024 18:49
@lovromazgon lovromazgon mentioned this pull request Feb 16, 2024
4 tasks
lovromazgon added a commit that referenced this pull request Mar 7, 2024
* Refactor pkg/plugin and isolate connector plugin (#1302)

* refactor pkg/plugin and isolate connector plugin

* fix imports

* fix tests

* fix import

* change registry to connectorRegistry

---------

Co-authored-by: Haris Osmanagić <[email protected]>

* BP: WASM: Load available WASM processors (#1322)



Co-authored-by: Maha Hajja <[email protected]>

---------

Co-authored-by: Lovro Mažgon <[email protected]>
Co-authored-by: Maha Hajja <[email protected]>

* Refactor pkg/plugin and isolate connector plugin (#1302)

* refactor pkg/plugin and isolate connector plugin

* fix imports

* fix tests

* fix import

* change registry to connectorRegistry

---------

Co-authored-by: Haris Osmanagić <[email protected]>

* BP: WASM: Load available WASM processors (#1322)

Co-authored-by: Maha Hajja <[email protected]>

---------

Co-authored-by: Lovro Mažgon <[email protected]>
Co-authored-by: Maha Hajja <[email protected]>

* Refactor processor service, processor and node (#1354)

* linter

* Run processor examples, gather specifications (#1384)

* update processor-sdk, use config parameter from conduit-commons

* add unified diff lib

* add utilities for running processor examples and collecting specifications

* update diff readme

* fix markdown linter error

* fix liter errors

* update processor-sdk

* fix race condition in test

* BP: Display processor example raw data as string (#1397)

* display raw data in processor examples as string

* ensure examples are formatted the same way in processors.json

* BP: add builtin processor part#1 (#1371)

* add processors field.set & field.subset.exclude

* delete extra test

* fix tests

* add field.rename processor

* add exclusion validation to field.rename processor

* refactor + fix error return in Process method

* add field.convert processor

* add filter processor

* only structured data

* address reviews part1

* use paramgen, update Configure method, update tests

* evaluate value for processor setField

* use a slice of referenceResolvers for processors with multiple fields params

* do not allow .Position to be set in setField processor

* update setField test, add non existent field test

* use the new Rename() method from referenceResolver

* address reviews

* address reviews2

* fix renameField bug + add New() method to each processor

* linter fix

* add processor examples

* processor json

* Update pkg/plugin/processor/builtin/convertField.go

Co-authored-by: Lovro Mažgon <[email protected]>

* address reviews

* setFeild example

* setFeild example

* make generate

* fix paramgen regex

* generate

* fix regex

* add paramgen to tools

---------

Co-authored-by: Lovro Mažgon <[email protected]>

* BP: Add unwrap.debezium (#1393)

* BP: Add unwrap.kafkaconnect (#1395)

* BP: Add webhook.http (#1382)

* BP: Add unwrap.opencdc (#1387)

* BP: Add encode.avro (#1401)

* updated docs, regenerate

* config

* first pass

* downgrade lib

* processing

* example

* fix config parsing

* regenerate

* fix config

* regenerate, tests

* linter, move to opencdc

* add test certs

* config refactor, tests

* more tests

* linter

* Regenerate

* fix linter

* git diff

* git diff

* regenerate

* avro examples

* run schema registry in examples on port 54321

* fix linter error

* rename processor

* fix schema registry integration test

* don't run avro examples in integration tests

---------

Co-authored-by: Lovro Mažgon <[email protected]>

* BP: Add custom.javascript (#1374)

* add json.decode processor (#1405)

* add json.decode processor

* make generate

* processors

* address reviews

* make generate

* add support for decoding JSON types other than map

* make generate

* address review comments

---------

Co-authored-by: Lovro Mažgon <[email protected]>

* BP: List processor plugins in API (#1406)

* update proto API definitions

* fix proto linter warnings

* deprecate ListPlugins

* implement plugin list endpoint

* processor plugin orchestrator

* move plugin endpoints under connector and processor service

* BP: add avro.decode (#1410)

Co-authored-by: Lovro Mažgon <[email protected]>

* avro.decode example, remove base64 decoding

---------

Co-authored-by: Lovro Mažgon <[email protected]>

* BP: Base64 decode / encode processors (#1412)

* encode

* base64 decode

* organize imports

* add encodeJson processor (#1413)

* encode json processor

* delete forgotten test

* address reviews

* linter fix

* add processor examples

* don't encode nil, output string

---------

Co-authored-by: Lovro Mažgon <[email protected]>

* Builtin processors refactoring, housekeeping (#1402)

* refactor builtin processors, generate specs into separate files

* move processors into folder impl

* go mod tidy

* add example summaries

* add example summaries

* fix name of avro.decode processor

* add example summaries

* regenerate specs

* update default registry

* update default registry: fix tests, linter, regenerate

* dep upgrades

* refactorings

* move json encode processor to impl/json
* rename field.subset.exclude to field.exclude
* rename constructor for unwrap.debezium processor
* regenerate specs

* dep downgrades

---------

Co-authored-by: Haris Osmanagic <[email protected]>

* fix linter

* BP: call the conditional execution methods in the processors (#1411)

* init

* simplify

* fix evaluation

* add support for batches in runnable processor

* batch test in runnable processor

* optimize case when all records are passed through in a processor

* make sure remaining records are copied over

* fix len check

* make sure processor doesn't return more records

---------

Co-authored-by: Lovro Mažgon <[email protected]>

* update examples and specs

* disable goconst in processors (docs are repetitive)

* upgrade processor sdk

* add log when loading wasm processors, more docs updates

---------

Co-authored-by: Haris Osmanagić <[email protected]>
Co-authored-by: Maha Hajja <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants