-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run processor examples, gather specifications #1384
Merged
lovromazgon
merged 8 commits into
feature/better-processors
from
lovro/processor-gather-specs
Feb 16, 2024
Merged
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
30a55c0
update processor-sdk, use config parameter from conduit-commons
lovromazgon a0505d8
add unified diff lib
lovromazgon 91cb369
add utilities for running processor examples and collecting specifica…
lovromazgon 3ab113b
update diff readme
lovromazgon e907fb9
fix markdown linter error
lovromazgon f3798b6
fix liter errors
lovromazgon dbac2a3
update processor-sdk
lovromazgon da18119
fix race condition in test
lovromazgon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| // Copyright © 2024 Meroxa, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| //go:build export_processors | ||
|
|
||
| package builtin | ||
|
|
||
| import ( | ||
| "io" | ||
| "log" | ||
| "os" | ||
| "sort" | ||
| "strings" | ||
| "testing" | ||
|
|
||
| "github.com/goccy/go-json" | ||
| ) | ||
|
|
||
| func TestMain(m *testing.M) { | ||
| code := m.Run() | ||
| if code > 0 { | ||
| os.Exit(code) | ||
| } | ||
|
|
||
| // tests passed, export the processors | ||
| const outputFile = "processors.json" | ||
|
|
||
| f, err := os.OpenFile(outputFile, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0600) | ||
| if err != nil { | ||
| log.Fatalf("failed to open %s: %v", outputFile, err) | ||
| } | ||
| defer f.Close() | ||
|
|
||
| exportProcessors(f) | ||
| } | ||
|
|
||
| func exportProcessors(output io.Writer) { | ||
| sorted := sortProcessors(processors) | ||
|
|
||
| bytes, err := json.MarshalIndent(sorted, "", " ") | ||
| if err != nil { | ||
| log.Fatalf("failed to marshal processors to JSON: %v", err) | ||
| } | ||
|
|
||
| _, err = output.Write(bytes) | ||
| if err != nil { | ||
| log.Fatalf("failed to write processors to output: %v", err) | ||
| } | ||
| } | ||
|
|
||
| func sortProcessors(processors map[string]*procInfo) []*procInfo { | ||
| names := make([]string, 0, len(processors)) | ||
| for k, _ := range processors { | ||
| names = append(names, k) | ||
| } | ||
| sort.Strings(names) | ||
|
|
||
| sorted := make([]*procInfo, len(names)) | ||
| for i, name := range names { | ||
| // also sort examples for each processor | ||
| proc := processors[name] | ||
| proc.Examples = sortExamples(proc.Examples) | ||
| sorted[i] = proc | ||
| } | ||
|
|
||
| return sorted | ||
| } | ||
|
|
||
| func sortExamples(examples []example) []example { | ||
| sort.Slice(examples, func(i, j int) bool { | ||
| if examples[i].Order != examples[j].Order { | ||
| return examples[i].Order < examples[j].Order | ||
| } | ||
| return strings.Compare(examples[i].Description, examples[j].Description) < 0 | ||
| }) | ||
| return examples | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| // Copyright © 2024 Meroxa, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| //go:generate go test -count=1 -tags export_processors . | ||
|
|
||
| package builtin | ||
|
|
||
| import ( | ||
| "context" | ||
| "encoding/json" | ||
| "fmt" | ||
| "log" | ||
|
|
||
| "github.com/conduitio/conduit-commons/opencdc" | ||
| sdk "github.com/conduitio/conduit-processor-sdk" | ||
| "github.com/conduitio/conduit/pkg/plugin/processor/builtin/internal/diff" | ||
| "github.com/google/go-cmp/cmp" | ||
| "github.com/google/go-cmp/cmp/cmpopts" | ||
| ) | ||
|
|
||
| // -- HELPERS ------------------------------------------------------------------ | ||
|
|
||
| var processors = map[string]*procInfo{} | ||
|
|
||
| type procInfo struct { | ||
| Specification sdk.Specification `json:"specification"` | ||
| Examples []example `json:"examples"` | ||
| } | ||
|
|
||
| type example struct { | ||
| // Order is an optional field that is used to order examples in the | ||
| // documentation. If omitted, the example will be ordered by description. | ||
| Order int `json:"-"` | ||
| Description string `json:"description"` | ||
| Config map[string]string `json:"config"` | ||
| Have opencdc.Record `json:"have"` | ||
| Want sdk.ProcessedRecord `json:"want"` | ||
| } | ||
|
|
||
| // RunExample runs the given example with the given processor and logs the | ||
| // result. It is intended to be used in example functions. Additionally, it | ||
| // stores the processor specification and example in a global map so it can be | ||
| // used to generate documentation. | ||
| func RunExample(p sdk.Processor, e example) { | ||
| spec, err := p.Specification() | ||
| if err != nil { | ||
| log.Fatalf("failed to fetch specification: %v", err) | ||
| } | ||
|
|
||
| pi, ok := processors[spec.Name] | ||
| if !ok { | ||
| pi = &procInfo{Specification: spec} | ||
| processors[spec.Name] = pi | ||
| } | ||
|
|
||
| ctx := context.Background() | ||
| err = p.Configure(ctx, e.Config) | ||
| if err != nil { | ||
| log.Fatalf("failed to configure processor: %v", err) | ||
| } | ||
|
|
||
| err = p.Open(ctx) | ||
| if err != nil { | ||
| log.Fatalf("failed to open processor: %v", err) | ||
| } | ||
|
|
||
| got := p.Process(ctx, []opencdc.Record{e.Have.Clone()}) | ||
| if len(got) != 1 { | ||
| log.Fatalf("expected 1 record to be returned, got %d", len(got)) | ||
| } | ||
|
|
||
| if d := cmp.Diff(e.Want, got[0], cmpopts.IgnoreUnexported(sdk.SingleRecord{})); d != "" { | ||
| log.Fatalf("processed record did not match expectation:\n%v", d) | ||
| } | ||
|
|
||
| switch rec := got[0].(type) { | ||
| case sdk.SingleRecord: | ||
| // produce JSON diff | ||
| havePrettyJSON, err := json.MarshalIndent(e.Have, "", " ") | ||
| if err != nil { | ||
| log.Fatalf("failed to marshal test record to JSON: %v", err) | ||
| } | ||
|
|
||
| gotPrettyJSON, err := json.MarshalIndent(rec, "", " ") | ||
| if err != nil { | ||
| log.Fatalf("failed to marshal processed record to JSON: %v", err) | ||
| } | ||
|
|
||
| edits := diff.Strings(string(havePrettyJSON), string(gotPrettyJSON)) | ||
| unified, err := diff.ToUnified("before", "after", string(havePrettyJSON)+"\n", edits, 100) | ||
| if err != nil { | ||
| log.Fatalf("failed to produce unified diff: %v", err) | ||
| } | ||
|
|
||
| fmt.Printf("processor transformed record:\n%s\n", unified) | ||
| case sdk.FilterRecord: | ||
| fmt.Println("processor filtered record out") | ||
| case sdk.ErrorRecord: | ||
| fmt.Printf("processor returned error: %s\n", rec.Error) | ||
| } | ||
|
|
||
| // append example to processor | ||
| pi.Examples = append(pi.Examples, e) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # Diff | ||
|
|
||
| This package contains code taken from <https://github.com/golang/tools/tree/master/internal/diff> | ||
| on February 15th, 2024. We need the code to create a unified diff between two strings. | ||
|
|
||
| The code is left as-is, except 3 changes: | ||
|
|
||
| - The imports were changed to reference the Conduit module path. This was done | ||
| using the following command: | ||
|
|
||
| ```sh | ||
| find . -type f -exec sed -i '' 's/golang.org\/x\/tools\/internal/github.com\/conduitio\/conduit\/pkg\/plugin\/processor\/builtin\/internal/g' {} + | ||
| ``` | ||
|
|
||
| - The package `golang.org/x/tools/internal/diff/myers` was removed, as it's deprecated. | ||
|
|
||
| - The package `golang.org/x/tools/internal/testenv` was added into the `diff` package, | ||
| as that's the only place it's used. It also only includes the required functions. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This package doesn't really need to be reviewed, see readme.