Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support fanning out parent record into multiple child syncs #1862

Closed
edgarrmondragon opened this issue Jul 20, 2023 · 0 comments · Fixed by #2144
Closed

feat: Support fanning out parent record into multiple child syncs #1862

edgarrmondragon opened this issue Jul 20, 2023 · 0 comments · Fixed by #2144

Comments

@edgarrmondragon
Copy link
Collaborator

edgarrmondragon commented Jul 20, 2023

Feature scope

Taps (catalog, state, stream maps, tests, etc.)

Description

The current paradigm assumes developers want to sync a child stream once for every parent record. However, it's possible that a single parent record has information to sync multiple instances of the child stream.

I think it's possible to support this in a backwards-compatible way by adding a new Stream.generate_child_contexts method:

class Stream(metaclass=abc.ABCMeta):
    def generate_child_contexts(
        self,
        record: dict,
        context: dict | None,
    ) -> t.Iterable[dict | None]:
        context = self.get_child_context(record=record, context=context)
        if context:
            yield context

    def _process_record(
        self,
        record: dict,
        child_context: dict | None = None,
        partition_context: dict | None = None,
    ) -> None:
    # Update this method to use `generate_child_contexts`

There's a user in Slack that would need this to build their tap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant