Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter programming #15

Open
jannikrdev opened this issue Dec 1, 2022 · 1 comment
Open

Filter programming #15

jannikrdev opened this issue Dec 1, 2022 · 1 comment
Assignees

Comments

@jannikrdev
Copy link
Collaborator

jannikrdev commented Dec 1, 2022

We'll use code to dynamically create tasks on the fly with the right parameters set in a yaml file, see dynamic DAG's and task groups

@jannikrdev jannikrdev added this to the Transformation Layer milestone Dec 1, 2022
@yk2684 yk2684 self-assigned this Dec 7, 2022
@jannikrdev
Copy link
Collaborator Author

Update: for simplicity reasons it makes most sense to cache the results of each step of the cleaning pipeline in a table of a temporary schema, "staging". There, the table from issue #16 will be cached and can be easily retrieved for further analysis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants