Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(tap): Adding support for schema #66

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

rvantunes
Copy link

Context

We need to read JSONL files with complex structs, including many data types like array, object, datetime, string, etc...
The current implementation offers jsonl_type_coercion_strategy to decide how the tap is going to infer the schema. The problem is that none of the coercion strategies works for our use case. Besides that, the jsonl_sampling_strategy only supports first which does not work for our use case.

This PR implements a possible solution, by adding a new parameter named schema. This parameter would be parsed by Meltano SDK, see here.

If the user decides to specify a fixed schema, the tap does not need to create the schema dynamically, ignoring the coercion strategy.

Changes

  • Adding a new parameter to specify schema
  • Updating documentation
  • Fixing typos
  • Adding IDE files in .gitignore

@rvantunes rvantunes changed the title Adding support for schema feat(tap): Adding support for schema Sep 28, 2023
@leonardoeloy
Copy link

leonardoeloy commented Sep 28, 2023

@visch this approach seemed to be the best solution when we have full control of the schema

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants