Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better way to send resource fields information #22

Open
hannelita opened this issue Jun 16, 2020 · 1 comment
Open

Better way to send resource fields information #22

hannelita opened this issue Jun 16, 2020 · 1 comment
Labels
enhancement New feature or request question Further information is requested

Comments

@hannelita
Copy link
Contributor

At this time they are sent via array in the DAG params (e.g. "schema_fields_array": "['field1', 'field2']") and everything is treated as text type.
What are good alternatives?
Questions:

  • Modify this array to become a dictionary of names and types. What is a good way to pass it? I believe doing it in plain text can be tedious. What about adding another node on the DAG to fetch the header of the CSV and automatically create a dictionary of fields, hard-coding everything to text?
  • Should we start considering different types?
  • If yes, define how to treat errors
@hannelita hannelita added enhancement New feature or request question Further information is requested labels Jun 16, 2020
@hannelita
Copy link
Contributor Author

Waiting for @rufuspollock analysis to ETL pipeline

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant