New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

consider ELT with BigQuery as pseudo-schemaless db #1

Open

maxheld83 opened this issue Mar 30, 2021 · 3 comments

Contributor

maxheld83 commented Mar 30, 2021

These approaches might work for us.
Have to do more research, just to remember the links:

Contributor Author

maxheld83 commented Mar 30, 2021

this is also what (some) big nerds apparently do: mozilla/gcp-ingestion#423

Contributor Author

maxheld83 commented Mar 30, 2021

this is a good summary of the case for ELT https://www.stitchdata.com/resources/what-is-elt/

Contributor Author

maxheld83 commented Mar 30, 2021 •

edited

Loading

prep been implemented here:

https://github.com/The-Academic-Observatory/observatory-platform/blob/ecf20bae0d5bada3e44cd1c7c236958ff0a77364/observatory-dags/observatory/dags/telescopes/crossref_metadata.py#L124

and schema here: https://github.com/The-Academic-Observatory/observatory-platform/blob/6f4a67b6119492d4ebc55e89be1c4a73c0be833c/observatory-dags/observatory/dags/database/schema/crossref_metadata_2020-09-01.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment