Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First pass at duckdb data interface #379

Draft
wants to merge 17 commits into
base: new-data-input
Choose a base branch
from
Draft

First pass at duckdb data interface #379

wants to merge 17 commits into from

Conversation

cc-a
Copy link
Collaborator

@cc-a cc-a commented Jun 27, 2024

Description

A first mocked up version of what a revised data input layer may look like using standalone (i.e. no sqlalchemy) duckdb. So far implemented:

  • reader functions for the region, commodity definition and commodity demand tables that populate the their data into duckdb whilst also returning a numpy array representation.
  • calculate_global_commodities function that takes the data from the commodity definition table and transforms it into the xarray representation required by Muse internally.
  • tests for each function implemented plus additional tests addressing any constraints applied to database columns.

Mostly pushing this for visibility at this point as @tsmbland is planning to expand this to cover the remaining database tables and xarray data structures.

Fixes # (issue)

Type of change

Please add a line in the relevant section of
CHANGELOG.md to
document the change (include PR #) - note reverse order of PR #s.

  • New feature (non-breaking change which adds functionality)
  • Optimization (non-breaking, back-end change that speeds up the code)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (whatever its nature)

Key checklist

  • All tests pass: $ python -m pytest
  • The documentation builds and looks OK: $ python -m sphinx -b html docs docs/build

Further checks

  • Code is commented, particularly in hard-to-understand areas
  • Tests added that prove fix is effective or that feature works

@cc-a cc-a requested review from alexdewar and tsmbland June 27, 2024 16:32
Copy link

codecov bot commented Jun 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.30%. Comparing base (f5bec10) to head (ae61de8).

Additional details and impacted files
@@               Coverage Diff               @@
##           new-data-input     #379   +/-   ##
===============================================
  Coverage           71.30%   71.30%           
===============================================
  Files                  44       44           
  Lines                5890     5890           
  Branches             1155     1155           
===============================================
  Hits                 4200     4200           
  Misses               1369     1369           
  Partials              321      321           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tsmbland tsmbland linked an issue Jun 28, 2024 that may be closed by this pull request
@cc-a cc-a mentioned this pull request Jun 28, 2024
8 tasks
@alexdewar alexdewar removed their request for review January 30, 2025 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

Create minimal example database with SQLAlchemy and/or DuckDB
2 participants