feat: Add command to generate a dbt exposures file #851
Conversation
bmtcril
force-pushed
the
bmtcril/dbt_lineage
branch
from
d61ed0d to
553c1b5
Compare
|
Exposures file looks like this: 
|
|
Exposure in the docs looks like this:
(If we fill in the dashboard descriptions in Superset they will propagate to the description there) And lineage works: |
| @@ -76,6 +76,8 @@ jobs: | |||
| tutor local do dump-data-to-clickhouse --options "--object course_overviews" | |||
| make extract_translations | |||
| tutor local do import-assets | |||
| tutor local do collect-dbt-lineage | |||
| tutor local run lms python manage.py lms transform_tracking_logs --source_provider LOCAL --source_config '{"key": "/openedx/data/", "prefix": "tracking.log", "container": "logs"}' --destination_provider LRS --transformer_type xapi | |||
There was a problem hiding this comment.
This is a somewhat random addition to this PR, but will test that we don't break the backfill command again
tutoraspects/templates/aspects/apps/superset/pythonpath/collect_dbt_lineage.py
Show resolved
Hide resolved
There was a problem hiding this comment.
👍 Works great for me, thank you @bmtcril !
- I tested this on my tutor dev aspects stack, and (after rebuilding the aspects-superset image) the command ran without issue.
- I read through the code
-
I checked for accessibility issuesN/A - Includes documentation
-
User-facing strings are extracted for translationN/A
tutoraspects/templates/aspects/apps/superset/pythonpath/collect_dbt_lineage.py
Show resolved
Hide resolved
|
@bmtcril should we move this to |
|
@Ian2012 The problem with putting this in aspects-dbt CI is that the exposures will be out of date and cause errors building the docs if tutor-contrib-aspects hasn't been updated to use them yet. If we do it in this repo we can get the correct versions of aspects-dbt & Superset dashboards. |
We'll use this in aspects-dbt CI to keep the model to dashboard mappings up to date.
bmtcril
force-pushed
the
bmtcril/dbt_lineage
branch
from
553c1b5 to
1a98712
Compare
| @@ -8,6 +8,8 @@ image: {{ DOCKER_IMAGE_SUPERSET }} | |||
| - ../../env/plugins/aspects/apps/superset/scripts:/app/scripts | |||
| - ../../env/plugins/aspects/build/aspects-superset/localization:/app/localization | |||
| - ../../env/plugins/aspects/build/aspects-superset/openedx-assets:/app/openedx-assets | |||
| # Shared with Aspects job so it has access to the state file for the lineage push job | |||
| - ../../data/aspects-dbt:/app/dbt_manifest | |||
There was a problem hiding this comment.
@Ian2012 I'm pondering how to make this work with k8s based on our earlier discussion
|
We're going to punt on k8s support for this temporarily and merge it as-is. Eventually we'd like to make the dbt docs using whatever the installed (potentially custom) version of dbt is and exposures from all (potentially custom) dashboards available in the Tutor environment. |
We'll use this in aspects-dbt CI to keep the model to dashboard mappings up to date. Leaving this in draft for the moment so we can discuss the direction. The idea is to have this command to generate an exposures file that we can bring back to aspects-dbt.
This should allow us visibility into which datasets are actually being used (or not used) in Superset and let us remove unused models. However there's a dependency problem where tutor-contrib-aspects needs to update to use a new aspects-dbt and there may be error-causing issues generating the exposures until it is updated. I haven't given much thought to how to resolve that yet, but we could move the dbt docs to that repo instead and use the combined official tutor-contrib-aspects / aspects-dbt version to make the file / docs.