diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 8d39ef6..0f51ed2 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -21,11 +21,8 @@ jobs: strategy: matrix: # commented out to save from github limits - # python-version: [ "3.8", "3.9", "3.10", "3.11" ] - # dbt-version: [ "1.6", "1.7", "1.8" ] - python-version: [ "3.9" ] + python-version: [ "3.8", "3.9", "3.10", "3.11" ] dbt-version: [ "1.7", "1.8" ] - steps: - uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} diff --git a/README.md b/README.md index 87991b4..531b1bb 100644 --- a/README.md +++ b/README.md @@ -16,5 +16,5 @@ custom Python methods tailored to your advanced needs. - Execute Python Model Locally - Enable Model-Level Orchestration Using Airflow -please see [EXAMPLES.md](docs/EXAMPLES.md). +please see [examples](docs/EXAMPLES.md). diff --git a/docs/EXAMPLES.md b/docs/EXAMPLES.md index 9807d45..4fa593b 100644 --- a/docs/EXAMPLES.md +++ b/docs/EXAMPLES.md @@ -1,23 +1,20 @@ # Example Use Cases -## Use customised adapter, provide jinja with custom python methods +## Use customised adapter, provide jinja with further python methods When you want to add more methods to existing adapter and make this methods available to jinja. you can use `dbt_custom_adapter` variable and provide your adapter class. -Example: -Extend existing adapter: -https://github.com/ismailsimsek/opendbt/blob/dd11150c1181b3488d123c7a0eeab7a3712ccf59/opendbt/examples.py#L10-L26 - -set `dbt_custom_adapter` variable in your `dbt_project.yml` file +**Step-1:** Extend existing adapter +https://github.com/memiiso/opendbt/blob/ddf0b1bac379aa42961900f35dfc9938b1bc19c4/opendbt/examples.py#L10-L26 +**Step-2:** Edit `dbt_project.yml` file, set `dbt_custom_adapter` variable to the class name of the new adapter. ```yml vars: dbt_custom_adapter: opendbt.examples.DuckDBAdapterV2Custom ``` -Run it: - +**Step-3:** Run dbt, nwo dbt uses the provided adapter class. ```python from opendbt import OpenDbtProject @@ -30,22 +27,43 @@ dp.run(command="run") Using customized adapter and a custom materialization we can extend dbt to run local python code. this is useful for the scenarios where data is imported from external API. -Extend existing adapter: add new adapter method which runs given python code. +**Step-1:** Extend existing adapter, Add new adapter method which runs given python code. -https://github.com/ismailsimsek/opendbt/blob/dd11150c1181b3488d123c7a0eeab7a3712ccf59/opendbt/examples.py#L10-L26 +https://github.com/memiiso/opendbt/blob/ddf0b1bac379aa42961900f35dfc9938b1bc19c4/opendbt/examples.py#L10-L26 -Create materialization to use this method +**Step-2:** Create materialization, where from the jonja we call this new adapter method -https://github.com/ismailsimsek/opendbt/blob/dd11150c1181b3488d123c7a0eeab7a3712ccf59/opendbt/macros/executepython.sql#L1-L26 +https://github.com/memiiso/opendbt/blob/ddf0b1bac379aa42961900f35dfc9938b1bc19c4/opendbt/macros/executepython.sql#L1-L26 -create model which runs local python and materialize the data using new materialization +**Step-3:** Create model using the materialization -https://github.com/ismailsimsek/opendbt/blob/dd11150c1181b3488d123c7a0eeab7a3712ccf59/tests/resources/dbttest/models/my_executepython_dbt_model.py#L1-L999 +https://github.com/memiiso/opendbt/blob/ddf0b1bac379aa42961900f35dfc9938b1bc19c4/tests/resources/dbttest/models/my_executepython_dbt_model.py#L1-L22 ## Enable Model-Level Orchestration Using Airflow -@TODO + +**Step-1:** Create Dag to run dbt project +https://github.com/memiiso/opendbt/blob/ddf0b1bac379aa42961900f35dfc9938b1bc19c4/tests/resources/airflow/dags/dbt_workflow.py#L17-L32 ![airflow-dbt-flow.png](assets%2Fairflow-dbt-flow.png) +#### Create dag using subset of dbt models + +```python +from opendbt.airflow import OpenDbtAirflowProject +# create dbt build tasks for models with given tag +p = OpenDbtAirflowProject(resource_type='model', project_dir="/dbt/project_dir", profiles_dir="/dbt/profiles_dir", + target='dev', tag="MY_TAG") +p.load_dbt_tasks(dag=dag, start_node=start, end_node=end) +``` + +#### Create dag to run tests + +```python +from opendbt.airflow import OpenDbtAirflowProject +# create dbt test tasks with given model tag +p = OpenDbtAirflowProject(resource_type='test', project_dir="/dbt/project_dir", profiles_dir="/dbt/profiles_dir", + target='dev', tag="MY_TAG") +p.load_dbt_tasks(dag=dag, start_node=start, end_node=end) +``` \ No newline at end of file