Refactor: Benefits Amplitude events #3468

thekaveman · 2024-09-19T21:24:35Z

Description

We recently completed a big refactor of the models in Benefits, see cal-itp/benefits#1666 for more background.

The last piece of this refactor is updating our new and historic analytics events. The following PRs update the logic for generating new events:

And this PR is for the warehouse side, to handle the new fields and adjust historical data already captured in GCS.

We don't want to merge this PR until all of the above PRs are merged and released to our prod environment.

Closes cal-itp/benefits#2247
Closes cal-itp/benefits#2248
Closes cal-itp/benefits#2249
Closes cal-itp/benefits#2390

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation

How has this been tested?

 poetry run dbt run -s +fct_benefits_events

$ poetry run dbt run -s +fct_benefits_events
19:38:34  Running with dbt=1.5.1
19:38:35  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
19:38:35  Found 420 models, 950 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
19:38:35  
19:39:53  Concurrency: 8 threads (target='dev')
19:39:53  
19:39:53  1 of 2 START sql view model kegan_staging.stg_amplitude__benefits_events ....... [RUN]
19:39:54  1 of 2 OK created sql view model kegan_staging.stg_amplitude__benefits_events .. [CREATE VIEW (0 processed) in 1.22s]
19:39:54  2 of 2 START sql table model kegan_mart_benefits.fct_benefits_events ........... [RUN]
19:40:15  2 of 2 OK created sql table model kegan_mart_benefits.fct_benefits_events ...... [CREATE TABLE (26.9m rows, 73.1 GiB processed) in 20.37s]
19:40:15  
19:40:15  Finished running 1 view model, 1 table model in 0 hours 1 minutes and 39.82 seconds (99.82s).
19:40:15  
19:40:15  Completed successfully
19:40:15  
19:40:15  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

Go through any charts etc. using the deprecated fields, and update to the new fields
Go through any charts etc. using old values for e.g. eligibility_verifier, and update to the new values
Delete the deprecated fields

github-actions · 2024-09-19T21:28:27Z

Warehouse report 📦

DAG

Legend (in order of precedence)

Resource type	Indicator	Resolution
Large table-materialized model	Orange	Make the model incremental
Large model without partitioning or clustering	Orange	Add partitioning and/or clustering
View with more than one child	Yellow	Materialize as a table or incremental
Incremental	Light green
Table	Green
View	White

angela-tran

Thank you for the code walkthrough and explanations! These changes look good to me. 👍

evansiroky

Can you output the logs of dbt run to ensure this works properly? See #3502 for an example of how this is done.

thekaveman · 2024-10-17T21:21:58Z

Can you output the logs of dbt run to ensure this works properly? See #3502 for an example of how this is done.

@evansiroky @vevetron I'm following these instructions: https://github.com/cal-itp/data-infra/blob/main/warehouse/README.md

And I have to say, this is just a brutal developer experience...

Need poetry installed to be able to run poetry install
Need graphviz installed to be able to install pygraphviz (from poetry install)
All instructions assume brew, which is a MacOS tool. I'm on Linux.
None of this works from within the included devcontainer config

Does everyone run this on a Mac? I've tried to update the devcontainer to be able to get all this running locally. I got as far as:

installing poetry and brew
running brew install graphviz
using the workaround export CFLAGS... and export LDFLAGS mentioned in the above README

But I still get an error when running poetry install at the pygraphviz step:

/workspaces/data-infra/warehouse$ echo $CFLAGS
-I /home/linuxbrew/.linuxbrew/opt/graphviz/include

/workspaces/data-infra/warehouse$ echo $LDFLAGS
-L /home/linuxbrew/.linuxbrew/opt/graphviz/lib

/workspaces/data-infra/warehouse$ poetry install
The currently activated Python version 3.8.17 is not supported by the project (~3.9).
Trying to find and use a compatible version. 
Using python3.9 (3.9.2)
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  - Installing pygraphviz (1.11): Failed

...

creating build/temp.linux-x86_64-cpython-39/pygraphviz
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -I /home/linuxbrew/.linuxbrew/opt/graphviz/include -fPIC -DSWIG_PYTHON_STRICT_BYTE_CHAR -I/tmp/tmpuiqe4_ep/.venv/include -I/usr/include/python3.9 -c pygraphviz/graphviz_wrap.c -o build/temp.linux-x86_64-cpython-39/pygraphviz/graphviz_wrap.o
  pygraphviz/graphviz_wrap.c:168:11: fatal error: Python.h: No such file or directory
    168 | # include <Python.h>
        |           ^~~~~~~~~~

Any idea how to get this working?

thekaveman · 2024-10-17T21:39:42Z

Alternatively, if you all are already setup to run these DBT commands for verification, that would be really helpful.

warehouse/models/mart/benefits/fct_benefits_events.sql

vevetron · 2024-10-18T19:19:19Z

I think everyone who works with DBT right now either uses a local mac or jupyterhub to run and test changes. Linux should work as well, but I don't think anyone is using devcontainers.

thekaveman · 2024-10-18T20:23:03Z

Thanks @vevetron. I got a hold of a Macbook and got as far as running poetry run dbt debug but it gave me this output:

(.venv) kegans-MBP:warehouse kegan$ poetry run dbt debug
20:17:52  Running with dbt=1.5.1
20:17:52  dbt version: 1.5.1
20:17:52  python version: 3.9.6
20:17:52  python path: /Users/kegan/git/data-infra/warehouse/.venv/bin/python
20:17:52  os info: macOS-14.2-arm64-arm-64bit
20:17:52  Using profiles.yml file at /Users/kegan/.dbt/profiles.yml
20:17:52  Using dbt_project.yml file at /Users/kegan/git/data-infra/warehouse/dbt_project.yml
20:17:52  Configuration:
20:17:52  Error importing adapter: No module named 'dbt.adapters.bigquery'
20:17:52    profiles.yml file [ERROR invalid]
20:17:52    dbt_project.yml file [OK found and valid]
20:17:52  Required dependencies:
20:17:52   - git [OK found]

20:17:52  1 check failed:
20:17:52  Profile loading failed for the following reason:
Runtime Error
  Credentials in profile "calitp_warehouse", target "dev" invalid: Runtime Error
    Could not find adapter type bigquery!

My ~/.dbt/profiles.yml file looks like:

calitp_warehouse:
  outputs:
    dev:
      dataproc_batch:
        runtime_config:
          container_image: gcr.io/cal-itp-data-infra/dbt-spark:2023.3.28
          properties:
            spark.dynamicAllocation.maxExecutors: '16'
            spark.executor.cores: '4'
            spark.executor.instances: '4'
            spark.executor.memory: 4g
      dataproc_region: us-west2
      fixed_retries: 1
      gcs_bucket: test-calitp-dbt-python-models
      location: us-west2
      maximum_bytes_billed: 2000000000000
      method: oauth
      priority: interactive
      project: cal-itp-data-infra-staging
      schema: kegan
      submission_method: serverless
      threads: 8
      timeout_seconds: 3000
      type: bigquery
  target: dev

And bq ls has output that seems like I have a connection:

                datasetId                 
 ---------------------------------------- 
  airtable                                
  amplitude                               
  audit                                   
  calitp_py                               
  charlie                                 
  charlie_dbt_test__audit                 
  charlie_gtfs_schedule                   
  charlie_gtfs_views_staging              
  charlie_intermediate                    
  charlie_mart_ad_hoc                     
  charlie_mart_agency_service             
  charlie_mart_feed_aggregator_checks     
  charlie_mart_gtfs                       
  charlie_mart_gtfs_guidelines            
  charlie_mart_gtfs_quality               
  charlie_mart_ntd                        
  charlie_mart_payments                   
  charlie_mart_transit_database           
  charlie_payments                        
  charlie_staging                         
  charlie_views                           
  christian                               
  christian_mart_ad_hoc                   
  christian_mart_audit                    
  christian_mart_benefits                 
  christian_mart_gtfs                     
  christian_mart_gtfs_quality             
  christian_mart_gtfs_schedule_latest     
  christian_mart_ntd                      
  christian_mart_payments                 
  christian_mart_transit_database         
  christian_mart_transit_database_latest  
  christian_staging                       
  ci_staging                              
  eric                                    
  eric_mart_ad_hoc                        
  eric_mart_audit                         
  eric_mart_benefits                      
  eric_mart_gtfs                          
  eric_mart_gtfs_quality                  
  eric_mart_gtfs_schedule_latest          
  eric_mart_ntd                           
  eric_mart_payments                      
  eric_mart_transit_database              
  eric_mart_transit_database_latest       
  eric_payments                           
  eric_staging                            
  eric_views                              
  erika                                   
  erika_dbt_test__audit

Will come back to this a little later and look into it more.

vevetron · 2024-10-18T21:48:47Z

Your profiles.yml looks exactly the same as mine. My debug statement is almost the same as well.

Maybe retry poetry install? or pip install dbt-bigquery? or maybe it's running the wrong environment.

thekaveman · 2024-10-21T19:33:55Z

Finally got it running!

I am seeing the same error output that you showed:

$ poetry run dbt run -s +fct_benefits_events
19:32:16  Running with dbt=1.5.1
19:32:16  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
19:32:17  Found 420 models, 950 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
19:32:17  
19:32:20  Concurrency: 8 threads (target='dev')
19:32:20  
19:32:20  1 of 2 START sql view model kegan_staging.stg_amplitude__benefits_events ....... [RUN]
19:32:21  1 of 2 OK created sql view model kegan_staging.stg_amplitude__benefits_events .. [CREATE VIEW (0 processed) in 1.26s]
19:32:21  2 of 2 START sql table model kegan_mart_benefits.fct_benefits_events ........... [RUN]
19:32:23  BigQuery adapter: https://console.cloud.google.com/bigquery?project=cal-itp-data-infra-staging&j=bq:us-west2:ee6d3a66-62ef-49c5-818c-709b8d75e98a&page=queryresults
19:32:23  2 of 2 ERROR creating sql table model kegan_mart_benefits.fct_benefits_events .. [ERROR in 2.17s]
19:32:23  
19:32:23  Finished running 1 view model, 1 table model in 0 hours 0 minutes and 6.64 seconds (6.64s).
19:32:23  
19:32:23  Completed with 1 error and 0 warnings:
19:32:23  
19:32:23  Database Error in model fct_benefits_events (models/mart/benefits/fct_benefits_events.sql)
19:32:23    Unrecognized name: event_properties_claims_provider at [158:9]
19:32:23    compiled Code at target/run/calitp_warehouse/models/mart/benefits/fct_benefits_events.sql
19:32:23  
19:32:23  Done. PASS=1 WARN=0 ERROR=1 SKIP=0 TOTAL=2

Will work on getting these corrected.

thekaveman · 2024-10-21T19:42:59Z

@vevetron I updated the PR description with the results of running locally, which is now passing.

deprecate old cols in mart definition

default to 'digital' for historical events

caused by a bug in the Docker build process

causing a DBT build error

thekaveman added the project-benefits label Sep 19, 2024

thekaveman self-assigned this Sep 19, 2024

This was referenced Oct 1, 2024

Refactor: analytics for claims_provider cal-itp/benefits#2401

Merged

Model refactor: plan for impacts to analytics cal-itp/benefits#2231

Closed

This was referenced Oct 8, 2024

Backfill the EnrollmentEvents table from Amplitude/Metabase data cal-itp/benefits#2436

Open

Refactor: rename the finished card tokenization event cal-itp/benefits#2450

Merged

thekaveman force-pushed the refactor/benefits-events branch from 4c73df3 to 7aa86fb Compare October 10, 2024 17:53

thekaveman marked this pull request as ready for review October 10, 2024 18:04

thekaveman requested review from evansiroky, vevetron and tiffanychu90 as code owners October 10, 2024 18:04

thekaveman requested review from machikoyasuda, angela-tran, vevetron, evansiroky, tiffanychu90, ohrite and erikamov and removed request for evansiroky, tiffanychu90 and vevetron October 10, 2024 18:04

thekaveman force-pushed the refactor/benefits-events branch 2 times, most recently from cca1169 to 7a794db Compare October 15, 2024 15:43

angela-tran approved these changes Oct 15, 2024

View reviewed changes

evansiroky reviewed Oct 17, 2024

View reviewed changes

vevetron requested changes Oct 18, 2024

View reviewed changes

warehouse/models/mart/benefits/fct_benefits_events.sql Outdated Show resolved Hide resolved

thekaveman requested a review from vevetron October 21, 2024 19:42

evansiroky approved these changes Oct 22, 2024

View reviewed changes

vevetron approved these changes Oct 23, 2024

View reviewed changes

thekaveman added 9 commits October 25, 2024 11:38

refactor(benefits): rename 'selected enrollment flow' event

fe74b32

refactor(benefits): coalesce enrollment_flows props

0d37b2a

deprecate old cols in mart definition

chore(benefits): update historical eligibility verifier names

bb35808

refactor(benefits): coalesce claims_provider props

395d193

deprecate old cols in mart definition

refactor(benefits): coalesce enrollment_group props

ec62b1e

deprecate old cols in mart definition

refactor(benefits): rename card tokenization events

3e26875

feat(benefits): add new enrollment_method cols

4f305e0

default to 'digital' for historical events

fix(benefits): correct app versions

2512895

caused by a bug in the Docker build process

fix(benefits): remove aliases for old columns

34aa4a0

causing a DBT build error

thekaveman force-pushed the refactor/benefits-events branch from 83ff624 to 34aa4a0 Compare October 25, 2024 18:38

thekaveman removed request for ohrite, erikamov, machikoyasuda and tiffanychu90 October 25, 2024 18:39

thekaveman merged commit dfab12a into main Oct 25, 2024
4 checks passed

thekaveman deleted the refactor/benefits-events branch October 25, 2024 18:43

This was referenced Oct 25, 2024

Refactor: run DBT from devcontainer #3515

Open

Fix: reorder Benefits mart fields #3518

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor: Benefits Amplitude events #3468

Refactor: Benefits Amplitude events #3468

thekaveman commented Sep 19, 2024 •

edited

Loading

github-actions bot commented Sep 19, 2024 •

edited

Loading

angela-tran left a comment

evansiroky left a comment

thekaveman commented Oct 17, 2024 •

edited

Loading

thekaveman commented Oct 17, 2024

vevetron commented Oct 18, 2024

thekaveman commented Oct 18, 2024 •

edited

Loading

vevetron commented Oct 18, 2024

thekaveman commented Oct 21, 2024

thekaveman commented Oct 21, 2024 •

edited

Loading

Refactor: Benefits Amplitude events #3468

Refactor: Benefits Amplitude events #3468

Conversation

thekaveman commented Sep 19, 2024 • edited Loading

Description

Type of change

How has this been tested?

Post-merge follow-ups

github-actions bot commented Sep 19, 2024 • edited Loading

DAG

angela-tran left a comment

Choose a reason for hiding this comment

evansiroky left a comment

Choose a reason for hiding this comment

thekaveman commented Oct 17, 2024 • edited Loading

thekaveman commented Oct 17, 2024

vevetron commented Oct 18, 2024

thekaveman commented Oct 18, 2024 • edited Loading

vevetron commented Oct 18, 2024

thekaveman commented Oct 21, 2024

thekaveman commented Oct 21, 2024 • edited Loading

thekaveman commented Sep 19, 2024 •

edited

Loading

github-actions bot commented Sep 19, 2024 •

edited

Loading

thekaveman commented Oct 17, 2024 •

edited

Loading

thekaveman commented Oct 18, 2024 •

edited

Loading

thekaveman commented Oct 21, 2024 •

edited

Loading