Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finishing touches on superset before user testing #3888

Merged

Conversation

bendnorman
Copy link
Member

@bendnorman bendnorman commented Oct 1, 2024

Overview

This PR addresses the remaining high priority tasks in #3867:

  • Adds a script that can programmatically create a PUDL table dashboard on superset
  • Doubles the memory of a cloud run instance for superset
  • Adds a CPU start up boost to the cloud run instance for superset
  • Adds the catalyst logo file to git for superset

Tasks

Preview Give feedback

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

Preview Give feedback

@bendnorman bendnorman requested a review from jdangerx October 1, 2024 04:31
@bendnorman bendnorman self-assigned this Oct 1, 2024
@bendnorman bendnorman linked an issue Oct 1, 2024 that may be closed by this pull request
Comment on lines 224 to +228
limits = {
cpu = "4"
memory = "2048Mi"
memory = "4096Mi"
}
startup_cpu_boost = true
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll have to play around with the Cloud Run resources and superset row limit to make everything feel smooth. I decided not to allocate a minimum number of instances because it would cost us about $200 a month. I figured a memory and start CPU boost was reasonable.

Comment on lines -22 to -23
depends_on:
- postgres
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docker throws an error when you run the prod profile because the postgres services isn't started.

python ./automation/create_table_dashboards.py [TABLE_NAMES]...
```

### Limitations / Open questions
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is deserving of a more thorough right up which we can do later if we want to move superset into production.

@bendnorman bendnorman added superset terraform Issue related to terraform labels Oct 1, 2024
Copy link
Member

@jdangerx jdangerx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bulk of this is the Superset API client, which seems fine, though the whole system feels a little bit like a house of cards - we're definitely trying to force Superset into something it doesn't want to do with all this dashboard-automation-for-an-alternative-UI thing...

@jdangerx jdangerx merged commit f49ea32 into adjust-usage-metrics-gcp-permissions Oct 4, 2024
17 checks passed
@jdangerx jdangerx deleted the add-superset-automation-script branch October 4, 2024 15:03
jdangerx pushed a commit that referenced this pull request Oct 4, 2024
* Add dashboard creation automation, fix docker compose error, add logo, increase superset cloud run resources

* Add jinja extensnion to template
github-merge-queue bot pushed a commit that referenced this pull request Oct 4, 2024
* Give usage metric archiver sa bucket get permission and add mapbox api key to superset

* Add logo to superset, enable horizontal filtering and tagging system

* Finishing touches on superset before user testing (#3888)

* Add dashboard creation automation, fix docker compose error, add logo, increase superset cloud run resources

* Add jinja extensnion to template
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
superset terraform Issue related to terraform
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Make superset dashboard improvements for user testing
2 participants