Cross-program delivery metrics

[widal001]

Summary

Project stakeholders can access a public dashboard with up-to-date metrics that help us track the status of agile deliverables and monitor our overall delivery health.

Press release

The SimplerGrants team is excited to launch our cross-program delivery metrics dashboard! This tool helps stakeholders track progress toward key deliverables in the roadmap and answer important questions like:

• Are the planned deliverables on track?
• What bottlenecks, if any, are slowing delivery?
• Does the current scope of work exceed available capacity?

The dashboard provides a high-level pulse check that complements direct observations about team capacity and delivery health, and helps the team see how closing tickets moves the needle on our measures of progress.

Acceptance criteria

• Public dashboard: Without logging in, stakeholders can access a public dashboard with metrics about deliverables in the SimplerGrants product roadmap.
• Delivery metrics: At a minimum, this dashboard displays the following metrics:
  • Percentage complete: Percentage of all points completed per deliverable
  • Percentage pointed: Percentage of all tickets that have been pointed per deliverable
  • Deliverable burndown: Number of open tickets/points remaining for a given deliverable per day
  • Deliverable burnup: Number of tickets/points opened and closed for a given deliverable per day
• Cross-program reporting: The dashboard reports on deliverables from multiple vendors in the SimplerGrants program (e.g. Nava, Agile Six, Bloom Works).

Metrics

• Usage: The delivery dashboard has had at least 1000 views. Note: Metabase tracks all views of a public dashboard as an “anonymous user view”.
• Refresh rate: The data in the dashboard is refreshed at least once every 24 hours.
• Lead time for changes: The average lead time for adding new metrics or modifying existing delivery metrics is less than 2 weeks, as measured by the time between the opened and closed date for related GitHub issues.

Assumptions and dependencies

• Assumptions
  • Agile Six will have edit access to a production instance of Metabase
  • Metabase will be connected to a Postgres data warehouse
  • Metabase has received security approval to host dashboards publicly
• Dependencies
  • Public Metabase Dashboard #2225

Related goals

• Develop technical product strategy
  • Building a dashboard to track delivery metrics helps foster a “metrics-based approach to implementation so all of our work is driven by leading and lagging indicators of client satisfaction”
  • Measuring our progress toward key milestones in the roadmap also helps us “ensure that work is delivered in production to real people, delivering real value along the way”
  • This deliverable involves “developing automated tools that assist in tracking project performance and success, such as linters, reporting infrastructure, [and] delivery dashboards”
• Provide delivery management
  • This delivery dashboard is an important part of “a reporting systems that track progress against goals [and] make it easy to report out on the current status of that progress”

Additional information

N/A

[widal001]

Sprint 1.1 updates

Status: 🟢 On track

Sprint goal

The main goal for this deliverable in Sprint 1.1 was to prototype a new delivery dashboard

Accomplishments

• Created a sandbox repo and GitHub projects to reflect the new GitHub norms and use of issue types
• Created a script to export and flatten data from this sandbox repo and project
• Created a script to set up a local database file with a more flexible data schema for reporting
• Created a script to transform and load the flattened data export into this new schema

Rollover

• Finish calculating % complete per deliverable using the new data schema
• Finish calculating burndown/burnup by sprint using the new data schema

Risks

• The rollover of some of the tasks from Sprint 1.1 may defer some work planned for Sprint 1.2
• The team has more limited capacity in Sprint 1.2 due to the onsite
• We're still working through the procurement details around Metabase, which may influence the strategy we adopt to publish the first dashboard

Next sprint

Finish the outstanding items for the prototype, then migrate the prototype code to HHS/simpler-grants-gov and re-enable posting delivery metrics to Slack

[widal001]

Sprint 1.2 updates

Status: 🟡 A little behind

Sprint goal

Finish the outstanding items for the prototype, then migrate the prototype reporting to HHS/simpler-grants-gov and re-enable posting delivery metrics to Slack

Accomplishments

• Finished the delivery metrics prototype
• Migrated the new export step for GitHub data into HHS/simpler-grants-gov repo
• Tested loading GitHub data into Postgres (with @coilysiren's help)
• Updated sprint burndown and sprint burnup to use the new GitHub data format

Rollover

• Update percent complete by deliverable to use the new GitHub data format
• Finish migrating the code that loads GitHub data into Postgres using the new data schema
• Turn on Sprinty McBurndown to post metrics to Slack

Risks

• I'll be out for a most of Sprint 1.3, which may mean we'll need to downscope or defer some work around turning Sprinty McBurndown back on
• We haven't yet tested Metabase in production, so there may be unforeseen challenges that we'll uncover this sprint

Next sprint (Sprint 1.3)

• Create first delivery dashboard in Metabase
• Turn on Sprinty McBurndown reports in Slack

[widal001]

Sprint 1.3 updates

Status: 🟡 Behind

Sprint goal

• Create first delivery dashboard in Metabase
• Turn on Sprinty McBurndown reports in Slack

Accomplishments

• Updated Sprinty McBurndown metrics to use the new GitHub dataset
• Finished migrating code to transform and load data into Postgres into HHS/simpler-grants-gov
• Triaged issues with connecting to Postgres (data warehouse) from the analytics codebase
• Successfully loaded GitHub data into Postgres using the transform and load step (in dev)

Kudos to Kai and Michael for their help!!

Rollover

• Automatically create analytics schema in staging and prod
• Deploy transform and load step for GitHub
• Create first dashboard in Metabase with data from GitHub
• Schedule daily runs of Sprinty McBurndown reports

Risks

• We may continue to encounter errors with the database analytics database connection as we promote the ETL code from dev to staging and production.
• Creating charts in Metabase (once the data is present) may be more difficult than anticipated -- which is why we are planning to spike this in parallel to troubleshooting the database connection.

Next sprint (Sprint 1.4)

• Turn on Sprinty McBurndown reports in Slack
• Create first delivery dashboard in Metabase

[lucasmbrown-usds]

Do we have screenshots of the reporting dashboards working locally (so I can get a sense of what exactly we're building here), or are we just getting the data into Postgres and then we'll load into Metabase and create the dashboards there for the first time?

I guess the actual question is: are there any screenshots or mockups I can see of what we'll be reporting on once we get all this working?

[widal001]

Do we have screenshots of the reporting dashboards working locally (so I can get a sense of what exactly we're building here), or are we just getting the data into Postgres and then we'll load into Metabase and create the dashboards there for the first time?

I guess the actual question is: are there any screenshots or mockups I can see of what we'll be reporting on once we get all this working?

@lucasmbrown-usds Here's an example of the first dashboard for delivery metrics which has both percent of issues and percent of points complete by deliverables. Hovering over the bars shows you the percentage complete by deliverable.

I'll also be posting screenshots to the following tickets:

• Create percent complete by deliverable chart in Metabase #2667
• Create sprint burndown chart in Metabase #2666

[widal001]

Sprint 1.4 updates

Status: 🟡 Mostly caught up

Sprint goal

• Turn on Sprinty McBurndown reports #2661
• Create first delivery dashboard in Metabase #2560

Accomplishments

• Turned on Sprinty McBurndown for QA against Metabase dashboards
• Set up schema versioning and migrations for the analytics data warehouse
• Updated and expanded both delivery metrics and sprint metrics dashboard
• Demonstrated ability to make dashboards visible publicly (in dev)

Rollover

Fixing issue with docker image to unblock automatic migrations in Staging and Prod

Risks

Some of the remaining work to get the dashboards into production has some dependencies on the Nava infrastructure team and capacity is tight.

Next sprint (Sprint 1.5)

• Run ETL process for GitHub data in staging environments #2857
• Update and expand sprint metrics dashboard #3018
• Update and expand delivery metrics dashboard. #3019

Examples

Links to public dashboards

• Delivery metrics
• Sprint metrics

Delivery metrics

Sprint metrics

[widal001]

Sprint 1.5 updates

Status: 🟢 Back on track

Sprint goal

• Run ETL process for GitHub data in staging environments #2857
• Update and expand sprint metrics dashboard #3018
• Update and expand delivery metrics dashboard. #3019

Accomplishments

• Ran ETL for delivery and sprint data in staging
• Finished implementing most of the core metrics for the delivery dashboard
• Added Nava sprint data to the sprint metrics dashboard
• Improved the automated sprint and points linter for closed issues

Rollover

• Update and expand sprint metrics dashboard #3018

Risks

The dashboard is public in dev and ETL has run successfully in staging, the only major risk remaining is if we hit further snags with permission in PROD after the next release that requires further input/troubleshooting with Kai.

Next sprint (Sprint 1.6)

• Finish: Update and expand sprint metrics dashboard #3018
• Promote sprint and delivery metrics dashboards to prod #3155

Acceptance Criteria (ACs) completed

We checked off the following AC on having a live dashboard available publicly, because members of the public can access the following dashboards:

• Delivery metrics
• Sprint metrics

Note: We'll be migrating those dashboards to prod once the next prod release is cut. And the URLs will change once the Nava team creates a domain for Metabase in each environment.

• Public dashboard: Without logging in, stakeholders can access a public dashboard with metrics about deliverables in the SimplerGrants product roadmap.

We also checked off all but one of the individual metrics listed in the ACs. We should have the final metric (% pointed) included by the end of this week.

• Delivery metrics: At a minimum, this dashboard displays the following metrics:
  • Percentage complete: Percentage of all points completed per deliverable
  • Percentage pointed: Percentage of all tickets that have been pointed per deliverable
  • Deliverable burndown: Number of open tickets/points remaining for a given deliverable per day
  • Deliverable burnup: Number of tickets/points opened and closed for a given deliverable per day

[widal001]

Sprint 1.6 updates

Status: 🟢 Core functionality delivered

Sprint goal

• Finish: Update and expand sprint metrics dashboard #3018
• Promote sprint and delivery metrics dashboards to prod #3155

Accomplishments

• Promoted delivery and sprint metrics dashboards to prod 🎉
• Simplified sprint burndown and burnup queries and added sprint-based filtering

Rollover

Note: these elements are off-critical path for the deliverable spec.

• Fixing a bug with the sprint-based filtering
• Adding a few remaining elements to the interactive sprint dashboard

Risks

• Since we just got Metabase working in production, we'll be slow to ramp up to 1000 views

Next sprint (Sprint 1.7)

• Finish: Update and expand sprint metrics dashboard #3018
• Transfer and organize tickets in [Delivery metrics 2.0] Improve analytics/ codebase #2930 into a separate Quad 2 deliverable

ACs completed

The the following ACs were completed now that we have the dashboards live in prod:

Public dashboard

We've checked off the public dashboard deliverable because the dashboards are available in prod:

• Sprint metrics
• Delivery metrics

• Public dashboard: Without logging in, stakeholders can access a public dashboard with metrics about deliverables in the SimplerGrants product roadmap.

Delivery metrics

After adding a chart for percent pointed (see screenshot below), we've checked off all of the criteria below

• Delivery metrics: At a minimum, this dashboard displays the following metrics:
  • Percentage complete: Percentage of all points completed per deliverable
  • Percentage pointed: Percentage of all tickets that have been pointed per deliverable
  • Deliverable burndown: Number of open tickets/points remaining for a given deliverable per day
  • Deliverable burnup: Number of tickets/points opened and closed for a given deliverable per day

Cross-program reporting

The dashboards report on both Agile Six and Nava sprints and deliverables. Note: We are not currently able to report on Bloom Works deliverables and sprints because they are not tracked in GitHub, but it would be easy to add their data to the existing charts if/when they start using GitHub.

• Cross-program reporting: The dashboard reports on deliverables from multiple vendors in the SimplerGrants program (e.g. Nava, Agile Six, Bloom Works).

The following screenshot shows data for deliverables that Nava is driving on (e.g. Search, Engagement Sessions, etc.) as well as those that Agile Six is driving on (e.g. Cross-program delivery metrics, Cross-program team health, etc.)

[DavidDudas-Intuitial]

Sprint 1.7 updates

Status: 🟢 Core functionality delivered

Sprint goal

• Finish: Update and expand sprint metrics dashboard #3018
• Organize tickets in [Delivery metrics 2.0] Improve analytics/ codebase #2930 into a Quad 2 deliverable

Accomplishments

• Added new interactive charts to Sprint Metrics dashboard (see screenshot)
  • table of tickets in sprint
  • percent of tickets pointed
  • sprint start and end dates
  • days left in sprint
• Validated correctness of filtering behavior in Sprint Metrics dashboard
• Created Quad 2 deliverable and epics: [DRAFT] Delivery metrics 2.0 #3377

Rollover

One item remains which precludes us from marking this deliverable as complete:

• Get access to usage analytics for our dashboards #3402

Risks

• Restricted access to usage metrics could continue to block visibility into usage patterns

Next sprint (Sprint 2.1)

• Get access to usage analytics for our dashboards #3402
• Deprecate old analytics/ code #3465
• Gather feedback from delivery dashboard users #3466

[DavidDudas-Intuitial]

Update: usage analytics now available for our dashboards.