Batch Job for Generating Bill Summaries

[Mephistic]

Problem

We now have an LLM-based summary generator for Bills and a data model that supports storing bill summaries - we just have to link the two together and actually generate summaries for our existing bills. We need a job that can be manually invoked whenever there is either a new batch of bills (e.g. with a new session) or a new model version to summarize all existing bills with the current model.

Success Criteria

• Idempotent, Manually-invoked Script that iterates over existing bills, calls the Cloud Function to generate summaries, and saves the resulting summary to our Firestore DB
  • Ideally, job should have following flags (for testing + preserving external AI credits):
    • --dry-run (Doesn't actually generate summaries, only prints out the billIds that we would generate summaries for)
    • --bills billId1 billId2 (If specified, only generates summaries for the given bills. If not specified, defaults to generating summaries for all bills).
    • --version V1 (If specified, the version should be propagated into the resulting BillSummary document saved in the DB. We should also only generate and save a new summary if there is not an existing summary with the supplied version (for idempotency if a job fails halfway through).

Additional Links

• Ticket for Cloud Function Wrapper for LLM Bill Summary Generator:
• Ticket for the base Bill Summary data model setup:

[Mephistic]

We already have a rough version of this that scrapes data once and then allows us to bulk upload the results to QA/PROD thanks to Pranay.

My hunch is that the batch version of this is always going to be a manual process so the existing script can just be tweaked to fit our criteria. I'm going to close this in favor of continuing to use Pranay's script (which we can update with additional features as needed). The work to generate summaries/tags for new bills that come up after the initial dump is still valuable and in progress (though we should likely consider adding a cutoff to the ongoing worker to ensure that it does not process bills from the next court before we can run the batch job on it).