Merge pull request #131 from openstates/db-migrations-voteevent-dedup…

…e-index

Add instructions for migrating the database tables representing os-core models

CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## 6.19.3 - Jul 8, 2024
+
+* Add instructions on DB migrations to docs
+* Change schema to fix two DB issues:
+    * Event Document `note` changed to Text type to avoid character limit errors
+    * Vote Event `dedupe_key` index added to improve performance on Vote Event lookups during import
+
 ## 6.19.2 - Jun 7, 2024
 
 * Turn off archiving processed realtime bills by default.
@@ -53,11 +60,11 @@
 ## 6.17.3 - August 1, 2023
 
 * update pyyaml dependency
-      
+
 ## 6.17.2 - August 1, 2023
 
 * Support additional territories in db init
-      
+
 ## 6.17.0 - July 14, 2023
 
 * Add additional US territory support#113
@@ -156,7 +163,6 @@
 
 * add SQS for batching
 
-
 ## 6.13.8 - January 9, 2023
 
 * update us legacy districts
@@ -349,7 +355,7 @@
 * fix os-us-to-yaml phone numbers
 * added latest_bill_update and latest_people_update to Jurisdiction table
   (these fields are now automatically updated when os-update or os-people
-   update the relevant tables)
+  update the relevant tables)
 * moved openstates.reports into openstates.data
 
 ## 6.3.4 - August 2 2021
@@ -460,8 +466,8 @@
 ## 5.8.0 - April 5 2021
 
 * remaining implementation of OSEP #5
-  * remove pupa_id backwards compatibility
-  * migration: drop reports.Identifier
+    * remove pupa_id backwards compatibility
+    * migration: drop reports.Identifier
 * Add veto and veto-override vote classifications
 
 ## 5.7.1 - April 5 2021
@@ -471,9 +477,9 @@
 ## 5.7.0 - April 2 2021
 
 * Implementation of OSEP #5
-  * migration: added dedupe_key to replace pupa_id
-  * added dedupe_key to replace pupa_id in scrape as well
-  * alter import logic to stop using reports.Identifier
+    * migration: added dedupe_key to replace pupa_id
+    * added dedupe_key to replace pupa_id in scrape as well
+    * alter import logic to stop using reports.Identifier
 
 ## 5.6.0 - March 23 2021
 
@@ -542,26 +548,25 @@
 
 * merge openstates_metadata into openstates.metadata
 * large database cleanup:
-  * remove unused fields:
-    * BillAbstract.date
-    * BillAction.extras
-    * BillIdentifier.scheme & note
-    * Jurisdiction.feature_flags
-    * Membership.label
-    * Organization.image, founding_date, dissolution_date
-    * Person.summary
-    * PersonContactDetail.label
-    * Post.start_date & end_date
-    * VoteEvent.end_date
-    * BillDocumentLink & BillVersionLink.text
-    * *.locked_fields
-  * add VoteEvent.order
+    * remove unused fields:
+        * BillAbstract.date
+        * BillAction.extras
+        * BillIdentifier.scheme & note
+        * Jurisdiction.feature_flags
+        * Membership.label
+        * Organization.image, founding_date, dissolution_date
+        * Person.summary
+        * PersonContactDetail.label
+        * Post.start_date & end_date
+        * VoteEvent.end_date
+        * BillDocumentLink & BillVersionLink.text
+        * *.locked_fields
+    * add VoteEvent.order
 
 ## 4.8.0 - August 21 2020
 
 * remove GIS dependency, alter (unused) EventLocation model accordingly
 
-
 ## 4.7.1 - August 5 2020
 
 * remove person_role_division_id

README.md

@@ -11,6 +11,30 @@ This repository contains the Open States data model and scraper backend.
 
 See [RELASE.md](./RELEASE.md)
 
+## Database Migrations
+
+openstates-core data models may occasionally change, requiring changes to the Open States database in production. Here
+are the steps to create and execute a new migration:
+
+* Follow the steps in [Running a Local Database doc](https://docs.openstates.org/contributing/local-database/) to get
+  a copy of the database running locally with the prior schema. It is important to develop and test your migration
+  locally before executing it on the production DB.
+* In most cases, you will auto-generate a migration by first modifying the model file. Make a change to one or more of
+  the files in `openstates/data/models`.
+* Test the migration locally:
+    * Identify the DB connection URL that is accurate to your local database. In most cases, it should
+      be: `postgis://openstates:openstates@localhost:5405/openstatesorg`
+    * In the repo's root folder, run the `os-dbmakemigrations` command to generate a migration file based on your
+      changes: `DATABASE_URL=postgis://openstates:openstates@localhost:5405/openstatesorg poetry run os-dbmakemigrations`
+    * Look at the generated migration file in `openstates/data/migrations` and ensure that it looks correct.
+    * Execute the migration by running the `os-initdb`
+      command: `DATABASE_URL=postgis://openstates:openstates@localhost:5405/openstatesorg poetry run os-initdb`
+* Once the migration is verified by local testing, you can execute it against the production DB
+    * Identify the DB connection URL that is accurate for the PROD database. This typically should use the same Postgres
+      user that owns the tables you want to change. Contact an admin if you need help.
+    * Run the `os-initdb` command to
+      migrate: `DATABASE_URL=postgis://USERNAME_HERE:PASSWORD_HERE@PROD_DB_HOSTNAME_HERE:5432/openstatesorg poetry run os-initdb`
+
 ## Debugging openstates-core code
 
 ### Commands that do not integrate with openstates-scrapers

openstates/cli/makemigrations.py

@@ -0,0 +1,8 @@
+# type: ignore
+from ..utils.django import init_django
+from django.core import management
+
+
+def main() -> None:
+    init_django()
+    management.call_command("makemigrations")

openstates/data/migrations/0045_auto_20240705_1812.py

@@ -0,0 +1,22 @@
+# Generated by Django 3.2.14 on 2024-07-05 18:12
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('data', '0044_bill_citations'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='eventdocument',
+            name='note',
+            field=models.TextField(),
+        ),
+        migrations.AddIndex(
+            model_name='voteevent',
+            index=models.Index(fields=['dedupe_key'], name='opencivicdata_voteevent_dedupe__75a90b_idx'),
+        ),
+    ]

openstates/data/models/event.py

@@ -95,7 +95,7 @@ class Meta:
 
 class EventDocument(RelatedBase):
     event = models.ForeignKey(Event, related_name="documents", on_delete=models.CASCADE)
-    note = models.CharField(max_length=300)
+    note = models.TextField()
     date = models.CharField(max_length=25, blank=True)  # YYYY-MM-DD HH:MM:SS+HH:MM
     classification = models.CharField(
         max_length=50, choices=EVENT_DOCUMENT_CLASSIFICATION_CHOICES, blank=True

openstates/data/models/vote.py

@@ -63,6 +63,9 @@ class Meta:
             ["legislative_session", "identifier", "bill"],
             ["legislative_session", "bill"],
         ]
+        indexes = [
+            models.Index(fields=["dedupe_key"])
+        ]
         ordering = ["start_date", "order"]
 
 

pyproject.toml

@@ -1,13 +1,14 @@
 [tool.poetry]
 name = "openstates"
-version = "6.19.2"
+version = "6.19.3"
 description = "core infrastructure for the openstates project"
 authors = ["James Turk <[email protected]>"]
 license = "MIT"
 
 [tool.poetry.scripts]
 os-update = 'openstates.cli.update:main'
 os-initdb = 'openstates.cli.initdb:main'
+os-dbmakemigrations = 'openstates.cli.makemigrations:main'
 os-update-computed = 'openstates.cli.update_computed:main'
 os-text-extract = 'openstates.cli.text_extract:main'
 os-people = 'openstates.cli.people:main'