Auto testing of migrations

[brassy-endomorph]

Automatically test all (new) migrations.

This is a WIP for initial review. I'd forgotten what a pain in the ass it is to write these kinds of tests, so I wanted to get a sensibility check before I finished.

fixes #555

[jeremywmoore]

I love the ideas presented in this PR, but I wonder if this is trying to do too much, mainly around the UpgradeTester class. Looking at this class I don't have a good intuition around how this will evolve over time as schemas evolve. Maybe we could start out by establishing the goals of this PR? For example, ensuring linear migration history seems great. Another might be that upgrades and downgrades of each revision are applied successfully and that the schema after upgrade + downgrade is unchanged. Maybe a test that checks that the schema generated by migrations matches the dev schema?

If we did want to go to the point of checking data consistency after each upgrade, a dedicated/static test generated for each migration might be easier to maintain in the long run because it would only need to cover a specific migration and not worry about a changing data model. But even that feels like it could be an outside amount of work for the value.

[brassy-endomorph]

The Up/DowngradeTester classes don't evolve. One is written per revision, and they're loaded as tests dynamically. Each tester has a load_data function that loads a realistic set of data for that revision and then has the up/downgrade applied. The class holds data so it can run a check to ensure that the specific parts of the migration we expect to break can be tested. This is also the part of the migration that tends to break the most often, not the schemas matching. I can write tests for that too, but this was taking enough time as it is, and this type of test will be very necessary if we want to switch to UUIDs since that has a much biggest chance of everything going wrong.

[jeremywmoore]

The Up/DowngradeTester classes don't evolve. One is written per revision, and they're loaded as tests dynamically. Each tester has a load_data function that loads a realistic set of data for that revision and then has the up/downgrade applied. The class holds data so it can run a check to ensure that the specific parts of the migration we expect to break can be tested. This is also the part of the migration that tends to break the most often, not the schemas matching. I can write tests for that too, but this was taking enough time as it is, and this type of test will be very necessary if we want to switch to UUIDs since that has a much biggest chance of everything going wrong.

Thanks for the clarification, that makes more sense than my initial read. I think this is a good approach. In general, I assume we'd only expect to write migration test when we're updating existing schemas but not necessarily for additive migrations?

[brassy-endomorph]

Yes that's reasonable. We can add an array of "ignorable" migrations that get filtered out once that becomes an issue.

[micahflee]

Nice! I just read through the code, and this all makes sense. It definitely feels like a lot more work to write migrations because we also have to test them, but also they should be much more robust after this.