Various fixes for import scripts #613
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We removed 2010 WMC constituencies from the import scripts back in July, because we figured all 2010 data had already been imported into the live site, and we wouldn’t need to import any more. But it’s actually useful to be able to recreate the situation of the live site (with multiple generations of WMCs) in local development too, which means we need the import scripts to keep working with WMCs, rather than relying on developers just having the right data already in their local environments. This commit reinstates the WMC (rather than WMC23) AreaType, and updates our mapit utility function to accept a `generation` parameter that will easily allow us to change the MapIt generation in future again too. With both 2010 and 2023 constituencies coming straight from MapIt now, there is no longer any need for the import_new_constituencies command.
These import scripts all used QuerySet.get() insted of QuerySet.filter() which was causing an exception when both WMC and WMC23 AreaTypes were present in the database.
Also, run the scripts alphabetically for easier debugging.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
import_new_constituenciescommandQuerySet.get()instead ofQuerySet.filter(), which meant they failed when run on a database that contained bothWMCandWMC23AreaTypes.run_all_import_scriptsmanagement command – something to follow up on later!!script/import-all-datainto a dumb wrapper around./manage.py run_all_import_scripts, to reduce duplicationThe upshot of this is that—as long as you have all the required source data in the
/datadirectory—a freshdocker compose upand thenscript/dev-command ./manage.py run_all_import_scriptssuccessfully gets you a largely complete copy of the live site. (What’s missing, of course, is any scripts skipped inrun_all_import_scripts, and anyfeaturedoris_publicstates on DataSets, which must be set manually via the Django admin.)@struan I wasn’t sure whether to leave your debugging commits in, or remove them? Also, should we try to fix the five broken import scripts (the
ValueErrorandJSONDecodeErrorones now listed inrun_all_import_scripts) as part of this PR, or is that something for a separate ticket?