Generalise Register of Interest import for multiple parliaments

[ajparsons]

We want to be able to import registers of interests for the different Parliaments we cover.

How things work

For UK data, a scraper converts the HTML publication from the UK Parliament into an XML file, with the rough format: “Regmem (person) > category > record > items”.

This is then imported into TheyWorkForYou through mpinfoin.pl - where the xml is converted to a html structure and added to ‘register_member_interests_html’ and ‘register_member_interests_date’ attributes for the person.

The comparison over time is calculated directly off the XMLs files rather than the database. When the page is loaded for an MP, it loads the files and creates the comparison by section.

Adapting for multiple parliaments.

At a baseline, as the most complicated feature that works well, we don’t want to touch the comparison feature too much. So we need to produce files in the XML it expects, with a few switches to look at different folders.

But we also need to get some more data into the database to show the different indexes. Here I think we want to change the existing approach.

Rather than creating the html outside and storing that in the database - let’s convert the current xml to a json and put that in (possibly as a dict with options for different chambers e.g. [“uk”]. This also lets us be more flexible with sticking some extra info in the data for templates that support that.

(https://github.com/mysociety/mysoc-validator can convert the xml into a json - so possibly a new python script would be the right approach)

This would let us create a navigation by section element on the page - which currently we can’t without decoding the html to get the sections. This would also let us plug in descriptions explaining sections later on if we wanted.

Similarly, as the spreadsheet download is based on the XML, adding more to this isn’t especially difficult - but it depends on if you think different spreadsheets or one big spreadsheet is better (both?)