Fields seem to have disappeared from some forms

[cbunicef]

Here is an example:

https://zatest.iogt.site/admin/pages/305/edit/

There were submissions made to this survey in the past. However now no data is accessible because the form fields have been removed.

To my knowledge it's not possible for an Admin to remove all form fields. I tested this and pressing the delete button on a field which is the only field in a survey has no effect.

There are more forms that have been affected, but I'm not sure I see any commonality between them:

https://za.goodinternet.org/admin/pages/284/edit/
https://za.goodinternet.org/admin/pages/306/edit/
https://za.goodinternet.org/admin/pages/305/edit/
https://za.goodinternet.org/admin/pages/303/edit/
https://za.goodinternet.org/admin/pages/302/edit/
https://za.goodinternet.org/admin/pages/297/edit/
https://ss.goodinternet.org/admin/pages/148/edit/
https://tz.goodinternet.org/admin/pages/522/edit/

This isn't an exhaustive list but just the ones I've seen.

@istride @sheralim012 please take a look as some of these forms are live and receiving submissions but have no fields.

[sheralim012]

Looking into it.

[cbunicef]

This is an example of a form which has no fields listed in Admin Panel, but has fields listed in the /questionnaires/id/ API endpoint:

https://za.goodinternet.org/admin/pages/310/edit/

[istride]

This problem is happening due to the way that 'page revisions' (drafts) have been migrated from v1 i.e. verbatim and without the modifications that would be required to make them work in v2. As such, this problem could affect any migrated survey that is published without first saving a valid v2 page revision (save as draft).

In terms of the Wagtail admin UI, the problem can be avoided by saving a draft of a migrated survey before publishing it.

Saving as draft creates a new page revision, and publishing applies the changes in the page revision to the actual survey. If a draft is not saved first, Wagtail will attempt to apply a v1 page revision to a v2 survey, and the end result is that all questions are deleted because the references in the v1 page revision are only valid in a v1 database.

I have inspected the IoGT South Africa database and identified two categories of survey, in relation to this issue:

• affected: migrated surveys that have no questions
• at risk: migrated surveys that have not been saved as draft or published yet

The suggested remedies are:

• affected: restore questions from the latest v1 page revision
• at risk: save as draft

I have created a script that identifies the two types of survey and can automatically apply the suggested fix. I am confident that the remedy for 'at risk' surveys will always be the right thing to do. The remedy for 'affected' surveys is usually the right thing to do, but it might be necessary to double-check on a case-by-case basis to be sure. We should discuss this further before applying the fixes to the South Africa database.

The fixes will also have to be performed on all sites that have been migrated from v1.

[cbunicef]

Thanks for this investigation @istride

The fix for at risk surveys can go ahead.

For affected surveys, how do you propose that we manage the double-checking?

[istride]

I propose that we start with South Africa. I will create a backup of the database before starting, in case anything goes wrong, then apply the fix. Only surveys without questions will be affected (6 for ZA), and they will have their questions restored from the latest version of the survey before the migration to v2.

I think this will be sufficient to fix the problem because I have tested the fix against a replica of the South Africa database, however, to make sure, I would like a second opinion, either from someone at the country office, or yourself, @cbunicef. The fix can be applied next week, with no downtime, and the verification can come whenever.

How does that sound?

[cbunicef]

Sounds good Ian. Let's apply once we are finished with Matomo setup, starting with South Africa and moving down the line.