Sync v5

[Chris-Petty]

note: Carry over tasks from this epic were moved to #608

• Send correct headers app-name and app-version (possibly already done, needs to work for v3 push for short term as well)
• Create sync-v5 feature branch
• Sync v5 sync push to use sync v5 instead of v3 #165
• Sync V5 handle awaiting integration
• Handle deletes when pulling from queued_records #246
• Handle deletes when pulling central records #244
• Number table sync adjustment #177
• Sync integration tests #254
• Consolidate Sync Buffer and Translations #229
• Tests to have info
• Shouldn't need to change code to run remote_sync_integration_test #257
• Name store visibility not updating #249
• #81 (1) JsonForms skeleton #148
• Relocate sync to services #197
• Is it possible to change visibility via name store join without deleting it ?
• Do we need to only integrate remote data deletes if site is not current site ?
• Updates to name is customer and is supplier to reflect in name store join #250
• Add v5 'site' info #357
• Central sync integration test #327
• Deletes for remote integration test #330
• Documentation for sync integration tests #333
• PR review changes up to 333 #338

Transfers

• Handle generating inbound shipments after pulling from queued_records
• Handle generating response requisitions after pulling from queued_records
• In open-msupply, only immediately generate transfers if both stores are active on the remote

For the above, we basically want the sync processor to use change_log (crawl though it and do the actions it needs), and we just trigger sync processor when we need to (after sync, after insert/update/delete of request requisition and outbound shipment). Using change_log would mean it needs to be more consumable (should not just return id and record type, but the records themselves). Also we shouldn't try to push records that don't belong to current site.
Sync processor log ?

[andreievg]

To do to the list:

• V5 push
• V5 await integration
• Make sure transfers work when initialised (i.e. when other half is received during initialisation, and it doesn't have duplicate, duplicate)
• When name becomes a store and initialised (store.created_date) ?

Currently, omSupply is working with V3 push and V5 pull (remote, central). A few things are missing though (like inter sync site transfers), but this is not required for immediate customers (the ones in pipeline for June). To keep things 'stable', we aim to:

1. Create sync/v5 branch for omSupply (since omSupply main is tested, existing v3 pull and v5 push works, so let's not break it with v5 push PRs).
2. Sync/v5 branch in mSupply to be updated from master (not develop), so that it can be built at any time and be a 'stable' branch for omSupply demo/deploy.

If there are any breaking changes to v3 push or v5 pull, we would need to put those through straight away to omSupply and redeploy (so hopefully this is avoided). And in sync/v5 omSupply branch we can work on v5 push and other associated bits and pieces.

Quite a bit of testing is needed to get sync/v5 mSupply branch merged into mainstream (v5 push required some minor changes to v1, and thus needs to be thoroughly tested), but we would only need sync/v5 branch merged if we want existing mSupply sites to use omSupply (which implies backwards compatibility), it doesn't look like we have immediate customers wanting to do this, so technically speaking, merge of sync/v5 is not urgent as long as it doesn't lag behind main. (and we can use it as a fork/trunk mSupply build for omSupply)

[andreievg]

• Use cursor instead of record acknowledgement (also would mean we can use one combined type for central and remote sync record)

[andreievg]

• Dynamic batch size, https://github.com/openmsupply/open-msupply/pull/328/files/351494eeb99c88e1d253d03f1c254961381ba585#r922920627

[andreievg]

Had a discussion with Chris:

• After transfer work, will close this issue and leftovers will go into sync cary over/improvements issue
• Edge case that we should deal with, is item not in master list but have stock for it, a few ways to achieve this scenario (master list adjustment, inbound shipment transfer received with items not in master list for store, etc..)
• Use cursor when pulling remote records (requires cursor being added to remote queue pull)
• Integration test for transfers should be 2 sites, but extend to be 3 site in the future
• Some extra validation (like delete data pull translations should check that site_id is not current site, i.e. only forwarded remote records can be deleted through sync)
• When name on outbound shipment or request requisition is changed, we need to fully re-create the invoice (deleting the old one) issue Chris created

[andreievg]

Transfers

• Add documentation for transfers logic #570
• Changelog to be filterable by name_id and store_id #572
• Processors driver #575
• Requisition transfer processor #578
• Shipment transfer processor #580
• Update sync to use filtered change log #582
• Transfer sync integration tests #584
• Transfer logic head and PR change request #586

Other

• Remove site id from sync settings input/mutations, and relocate query to get site id before going out of bootstrap mode
• Update from develop

More edge cases

• If there are deletes of transfers that don't belong to current site in sync queue, and initialisation is requested, they are removed, could lead to InboundShipment of picked status not being deleted
• Similar, merge records might be removed when initialisation is requested

Below diagram in google drive

• Remove is_initialised from syncrhoniser ?
• Use enums vs inde check of invoice status (there is a branch for it somewhere)
• Improve ActiveRecordCheck, in memory active store/name list
• Test use cases
• Logging
• Uniform test creator (that produces service provider, etc..)
• More protection of infinite looping (578 requisition transfer processor #579 (comment))

[andreievg]

Added a comment about cary over tasks being move at the root comment in this issue, closing as it's getting a bit messy