Investigate initial sync speed to determine speed / record, review bottlenecks, approaches to increase speed

[Rkareko]

Describe the issue to be researched
Investigate way to fix performance issues reported following UAT done for WDF

Context
The following is the resource breakdown for the data used for the UAT
Total: 129037 Resources
Patients: 1927 Resources
QuestionnaireResponse: 1927
Observation: 29977
Flag: 15377
Condition: 2562
Appointment: 1740
Task: 542
Encounter: 74985

Phone details i.e make, model, Android OS version and RAM
We have the following two models

• Samsung Galaxy A05 (used for UAT)
  ONE UI 6.1
  OS: Android 14
  RAM: 4GB
• Samsung Galaxy A04

Issues reported (by the client)

1. Initial sync

• Reported to have taken a couple of hours
• For the initial first time sync the device screen should always be on. In case if the screen goes off, some resources are not synced at all

2. QR code search

• The query in the register took about 1 min
• If a query had a match, the "automatic opening" of the patient record took as long as 5 minutes

3. Register search

• Search by QR code (short code)

  • Sometimes it took around 8 - 10 mins and 0 Results
  • Sometimes few seconds delayed to start the search.

• Search by Name

  • Randomly works

• Search by PHN (ID)

  • 6 - 8 Mins
  • Sometimes (rarely) 3 -5 seconds to retrieve

• Overall

  • Sometimes does not return any results or un-relevant results. Example: when I search for 8979, the PHN (ID) / QR Code / Phone does not have 8979 as part of it. But still it returns one record.

• Based on Android studio profiler (when performing a search) - This is from the client

  • CPU usage not more than 60%
  • Memory usage not more 500MB

[Rkareko]

@Mstjamush Test data is needed for validation of the reported issues.

[Mstjamush]

@Rkareko
Please provide the following

1. payload metadata
2. Payload for creating a client/patient
3. payload for creating a task
4. Payload for creating a condition
5. payload for creating Questionnaires and QuestionnaireResponse
6. Payload for creating Encounters

[Rkareko]

Performance improvements when loading a profile.
We have identified potential fixes for this

• Removing nested related resources especially Encounters. The filtering for these would be done using rules. This has been implemented in the profile configs and has greatly improved performance.
• Applying sorting at the db level on the related resources instead of using rules
• Removing unused rules

[Rkareko]

Performance improvements for search.
The proposed solution is to move the search from the current implementation that uses rules to search against data loaded into the register paging source. The search would be moved to the db level.
The use of dynamic queries significantly improves the search time. The limitation here is that the SDK does not provide a way to specify complex operations e.g search for Patients that are active AND ( Patients whose name contains the search term OR Patients whose identifier is equal to the search term). There is a requirement for the search bar to support search by either patient name or patient identifier (QR code, PHN e.t.c)
The following are proposed solutions

1. Enhance the SDK search functionality to support complex search operations such as Patients that are active AND ( Patients whose name contains the search term OR Patients whose identifier is equal to the search term). A Call has been schedule to discuss this with the SDK team on Tuesday 22nd October.

Pros

• Appears to be the more scalable option

Cons

• Due to the intricate nature of the search functionality the LOE for this will likely be higher

2. Implement a custom search param that will extract all the required items i.e Patient.name, Patient.identifier into the StringIndexEntity table. Then define a data query that will search for either the patient name or patient identifier in the newly defined custom search parameter (patient for this instance)
   The following image shows how the items would be indexed. The index name used is patient (A more appropriate name is required)

The data query would be as follows

"dataQueries": [
          {
            "paramName": "patient",
            "filterCriteria": [
              {
                "dataType": "STRING",
                "modifier": "CONTAINS",
                "dataFilterLinkId": "name-search-query"
              }
            ]
          }
        ]

Pros

• Considerably lower LOE since the changes are on FHIRCore

Cons

• Performance concerns for search when data grows since we are introduction new data into the StringIndexEntity table
• Scalability of the solution in terms of adding new search params. Part of this may be adressed by enhanceing the process of adding custom search parameters to be configurable

[dubdabasoduba]

@Rkareko Based on the project timelines, I would be okay with option 2 since it has less LOE.

@pld we will need time to update this to use Option 1 which is more scalable

[faith-mutua]

@Rkareko I have made the following observations when doing performance testing on the qa apk

1. How long QR code search and opening the patient profile - 52.45s
2. How long QR code search takes - 5s
3. How long name search takes - 5s
4. How long PHN search takes - 8.5s
5. How long opening a patient profile takes - 33.3s

[pld]

@Rkareko Based on the project timelines, I would be okay with option 2 since it has less LOE.

@pld we will need time to update this to use Option 1 which is more scalable

I am happy with that

[faith-mutua]

Apk Version - opensrp-dc-fhir-qa-performance_improvements.apk
Number of patients - 7783

@Rkareko I have made the following observations today

1. Searching by name - 1 min 12 seconds
   I searched using the name Ananya which returned 19 patients with that name
2. Subsequent search by name - 55.10s
   I searched using the name Swara which returned 73 patients with that name
3. Searching by PHN - 34.5s
4. Searching by QR code until the opening of patient profile - 22s
   Subsequent Searching by QR code until the opening of patient profile - 15.52s
5. Opening a patient profile - 12.68s

[faith-mutua]

@Rkareko @Gental-Giant @allan-on

I have made the following observations when testing on the release apk and the apk with performance improvements

Apk Version - opensrp-dc-performance-improvements-release.apk
Device Specs - BLUG60 - 3GB RAM, Android 9

Initial Sync - 4hrs 51 mins
Searching by name - 28.63s
Searching by PHN - 12s
Searching by QR code until the opening of patient profile - 5.4s
Opening patient's profile - 4.5s

Apk Version opensrp-dc-v0.1.5-release.apk - Without performance improvements
Device Specs - Samsung Galaxy Tab A(2017) - 2GB RAM, Android 9

Initial Sync - 5hrs 30 mins
Searching by name - The search was processed for 32 minutes then records that do not match the search name(Siya) were returned. Records matching the search keyword Siya were not returned
Searching by PHN - 10s
Searching by QR code until the opening of patient profile - The search was processed for 14 minutes then records that have value combinations matching any of the characters in the QR Code were returned.
Opening patient's profile - 10s

[faith-mutua]

@Rkareko @allan-on @Gental-Giant I have done one more test cycle on a 4 + 1 GB device on the apk with performance improvement and the app's performance has been observed to have improved. Below are the observations recorded

Apk Version opensrp-dc-performance-improvements-release.apk
Device Specs - Readmi 9 - 4 + 1 GB RAM, Android 11
Number of Patients/Records - 7,840

Initial Sync - 3hrs 45s
Searching by name - 23.35s
Searching by PHN - 7s
Searching by QR code until the opening of patient profile - 6s
Opening patient's profile - 3.1s

[allan-on]

SDK support for complex search with nested filtering using AND|OR operations will be added in google/android-fhir#2710

[dubdabasoduba]

@Rkareko @allan-on @Gental-Giant I have done one more test cycle on a 4 + 1 GB device on the apk with performance improvement and the app's performance has been observed to have improved. Below are the observations recorded

Apk Version opensrp-dc-performance-improvements-release.apk Device Specs - Readmi 9 - 4 + 1 GB RAM, Android 11 Number of Patients/Records - 7,840

Initial Sync - 3hrs 45s Searching by name - 23.35s Searching by PHN - 7s Searching by QR code until the opening of patient profile - 6s Opening patient's profile - 3.1s

@faith-mutua @allan-on @Gental-Giant Do you know how many resources translate to the 7840 patients.

[faith-mutua]

@dubdabasoduba unfortunately, the data I used in the test cycle didn't sync down any resources on the devices. @Rkareko will check whether there is a way to filter out the resources and revert.

[dubdabasoduba]

@dubdabasoduba unfortunately, the data I used in the test cycle didn't sync down any resources on the devices. @Rkareko will check whether there is a way to filter out the resources and revert.

Thanks, @faith-mutua I think @Gental-Giant mentioned we have a data dump. That would suffice.

[Rkareko]

@dubdabasoduba @Gental-Giant Here is the breakdown of the resources

Patient - 8114
QuestionnaireResponse - 8114
Observation - 87989
Condition - 9651
Flag - 21561
Appointment - 6808
Encounter - 210726

[Gental-Giant]

@ndegwamartin @dubdabasoduba Here is a link to the sample data.

[faith-mutua]

@Rkareko @Gental-Giant @allan-on I have made the following observations in the new apk with the initial sync improvements.

Apk Version - dc-quest-initial-sync-improvements-release.apk
Device Specs - Readmi 9 - 4 + 1 GB RAM, Android 11

Number of Patients/Records - 7,880
Initial sync - 3hrs 28mins 55s

Once initial sync was done, the screen was blank with no data and the Sync complete text did not show up. Doing a manual refresh, loaded all the data into the app.

[Rkareko]

@Rkareko @Gental-Giant @allan-on I have made the following observations in the new apk with the initial sync improvements.

Apk Version - dc-quest-initial-sync-improvements-release.apk Device Specs - Readmi 9 - 4 + 1 GB RAM, Android 11

Number of Patients/Records - 7,880 Initial sync - 3hrs 28mins 55s

Once initial sync was done, the screen was blank with no data and the Sync complete text did not show up. Doing a manual refresh, loaded all the data into the app.

@ndegwamartin Theses are the QA results using an apk built from the changes in FHIR Core Enhancements

[Rkareko]

The slow search and profile loading issues have been addressed.
The remaining item concerning the slow intiial sync will be tracked on the Perfomance Enhancements ticket. cc @ndegwamartin