From c0df815ff159f79b4fa356f110682c13a3c2acfa Mon Sep 17 00:00:00 2001 From: Malcolm Smith Date: Sun, 7 Jul 2024 00:03:37 +0100 Subject: [PATCH] Add performance test procedure --- release/README.md | 5 ++-- release/performance.md | 67 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+), 2 deletions(-) create mode 100644 release/performance.md diff --git a/release/README.md b/release/README.md index 3f3075860a..d2ff4cb48c 100644 --- a/release/README.md +++ b/release/README.md @@ -46,8 +46,9 @@ same devices. Open the pkgtest app in Android Studio, and temporarily edit the top-level build.gradle file to use the local Chaquopy version. -Record sizes and startup times (#5683), and investigate if significantly worse than the -previous version. Remember that the tests and the packages themselves may have changed. +Record performance data in performance.md, and investigate if significantly worse than +the previous version. Remember that the tests and the packages themselves may have +changed. ## Package tests diff --git a/release/performance.md b/release/performance.md new file mode 100644 index 0000000000..36f0780eec --- /dev/null +++ b/release/performance.md @@ -0,0 +1,67 @@ +# Performance test procedure + +Use the pkgtest app as follows (process for older versions varied as noted in the +table): + +* Chaquopy release build +* App debug build +* scipy and matplotlib +* abiFilters set to arm64-v8a and x86_64 + +Before each "First run", go to app info and clear storage. For "Second run", go to app +info and force stop, because swiping away in the Recents screen does not reliably kill +the process. + +Times are in seconds, best of 3: +* "Startup" is the time from the logcat message "Displayed com.application.id: +123ms". +* "Test" is the time reported by the tests themselves. + +Sizes are as reported in Settings: +* "App MB" includes not only the APK, but also extracted native libraries, and probably + AOT-compiled Java code. +* "Data MB" is difficult to pin down on the Pixel 7, maybe for reasons connected to the + WebView mentioned below: + * It's usually a few MB larger than the size of the /data/data subdirectory, except + perhaps the first time you check it after clearing storage and re-running. But then + leaving and re-entering the Storage page makes the extra size appear permanently, so + this is included in the table. + * Sometimes it increases by an *additional* 4 MB while the app is running, but this + always goes away when you force stop, so it's *not* included in the table. + +``` + First run Second run + Startup Test Startup Test App MB Data MB +Samsung J2 + 8.0.0 3.46 13.81 3.20 7.73 76.7 59.2 + 8.0.1 3.57 13.80 3.38 7.79 76.9 59.2 + 9.0.0 3.56 14.25 3.34 7.70 77.3 59.2 + 9.1.0 3.49 14.29 3.30 7.59 78.5 59.3 + +Nexus 4 + 9.1.0 3.52 17.49 3.48 8.91 78.8 59.3 + 10.0.1 3.82 19.50 3.61 9.04 76.8 59.4 + +* Updated NumPy from 1.17.4 to 1.19.5. +* Changed Chaquopy build type from debug to release. + + 10.0.1 3.62 19.46 3.31 11.01 75.7 58.7 + 11.0.0 3.62 20.39 3.28 11.08 75.7 58.6 + 12.0.0 3.65 19.03 3.35 10.47 76.0 58.7 + 12.0.1 3.54 20.68 3.47 10.43 76.0 58.6 + 13.0.0 3.45 18.17 3.08 10.76 76.2 58.8 + +* I think the Data increase here was caused by a Chrome update on the device, which + means we now have a 4 MB file app_webview/BrowserMetrics-spare.pma. Re-testing 13.0.0 + now shows a Data size of 62.7 MB, the same as 14.0.2. + + 14.0.2 3.40 18.97 3.06 11.31 76.1 62.7 + +* Changed abiFilters from armeabi-v7a and x86 to arm64-v8a and x86_64. +* Startup times now use the time from the "Displayed com.application.id: +123ms" message. + Previously we measured from the first to the last message mentioning the application + ID, which is roughly the same. + +Pixel 7 + 14.0.2 0.64 1.51 0.62 0.89 78.7 67.7 + 15.0.1 0.61 1.54 0.55 0.93 78.7 68.0 +``` \ No newline at end of file