[HOLD for payment 2024-07-10] Implement Flashlight tool to the testing suite

[mountiny]

Coming from this Slack proposal which was discussed in the thread.

Problem

When making changes to our codebase, we aim to identify any performance regressions before they infiltrate the production app. For this purpose, we have end-to-end (e2e) performance regression tests running in our Github actions. (These had been non-functional for a time but are now operational again!)

Recently, a significant performance regression entered the production app unnoticed. Our systems should be robust enough to intercept any form of performance regression. However, the issue slipped through because our system currently only measures one performance metric: Time to Interactive (TTI).

Other performance metrics, such as the app's UI Frames Per Second (FPS), RAM, and CPU usage, can also signal regressions. The recent performance regression would have get caught with these metrics.

Consider this scenario: You've implemented a new image component, and while testing the app, everything appears fast and efficient. The performance regression test suite shows no change in TTI, so everything seems in order. Yet, unbeknownst to you, RAM usage has tripled. Consequently, we begin to see Out of Memory (OOM) crashes on lower-end devices after your pull request is merged. These types of regressions should not go unnoticed, correct?

Solution

Our existing e2e testing suite is constructed in such a way that we can readily incorporate additional performance metrics into our test flows. I propose that we add a tool called "Flashlight" to our test suite.

Flashlight will measure the e2e flows for UI FPS, RAM, and CPU usage. We can then include these values in our test reports for comparison.

By employing this approach, we can identify a broader range of regressions. This will give us greater confidence in the updates we introduce to our production environment.

[melvin-bot]

Triggered auto assignment to @lschurr (NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details.

[hannojg]

👋

[lschurr]

From the Slack proposal, it looks like @hayata-suenaga will be owning this internally. Is that correct? Do we need anything from BZ here?

[hayata-suenaga]

No, we don't; sorry I forgot to remove the assignment 🙇

[hayata-suenaga]

@hannojg what are some benefits that Flashlight provides that popular end-to-end testing tools such as detox cannot provide?

[hannojg]

While detox is an automation tool to run tests, flashlight is more like a tool that you can use to get performance metrics while an app is running.
In detox you describe how things get executed and what you expect the UI to look like, while flashlight only helps with getting metrics.
We will use flashlight in our existing e2e test suite to collect performance metrics and use these as indicators to catch performance regressions.

[hayata-suenaga]

That makes sense thank you!

[hayata-suenaga]

changing the priority of this issue to monthly

[hayata-suenaga]

https://www.youtube.com/watch?v=XO8O1iL4Rxg

looks promising 🤩

[hayata-suenaga]

lower priority now

[hannojg]

(Still on my todo list! Probably will start with it by next week)

[hayata-suenaga]

@hannojg how is this project going?

[hannojg]

Hey, sorry, this project is on HOLD as we are fixing the e2e pipeline at the minute. Project should start next week though once everything is fixed!

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.3-7 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [NoQA] e2e: new common metrics (FPS, CPU, RAM) #43482

If no regressions arise, payment will be issued on 2024-07-10. 🎊

For reference, here are some details about the assignees on this issue:

• @hannojg does not require payment (Contractor)

[melvin-bot]

BugZero Checklist: The PR adding this new feature has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@hannojg] Please propose regression test steps to ensure the new feature will work correctly on production in further releases.
• [] Link the GH issue for creating/updating the regression test once above steps have been agreed upon.

[melvin-bot]

Skipping the payment summary for this issue since all the assignees are employees or vendors. If this is incorrect, please manually add the payment summary SO.

[hayata-suenaga]

No payment is necessary for this issue I think

[melvin-bot]

@hannojg, @hayata-suenaga Eep! 4 days overdue now. Issues have feelings too...

[kirillzyusko]

@hayata-suenaga I think the issue can be closed?..