Test fails 50% of time becuase Apk installation failures for test suites that used test sharding with 8 emulators at once

[KyleLearnedThis]

I am running 8 Android stock emulators for test sharding at once via Composer. It often fails due to Apk installation failure on 1 of emulators. This happened like 50% of time and it's random. With this APK installation error, the entire test just abort.

So far I have tried:

1. Uninstall APKs via scripts
2. Restart all emulators via scripts before running the test.
3. restart adb server before test run
4. reduce emulators to run at once (8 seemed to be reasonable as I look into system process monitor)

Environment:

• MacPro OS X 10.13.2
• 8 Android Stock Emulators API level 26
• Java 1.8
• Composor composer-0.2.9.jar

Exception in thread "main" java.lang.IllegalStateException: Process [script, -F, /Users/jenkins/workspace/FWAutomation/AndroidGenericSuite03-Composer-Run/80356821957255.output, /Users/jenkins/Library/Android/sdk/platform-tools/adb, -s, emulator-5556, install, -r, /Users/jenkins/catanandroid/Android/build/outputs/apk/worldFastProd/debug/Android-world-fast-prod-debug.apk] exited with non-zero code 1 Script started on Wed Jan 17 11:30:25 2018 command: /Users/jenkins/Library/Android/sdk/platform-tools/adb -s emulator-5556 install -r /Users/jenkins/catanandroid/Android/build/outputs/apk/worldFastProd/debug/Android-world-fast-prod-debug.apk adb: connect error for write: protocol fault (couldn't read status): Connection reset by peer

[KyleLearnedThis]

catan.log Log attached.

[KyleLearnedThis]

Also tried this test suite on test sharding on 6 real phones, also see this happen, but with lower probability (1 in 17 runs).

[artem-zinnatullin]

adb: connect error for write: protocol fault (couldn't read status): Connection reset by peer

Indicates that something is wrong with adb.

Can you please post output of adb version from machine you're running tests on?

Also if you don't use any sort of isolation between test runs (like Docker) adb can become flaky over time, you can try force restarting adb server before running tests with Composer: adb kill-server && adb start-server

[KyleLearnedThis]

@artem-zinnatullin that is what I meant by "restart adb server before test run" on every start of the test (adb kill-server; sleep 5; adb start-server)

I don't have docker in my MacPro.

Android Debug Bridge version 1.0.39 Version 0.0.1-4500957 Installed as /Users/jenkins/Library/Android/sdk/platform-tools/adb

[artem-zinnatullin]

I see, hm hm. We don't use macOS as CI machines, so we have no data on its stability in long term. However 9/10 developers use macOS for development and some of them run tests with Composer on macOS and haven't seen this problem

Can you make sure you have latest Intel HAXM installed, and maybe try to switch between x86 and x86-64 images of emulators?

[artem-zinnatullin]

Also I'd like to point that even on Linux we've found that running ~more emulators than you have cores available results in unstable behavior of one or two emulators, so it might be that you need to reduce number of emulators even more.

This has nothing to do with Composer btw, Composer just talks to adb as you would and automates the process

[KyleLearnedThis]

My HAXM installed on that MacPro is 6.2.1. I was running x86 emulators, let me try x86-64. I monitored CPU usage and Memory usage, when test runs, I don't see much of spike, so I got an impression I can run more. But I think 6 - 8 should be okay based on my findings for this MacPro.

I also have suspicion that it's not Composer but adb itself is flaky so can't handle 8 concurrent adb connections at once. But I can't reproduce this on real phones easily.

[artem-zinnatullin]

Yep, Composer just talks to adb for communication with devices/emulators

…

On Fri, Jan 19, 2018, 11:12 AM KyleLearnedThis ***@***.***> wrote: My HAXM installed on that MacPro is 6.2.1. I was running x86 emulators, let me try x86-64. I monitored CPU usage and Memory usage, when test runs, I don't see much of spike that is worry some, so I got an impression I can run more. I also have suspicion that is not Composer but adb itself is flaky so can't handle 8 concurrent adb connections at once. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#120 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7B3JYqVKbDv9T6scHzvh9Q0k2u9BXYks5tMOkVgaJpZM4Rh4zZ> .