Kokoro Check build failures due to NodeJS or OpenJDK installation failure

[vorburger]

#2295 (related to #2294) failed the Kokoro Check build:

https://source.cloud.google.com/results/invocations/80bddf42-e6cc-4be7-862b-c625032e4abb/targets/openhealthstack%2Fandroid-fhir%2Fgcp_ubuntu%2Fpresubmit/log

[ID: 9480442] Executing command via SSH:
export KOKORO_BUILD_NUMBER="3818"
export KOKORO_JOB_NAME="openhealthstack/android-fhir/gcp_ubuntu/presubmit"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ;  chmod 755 github/android-fhir/kokoro/gcp_ubuntu/kokoro_build.sh ;  PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/android-fhir/kokoro/gcp_ubuntu/kokoro_build.sh   ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0

npm WARN using --force I sure hope you know what you are doing.
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
+ [email protected]
updated 1 package in 1.022s
  installing : node-v16.18.0
       mkdir : /usr/local/n/versions/node/16.18.0
       fetch : https://nodejs.org/dist/v16.18.0/node-v16.18.0-linux-x64.tar.xz
curl: (18) transfer closed with 8144384 bytes remaining to read
xz: (stdin): Unexpected end of input
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

  Error: failed to download archive for 16.18.0

@omarismail94 have you seen this problem before? Often?

I suspect the cause is that that nodejs.org is "throttling" DLs from GCP.

To avoid this, perhaps https://github.com/google/android-fhir/blob/master/kokoro/gcp_ubuntu/kokoro_build.sh could be tweaked to e.g. install NodeJS via Package Manager?

Or is it possible to do Caching on Kokoro (of e.g. the Node installation), like GitHub Actions VM can do?

[vorburger]

https://github.com/tj/n#custom-source would allow to overwrite URL.

But I couldn't find a more reliable NodeJS mirror e.g. on GCP.

Caching this installation is probably the way to go.

This is only worth persuing if it happens more regularly.

[vorburger]

This is only worth persuing if it happens more regularly.

It just happened again, on https://source.cloud.google.com/results/invocations/e7f4bf2c-7a74-460e-b769-e7f9c59bd01d/targets/openhealthstack%2Fandroid-fhir%2Fgcp_ubuntu%2Fpresubmit/log, for #2298.

[vorburger]

This is a similar problem, which I just debugged as the root cause for why #2282 has a "stuck" Kokoro: Build and Device Tests Expected — Waiting for status to be reported:

Err:4 [http://ppa.launchpad.net/openjdk-r/ppa/ubuntu](https://www.google.com/url?q=http://ppa.launchpad.net/openjdk-r/ppa/ubuntu&sa=D) xenial/main amd64 openjdk-17-jdk amd64 17~11-1~16.04~2
  Connection timed out [IP: 185.125.190.52 80]
Fetched 285 MB in 2min 13s (2,129 kB/s)
E: Failed to fetch [http://ppa.launchpad.net/openjdk-r/ppa/ubuntu/pool/main/o/openjdk-17/openjdk-17-jdk_17~11-1~16.04~2_amd64.deb](https://www.google.com/url?q=http://ppa.launchpad.net/openjdk-r/ppa/ubuntu/pool/main/o/openjdk-17/openjdk-17-jdk_17~11-1~16.04~2_amd64.deb&sa=D)  Connection timed out [IP: 185.125.190.52 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?


[ID: 2992586] Command finished after 140 secs, exit value: 100

[omarismail94]

We could cache it, or save the package in a GCS Bucket that can be accessed in the script, this is the script that runs:
https://github.com/google/android-fhir/blob/master/kokoro/gcp_ubuntu/kokoro_build.sh

[vorburger]

This (the NodeJS one) already happened AGAIN, on https://source.cloud.google.com/results/invocations/35a3c980-1b62-4a4a-88a2-daf5f0c76bfd/targets/openhealthstack%2Fandroid-fhir%2Fgcp_ubuntu%2Fpresubmit/log, for #2316 ... to me, somehow fixing this one way or another seems a fairly high priority for this project.

We could cache it, or save the package in a GCS Bucket that can be accessed in the script

Yeah. What bothers me a tiny little bit is having to cobble this together here - this kind of problem has already been solved, of course; see e.g. https://github.com/actions/cache or https://github.com/marketplace/actions/cache-apt-packages.

I'm curious if simplifying the CI infrastructure overall with #2304 wouldn't be a better way forward instead of fixing this...

But if you do happen to have bandwidth to fix this, please do!

[vorburger]

N/A iff #2304 - but keeping open until decision if we'll do that (because if we don't, this should still be fixed).