Rev Alpine WithNode from 3.9 to 3.13 #57324

am11 · 2021-08-12T22:17:23Z

Also,

update helix queues from 3.12 to 3.13
add 3.14 helix queues
update performance job to install cargo from main branch

ghost · 2021-08-12T22:17:28Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Updating to 3.14 now will buy us ~2 years before this latest version reaches EOL.
Upstream support for 3.14 was added: dotnet/dotnet-buildtools-prereqs-docker@d90babf

Fixes #56672

Author:	am11
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

am11 · 2021-08-12T22:17:52Z

cc @jkotas, @janvorli we can also update helix queues to test the latest Alpine Linux version, but I haven't changed that as part of this PR.

jkotas · 2021-08-12T22:25:36Z

According to https://github.com/dotnet/core/blob/main/release-notes/6.0/supported-os.md#linux, we want to support Alpine 3.13+ for .NET 6. Are binaries built on Alpine 3.14 going to run on Alpine 3.13?

am11 · 2021-08-12T22:28:09Z

Are binaries built on Alpine 3.14 going to run on Alpine 3.13?

They should work (as time64 changes were included in 3.13). I will test it.

am11 · 2021-08-12T23:52:32Z

Locally I was able to build the main branch without any changes:

docker run -v $(pwd):/runtime \
  -it mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf \
  /runtime/src/coreclr/build-runtime.sh checked x64

but on CI it was failing to find python while it is clearly there, but when i worked around it, it is now failing with Failed to get PGO data package path. There is nothing in logs pointing to clues what's wrong, so it might be something specific to pipeline YAML that needs updating?

hoyosjs · 2021-08-13T01:11:11Z

@am11 looks like the container doesn't have curl? (and maybe doing the volume mount masks this as you may have a .dotnet directory?

am11 · 2021-08-13T01:20:10Z

@am11 looks like the container doesn't have curl? (and maybe doing the volume mount masks this as you may have a .dotnet directory?

@hoyosjs, I tried it with clean git repo. Even without mounting, git clone <runtime url> followed by build-runtime.sh release x64 succeeds in that container.

hoyosjs · 2021-08-13T20:18:43Z

I tried with copied IDs from the build logs and I was still able to get the SDK.

am11 · 2021-08-14T02:51:51Z

Yes, the docker container mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf is the one failing CI leg is using which I tested earlier. It has all the required stuff and capable of building runtime. Something else in AzDo is apparently going wrong. Nothing jumping out at me from logs of Initialize containers step, all looks normal until Build CoreCLR Runtime step, which seems to be using a different environment.

ViktorHofer · 2021-08-16T15:16:02Z

@am11 please let us know if we can help anyhow.

am11 · 2021-08-16T15:45:15Z

@ViktorHofer, it is still a mystery to me as CI error is not reproducible locally. e.g. this leg https://dev.azure.com/dnceng/public/_build/results?buildId=1291489&view=logs&jobId=0e64859c-a870-5a74-6b6e-333fc1003298&j=0e64859c-a870-5a74-6b6e-333fc1003298&t=5da9eef6-c013-592b-ccae-2e46c52786d0 is apparently using container mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf and failing on build step due to missing curl. Although the said container has curl, python along rest of the dependencies.

I just (temporarily) deleted > /dev/null 2>&1 to reveal the missing curl error from

runtime/src/coreclr/build-runtime.sh

Lines 55 to 58 in 6acb72b

    
           "$__RepoRootDir/eng/common/msbuild.sh" /clp:nosummary $__ArcadeScriptArgs $OptDataProjectFilePath $RestoreArg /t:DumpPgoDataPackagePath \ 
        
                                               ${__CommonMSBuildArgs} /p:PgoDataPackagePathOutputFile=${PgoDataPackagePathOutputFile} \ 
        
                                               -bl:"$__LogsDir/PgoVersionRead_$__ConfigTriplet.binlog" > /dev/null 2>&1 
        
           local exit_code="$?"

If we test it locally with same command used by this failing CI leg:

docker run \
  -it mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf \
  sh -c 'git clone https://github.com/dotnet/runtime --single-branch --depth 1 /runtime &&
  /runtime/src/coreclr/build-runtime.sh checked x64'

it builds coreclr successfully. Do you see anything unusual from the logs that could be causing Build CoreCLR Runtime step to use different environment than the container?

ViktorHofer · 2021-08-16T16:01:34Z

I'll take a look. Let me also cc @safern and @jkoritzinsky

ViktorHofer · 2021-08-20T14:51:12Z

@hoyosjs @jkoritzinsky any other ideas? I'm currently a bit short on time, maybe you have some additional ideas?

ViktorHofer · 2021-09-01T16:06:49Z

@MattGal do you think you can help here?

MattGal · 2021-09-01T16:12:01Z

@MattGal do you think you can help here?

No guarantees but sure, I'll take a look

MattGal · 2021-09-01T17:08:07Z

@ViktorHofer I can't reproduce this running main build inside a mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf container, it may be a transient error?

hoyosjs · 2021-09-01T17:47:47Z

~~Sigh. Looks like transient - cache or something. Kicked off again and the containers initialized just fine...~~ spoke too soon. Once more failed with the same error as three weeks back.

MattGal · 2021-09-01T17:54:35Z

Super weird but does seem to be noticed and known externally: dotnet/install-scripts#206.

I'm running the same image you used to run the same CI build from a Windows box. I wonder if you started from an Ubuntu 18.04/20.04 if you could get a repro.

hoyosjs · 2021-09-01T18:15:02Z

I couldn't repro on a physical Ubuntu box with that container on top.

MattGal · 2021-09-01T18:22:02Z

I couldn't repro on a physical Ubuntu box with that container on top.

Maybe just try alpine-3.13-WithNode-20210812132854-ddfc481? since , even if this works after a bunch of retries you don't want to merge like this...

am11 · 2021-09-01T21:04:07Z

even if this works after a bunch of retries

Flakiness is really not the problem. On other systems, you can build using this new container reliably and in AzDO it is always failing to run build step inside the container (it is apparently running build step on host system rather than the container). Logs are not indicating any problem.

eng/pipelines/coreclr/templates/helix-queues-setup.yml

eng/pipelines/libraries/helix-queues-setup.yml

Also, * update helix queues from 3.12 to 3.13 * add 3.14 helix queues * update performance job to install cargo from main branch

safern · 2021-09-24T19:29:25Z

eng/pipelines/libraries/helix-queues-setup.yml

@@ -41,15 +41,11 @@ jobs:

    # Linux musl x64
    - ${{ if eq(parameters.platform, 'Linux_musl_x64') }}:
-      - ${{ if eq(parameters.jobParameters.isFullMatrix, false) }}:


can we test on 3.13 on fullMatrix == true? Meaning on rolling CI?

Yup, didn't mean to press delete on rolling ones 🤣

safern

Thanks 🎉

hoyosjs

Thanks a lot @am11

safern · 2021-09-24T22:18:37Z

It seems like you have a yaml syntax error 😢 : https://dev.azure.com/dnceng/public/_build/results?buildId=1384687&view=results

aik-jahoda · 2021-09-29T15:08:38Z

There is a JSON failure on Alpine.314.Arm64.Open. @safern do you think the failure is related to the change or we can merge this PR?

danmoseley · 2021-09-29T15:24:13Z

The JSON crash has a dump. We should look at that dump -- there is a how-to-debug-dump.md file there to help.

The regex issue is known and fixed already.

am11 · 2021-09-29T15:46:58Z

There is a JSON failure on Alpine.314.Arm64.Open

Isn't it #57198?

safern · 2021-09-29T18:53:18Z

I think we should at least look at the dump to make sure if it is: #57198 before merging something that could potentially introduce a failure to CI.

am11 · 2021-09-29T19:18:22Z

The dump link on that issue is expired and Eirik had no success with clrstack when link was active (#57198 (comment)). Not sure how I can determine it.

Maybe rerun the job to see if the issue persist?

safern · 2021-09-29T19:19:16Z

Maybe rerun the job to see if the issue persist?

Sounds good to me.

danmoseley · 2021-09-29T19:30:35Z

Dump links from #57324 are still active, they're only hours old.

Console Logs

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57324-merge-9e9d1cbb6f224c0492/System.Text.Json.Tests/1/how-to-debug-dump.md?sv=2019-07-07&se=2021-10-19T14%3A08%3A15Z&sr=c&sp=rl&sig=wKHlTG%2F2YMVkeicpVl0W%2FlyaZ4RfBE74SXPTyvUs6b0%3D

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57324-merge-9e9d1cbb6f224c0492/System.Text.Json.Tests/1/core.1001.23?sv=2019-07-07&se=2021-10-19T14%3A08%3A15Z&sr=c&sp=rl&sig=wKHlTG%2F2YMVkeicpVl0W%2FlyaZ4RfBE74SXPTyvUs6b0%3D

from
https://dev.azure.com/dnceng/public/_build/results?buildId=1385273&view=ms.vss-test-web.build-test-results-tab&runId=40500360&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=185830

I believe it requires a certain access token to pull part of what you need down so @am11 you maybe can't.

am11 · 2021-09-29T19:34:06Z

Dump links from #57324 are still active, they're only hours old.

Yup, I can download dump from this PR but the one from #57198 (which I would like to compare with) is expired.

danmoseley · 2021-09-29T19:50:13Z

Ah gotcha. Is there anything that can be gotten out of this dump?

am11 · 2021-09-29T19:51:52Z

I am working on it. 🙂

am11 · 2021-09-29T22:03:37Z

clrstack results are posted here: #57198 (comment). I have the container saved, in case more info is needed.

aik-jahoda · 2021-09-30T08:14:08Z

The regex issue is known and fixed already.

clrstack results are posted here: #57198 (comment). I have the container saved, in case more info is needed.

Sounds like both test failures are known and tracked/fixed already. Can we merge this PR?

dotnet-issue-labeler bot added the area-Infrastructure label Aug 12, 2021

ghost added the community-contribution Indicates that the PR has been added by a community member label Aug 12, 2021

am11 marked this pull request as draft August 12, 2021 22:28

ViktorHofer mentioned this pull request Aug 16, 2021

Alpine builds use alpine-3.9 docker image that is EOL #56672

Closed

ViktorHofer added this to the 6.0.0 milestone Aug 20, 2021

ViktorHofer mentioned this pull request Aug 23, 2021

Update unsupported distros in helix queues to latest and add missing ones #57947

Open

14 tasks

am11 force-pushed the feature/ci/alpine3.14 branch 2 times, most recently from 37bf805 to 2888eaf Compare September 2, 2021 00:53

safern reviewed Sep 24, 2021

View reviewed changes

eng/pipelines/coreclr/templates/helix-queues-setup.yml Outdated Show resolved Hide resolved

safern reviewed Sep 24, 2021

View reviewed changes

eng/pipelines/libraries/helix-queues-setup.yml Show resolved Hide resolved

Rev Alpine WithNode from 3.9 to 3.13

7e1fb3e

Also, * update helix queues from 3.12 to 3.13 * add 3.14 helix queues * update performance job to install cargo from main branch

am11 force-pushed the feature/ci/alpine3.14 branch from 174c287 to 7e1fb3e Compare September 24, 2021 19:23

safern reviewed Sep 24, 2021

View reviewed changes

Add Alpine 3.13 in rolling CI

f0fd764

safern approved these changes Sep 24, 2021

View reviewed changes

hoyosjs approved these changes Sep 24, 2021

View reviewed changes

Add missing closing paren

a1e698c

This was referenced Sep 29, 2021

Segfault in tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh on Ubuntu. #59758

Closed

System.Text.Json.Tests crashing with segfault in CI #57198

Closed

hoyosjs merged commit a50e1e6 into dotnet:main Sep 30, 2021

am11 deleted the feature/ci/alpine3.14 branch September 30, 2021 10:30

kunalspathak mentioned this pull request Oct 7, 2021

[Perf] Changes at 9/30/2021 9:36:23 AM dotnet/perf-autofiling-issues#1672

Closed

CarnaViire mentioned this pull request Oct 8, 2021

Test failures: System.Diagnostics.Tests.ProcessStartInfoTests on windows.11.amd64.clientpre #60191

Open

ghost locked as resolved and limited conversation to collaborators Nov 3, 2021

Rev Alpine WithNode from 3.9 to 3.13 #57324

Rev Alpine WithNode from 3.9 to 3.13 #57324

Conversation

am11 commented Aug 12, 2021 • edited Loading

ghost commented Aug 12, 2021

am11 commented Aug 12, 2021

jkotas commented Aug 12, 2021

am11 commented Aug 12, 2021

am11 commented Aug 12, 2021

hoyosjs commented Aug 13, 2021

am11 commented Aug 13, 2021

hoyosjs commented Aug 13, 2021

am11 commented Aug 14, 2021

ViktorHofer commented Aug 16, 2021

am11 commented Aug 16, 2021 • edited Loading

ViktorHofer commented Aug 16, 2021

ViktorHofer commented Aug 20, 2021

ViktorHofer commented Sep 1, 2021

MattGal commented Sep 1, 2021

MattGal commented Sep 1, 2021

hoyosjs commented Sep 1, 2021 • edited Loading

MattGal commented Sep 1, 2021

hoyosjs commented Sep 1, 2021

MattGal commented Sep 1, 2021

am11 commented Sep 1, 2021

safern Sep 24, 2021

Choose a reason for hiding this comment

am11 Sep 24, 2021 • edited Loading

Choose a reason for hiding this comment

safern left a comment

Choose a reason for hiding this comment

hoyosjs left a comment

Choose a reason for hiding this comment

safern commented Sep 24, 2021 • edited Loading

aik-jahoda commented Sep 29, 2021

danmoseley commented Sep 29, 2021

am11 commented Sep 29, 2021

safern commented Sep 29, 2021

am11 commented Sep 29, 2021

safern commented Sep 29, 2021

danmoseley commented Sep 29, 2021 • edited by safern Loading

am11 commented Sep 29, 2021

danmoseley commented Sep 29, 2021

am11 commented Sep 29, 2021

am11 commented Sep 29, 2021

aik-jahoda commented Sep 30, 2021

am11 commented Aug 12, 2021 •

edited

Loading

am11 commented Aug 16, 2021 •

edited

Loading

hoyosjs commented Sep 1, 2021 •

edited

Loading

am11 Sep 24, 2021 •

edited

Loading

safern commented Sep 24, 2021 •

edited

Loading

danmoseley commented Sep 29, 2021 •

edited by safern

Loading