[VMR] Build failure with portable RID (e.g. /p:TargetRid=linux-x64)

[directhex]

I'm still working on the exact root cause, but I wanted to have a written record somewhere other than internal Teams.

Building for a generic, portable RID fails currently with the VMR, in aspnetcore, citing that it can't find the crossgen2 binary. Looking at the build, and comparing to a "good" ubuntu.22.04-x64 build, it seems the crossgen2 nupkg is created as expected during the runtime build, but never restored in aspnetcore.

It appears the build explicitly rejects most of the package references from earlier in the build, in fact, citing that they do not need to be explicitly depended on because they're part of the SDK.

This also blocks building an osx-x64 RID build on Mac.

[directhex]

    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Runtime.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Runtime.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Runtime.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Runtime.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Crossgen2.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Crossgen2.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Crossgen2.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Crossgen2.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Runtime.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Runtime.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Runtime.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Runtime.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Crossgen2.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Crossgen2.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Crossgen2.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/.dotnet/sdk/7.0.100-rc.2.22477.23/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): error NETSDK1023: A PackageReference for 'Microsoft.NETCore.App.Crossgen2.linux-x64' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
    /home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj(434,5): error : Could not find crossgen2 tools/crossgen2

[directhex]

Adding DisableImplicitFrameworkReferences to Microsoft.AspNetCore.App.Runtime.csproj reduces the error count from 17 to 13, which is not very meaningful or useful :|

[omajid]

cc @tmds

[tmds]

There are some changes on aspnetcore 7.0 branch for eliminating the portable build.

The PRs against the installer repo haven't been backported yet (dotnet/installer#14549 and dotnet/installer#14938). Backporting those may fix this issue.

cc @crummel @MichaelSimons

[directhex]

Funny story, I think the 7.0 aspnetcore changes (specifically dotnet/aspnetcore#44978) are the problem, not the cure.

Dependencies.props includes a hardcoded dependency on Microsoft.NETCore.App.Runtime.linux-x64 et al, and also includes a dependency on Microsoft.NETCore.App.Runtime.$(TargetRuntimeIdentifier) which (when building on Mac, or on Linux with /p:TargetRid=linux-x64) resolves to a duplicate dependency. This causes the CheckForImplicitPackageReferenceOverrides task to freak out, remove both copies of the reference, and fail the build.

Locally, removing one of the duplicated dependencies progresses the build further. Without the duplicate dependency, DisableImplicitFrameworkReferences has no impact at all either enabled or disabled, so it turns out this was a bad area of investigation.

[directhex]

The aspnetcore build still fails, mind you, but later in the build:

      Unhandled exception. System.CommandLine.CommandLineException: Multiple input files matching same simple name /home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net7.0/linux-x64/System.Formats.Asn1.dll /home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/package-cache/microsoft.netcore.app.runtime.linux-x64/8.0.0-alpha.1.22559.2/runtimes/linux-x64/lib/net7.0/System.Formats.Asn1.dll
         at System.CommandLine.Helpers.AppendExpandedPaths(Dictionary`2 dictionary, String pattern, Boolean strict)
         at System.CommandLine.Helpers.BuildPathDictionary(IReadOnlyList`1 tokens, Boolean strict)
         at System.CommandLine.Argument`1.<>c__DisplayClass5_0.<.ctor>b__1(ArgumentResult argumentResult, Object& value)
         at System.CommandLine.Parsing.ArgumentResult.Convert(Argument argument)
         at System.CommandLine.Parsing.ArgumentResult.GetArgumentConversionResult()
         at System.CommandLine.Parsing.ParseResultVisitor.ValidateAndConvertArgumentResult(ArgumentResult argumentResult)
         at System.CommandLine.Parsing.ParseResultVisitor.ValidateAndConvertArgumentResults(IReadOnlyList`1 arguments, Int32 commandArgumentResultCount)
         at System.CommandLine.Parsing.ParseResultVisitor.Stop()
         at System.CommandLine.Parsing.Parser.Parse(IReadOnlyList`1 arguments, String rawInput)
         at ILCompiler.Program.Main(String[] args)
      Aborted (core dumped)
    /home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj(496,5): error MSB3073: The command ""/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/package-cache/microsoft.netcore.app.crossgen2.linux-x64/8.0.0-alpha.1.22559.2/tools/crossgen2" --composite --targetarch:x64 --targetos:linux /home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/package-cache/microsoft.netcore.app.runtime.linux-x64/8.0.0-alpha.1.22559.2/runtimes/linux-x64/lib/net7.0/*.dll /home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net7.0/linux-x64/*.dll --out:"/home/directhex/Projects/dotnet/src/aspnetcore/artifacts/source-build/self/src/artifacts/obj/Composite/Release/full-composite.r2r.dll"" exited with code 134.

[directhex]

Well that's fascinating. In my non-portable build, those three assemblies exist only in microsoft.netcore.app.runtime.ubuntu.22.04-x64 but in the portable build, they're in both microsoft.netcore.app.runtime.linux-x64 and artifacts/bin/Microsoft.AspNetCore.App.Runtime

[directhex]

OK, no, the 2-assembly thing is a weird problem with a dirty build. Doing a 100% clean build, this is enough to make the build work:

index 6c44830a0..904a23489 100644
--- a/src/aspnetcore/eng/Dependencies.props
+++ b/src/aspnetcore/eng/Dependencies.props
@@ -100,7 +100,6 @@ and are generated based on the last package release.
     <LatestPackageReference Include="Microsoft.NETCore.App.Runtime.win-arm64" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Runtime.osx-x64" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Runtime.osx-arm64" />
-    <LatestPackageReference Include="Microsoft.NETCore.App.Runtime.linux-x64" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Runtime.linux-arm" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Runtime.linux-arm64" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Runtime.linux-musl-x64" />
@@ -117,7 +116,6 @@ and are generated based on the last package release.
     <LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.linux-musl-arm" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.linux-musl-arm64" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.linux-musl-x64" />
-    <LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.linux-x64" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.win-x64" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.win-x86" />
     <LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.win-arm" />

[directhex]

Got a Mac build. I'll post patches & file issues tomorrow morning, since changes are needed.