Build failing on tip : "./build.sh --subsetCategory CoreCLR -c Release -arch x64"

[sdmaclea]

  error: Could not read profile /home/stmaclea/.nuget/packages/optimization.linux-x64.pgo.coreclr/99.99.99-master-20200228.3/data/clrjit.profdata: Unsupported instrumentation profile format version
  [ 75%] Building CXX object src/utilcode/CMakeFiles/utilcode_dac.dir/configuration.cpp.o
  [ 76%] Building CXX object src/utilcode/CMakeFiles/utilcode_dac.dir/collections.cpp.o
  1 error generated.
  make[2]: *** [src/jit/standalone/CMakeFiles/clrjit.dir/__/alloc.cpp.o] Error 1
  make[1]: *** [src/jit/standalone/CMakeFiles/clrjit.dir/all] Error 2
  src/jit/standalone/CMakeFiles/clrjit.dir/build.make:79: recipe for target 'src/jit/standalone/CMakeFiles/clrjit.dir/__/alloc.cpp.o' failed
  CMakeFiles/Makefile2:2423: recipe for target 'src/jit/standalone/CMakeFiles/clrjit.dir/all' failed
  make[1]: *** Waiting for unfinished jobs....

[janvorli]

I have seen this problem in the past and in my case, doing a clean build (git clean -xdf executed in the root of the repo and then rebuild of everything) fixed the problem. Maybe I had to delete the .nuget folder too.

[sdmaclea]

I did try ./build.sh --clean which didn't work resolve the issue.

Perhaps --clean also needs to be fixed.

[hoyosjs]

Clean hasn't worked in a while or been supported I believe, at least in coreclr. I am not sure why it wasn't removed.

[ViktorHofer]

FWIW clean is working fine but it just cleans the artifacts folder.

[ViktorHofer]

I'm talking about .\build.cmd/sh -clean: https://github.com/dotnet/runtime/blob/master/eng/common/build.ps1#L123-L129

[hoyosjs]

The old coreclr clean used to do a bit more, like cleaning the package cache as it used to be directory based. we decided to defer the logic to nuget/dotnet nuget

[sdmaclea]

It seems strange that we are using caches outside of the build directory i.e.~/.nuget. It seems to make the build less robust. Perhaps CI doesn't do this?

[ViktorHofer]

The old coreclr clean used to do a bit more, like cleaning the package cache as it used to be directory based.

Clearing the nuget cache should be done via either dotnet nuget or nuget. There's no need to replicate that logic in dotnet/runtime: dotnet nuget locals -c global-packages. Do you have specific suggestion how to improve the current -clean target?

Perhaps CI doesn't do this?

Right, CI doesn't use the user wide cache but instead puts all packages under runtime/.packages.

[hoyosjs]

Indeed, ci generates a local .packages folder. Ah, Viktor got to it first :)

Do you have specific suggestion how to improve the current -clean target?

I honestly believe that what we are doing is the right thing. In this case it even feels like deleting the cached version of the optimization package is enough.

[sdmaclea]

I honestly believe that what we are doing is the right thing. In this case it even feels like deleting the cached version of the optimization package is enough.

Why?

I see several things we are doing wrong:

• The build is not reproducible.
• The build depending on global machine state.
• The dependency is undocumented.
• The --clean operation doesn't clean everything and there is no --deep-clean option
• The dev build is different from the CI build, The dev experience is effectively untested by CI.

[sdmaclea]

I retried the build like this and it still failed.

git clean -xdf
rm -rf ~/.nuget
./build.sh --subsetCategory CoreCLR -c Release -arch x64

With the same error.

[jashook]

Did we ever get to the root cause?

[sdmaclea]

No I never figured out root cause. I just reverted to an older previously working commit and forgot about it.

When I later updated to tip, the issue was resolved.

I don't think I could repro this anymore... Tempted to close because it cannot be reproed.

[ViktorHofer]

Ok let's close this then. Feel free to reopen.

[trylek]

I'm hitting this to this day when trying to build CoreCLR on Ubuntu in release mode; last time I hit this literally 2 minutes back. I'll be happy to assist any follow-up investigation by collecting artifacts, logs or anything else. @davidwrighton / @janvorli, have you got any suggestions as to how to continue the investigation? Is there an infra problem like pulling down a too old pgo data package or is it a compiler problem, what component consumes the optimization data on Linux, do we need to install a newer clang version, cmake or something else? Does there exist any tooling that would let me dump the pgo data and share its properties?

Thanks a lot

Tomas

[trylek]

  [ 42%] Building CXX object jit/CMakeFiles/clrjit.dir/alloc.cpp.o
  error: Could not read profile /home/trylek/.nuget/packages/optimization.linux-x64.pgo.coreclr/1.0.0-prerelease.21451.4/data/coreclr.profdata: Unsupported instrumentation profile format version

[akoeplinger]

I'm hitting this in #61001 as well, had to workaround by passing /p:NativeOptimizationDataSupported=false to build.sh

[davidwrighton]

I'll take a look at fixing the code flow here

[davidwrighton]

But my guess is that you have a version of clang that isn't able to consume the profile data we produce. Are you building with the same version of clang that is used in the offical docker build images?

[akoeplinger]

In #61001 I was updateing from Ubuntu 16.04 to 18.04 so it's possible that there is a different clang version being used.

[lambdageek]

Hitting this on Debian trixie/sid while trying to build release/9.0-rc2 with ./build.sh clr+libs -c Release :

...
  error: Error in reading profile /home/aleksey/.nuget/packages/optimization.linux-x64.pgo.coreclr/1.0.0-prerelease.24409.2/data/coreclr.profdata: unsupported instrumentation profile format version
...
$ clang --version
Debian clang version 16.0.6 (27+b1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

[am11]

$ clang --version

the relevant version shows up in cmake config logs. Do you see 16 in those logs too?

[krwq]

Started hitting this suddenly after rebasing. Current commit: 62bd39a and around 2 weeks ago when it was still working: 97a8af9.

Building with: ./build.sh -rc Release -s clr+libs on Ubuntu

• git clean -fdx + clearing ~/.nuget - didn't help

$ clang --version
Ubuntu clang version 18.1.3 (1ubuntu1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Relevant piece of build log:

  [ 50%] Building CXX object jit/CMakeFiles/clrjit_universal_arm64_x64.dir/cmake_pch.hxx.pch
  [ 50%] Building CXX object jit/CMakeFiles/clrjit_win_x64_x64.dir/cmake_pch.hxx.pch
  error: Error in reading profile /home/krwq/.nuget/packages/optimization.linux-x64.pgo.coreclr/1.0.0-prerelease.25171.2/data/coreclr.profdata: unsupported inst                                                                                                                                                                                                rumentation profile format version
  make[2]: *** [jit/CMakeFiles/clrjit.dir/build.make:77: jit/CMakeFiles/clrjit.dir/cmake_pch.hxx.pch] Error 1
  make[1]: *** [CMakeFiles/Makefile2:4338: jit/CMakeFiles/clrjit.dir/all] Error 2

[akoeplinger]

might be related to the updated packages from f9fc62a ?

[akoeplinger]

I see that the images were updated to use the mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-net10.0-opt-amd64 container in https://dev.azure.com/dnceng/internal/_git/dotnet-optimization/pullrequest/47824, which uses clang 20 now since dotnet/dotnet-buildtools-prereqs-docker#1369.

Presumably that creates a newer version of the profile format?

[am11]

The workaround is to just rename profdata mv /home/krwq/.nuget/packages/optimization.linux-x64.pgo.coreclr/1.0.0-prerelease.25171.2/data/coreclr.profdata{,_} for local development. We should probably add an explicit option to disable pgo_support in build.sh, as it's a supported scenario in inner builds

runtime/src/coreclr/pgosupport.cmake

Lines 41 to 43 in 6a23f1c

	# If we don't have profile data available, gracefully fall back to a non-PGO opt build
	if(NOT EXISTS ${ProfilePath})
	message("PGO data file NOT found: ${ProfilePath}")

[akoeplinger]

you can already pass /p:NativeOptimizationDataSupported=false as a workaround. we probably need to add some detection to the build scripts so it ignores the data if it's incompatible

[krwq]

would upgrading clang to 20 fix this locally? (I already do the workaround but would prefer permanent fix)

[am11]

would upgrading clang to 20 fix this locally? (I already do the workaround but would prefer permanent fix)

Yup, it would match the CI environment which appears to be working. On Ubuntu/Debian, one quick way to upgrade llvm toolchian is curl -sSL https://apt.llvm.org/llvm.sh | bash -s - 20 all. build.sh has mechanism to pick the latest clang, if multiple installations are found (just delete artifacts/ dir after installing the new toolchain).

[krwq]

In that case would it be reasonable to just early exit the build with error that clang-20 is required and update build instructions? I think this will be more reasonable than error most of the people don't understand. Or does it need to match specific version the CI is using? (either way I think it's more reasonable to require specific version of something than random error)

[akoeplinger]

Looks like we already have a check to skip PGO if clang is older than 16, that needs to be bumped:

runtime/src/coreclr/pgosupport.cmake

Line 55 in 6a23f1c

if((CMAKE_CXX_COMPILER_ID MATCHES "Clang") AND (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 16))

Would be nice if the PGO package could record the clang version so we don't need to manually keep it in sync.

[akoeplinger]

@LoopedBard3 @caaavik-msft how difficult would adding the clang version be?

[LoopedBard3]

We are not aware of any way to add the clang version to the PGO package, but we are also not very familiar using packages in this way. If there is a good way to do this, we don't see having any problem with doing it. Otherwise, if there is a different recommended way to flow this kind of information, we can likely do that as well.

[reflectronic]

The codespaces build is failing because of this https://github.com/dotnet/runtime/actions/runs/14099208201/job/39492189942