Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update LTTNG installation to workaround bug #4149

Open
LoopedBard3 opened this issue Apr 15, 2024 · 3 comments · Fixed by dotnet/runtime#101142
Open

Update LTTNG installation to workaround bug #4149

LoopedBard3 opened this issue Apr 15, 2024 · 3 comments · Fixed by dotnet/runtime#101142
Labels
bug Something isn't working reenable Issue that specifies something that was disabled and should eventually be reenabled

Comments

@LoopedBard3
Copy link
Member

LoopedBard3 commented Apr 15, 2024

The following bug caused us to remove the lttng-modules-dkms package for every Linux microbenchmark run: https://bugs.launchpad.net/ubuntu/+source/lttng-modules/+bug/2043004, thanks @caaavik-msft. This issue tracks updating how we install the lttng-modules to workaround this problem longterm. It seems that manually installing latest lttng rather than from the Ubuntu package repository may fix the issue.

Here is a sample of the original error:

+ sudo apt-get -y install python3-pip
Reading package lists...
Building dependency tree...
Reading state information...
python3-pip is already the newest version (22.0.2+dfsg-1ubuntu0.4).
The following packages were automatically installed and are no longer required:
  apport-symptoms python3-systemd
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up lttng-modules-dkms (2.13.8-1~ubuntu22.04.0) ...
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
Removing old lttng-modules-2.13.8 DKMS files...
Deleting module lttng-modules-2.13.8 completely from the DKMS tree.
Loading new lttng-modules-2.13.8 DKMS files...
Building for 6.5.0-27-generic
Building initial module for 6.5.0-27-generic
Error! Bad return status for module build on kernel: 6.5.0-27-generic (x86_64)
Consult /var/lib/dkms/lttng-modules/2.13.8/build/make.log for more information.
dpkg: error processing package lttng-modules-dkms (--configure):
 installed lttng-modules-dkms package post-installation script subprocess returned error exit status 10
Errors were encountered while processing:
 lttng-modules-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)
+ export PERF_PREREQS_INSTALL_FAILED=1

Looking at job 9e113869-98a8-4c88-893c-9433b5a33072, the only two machines that seem to be affected are PERFTIGER138 and 139.

Digging into the Azure Data Explorer, the failure rate of those two machines is >97% while all others are much lower. I messaged to get these machines taken offline for investigation. Looking at the daily failure rates, it seems that the failure started happening consistently on the 12. We will want to rerun the runs between the 12 and 15th.

@LoopedBard3 LoopedBard3 added bug Something isn't working pipeline blocker issue blocks perf pipeline stays green. This be considered high priority. labels Apr 15, 2024
@LoopedBard3
Copy link
Member Author

Machines have been removed: dotnet/dnceng#2596

@LoopedBard3
Copy link
Member Author

LoopedBard3 commented Apr 16, 2024

Seems other machines may now be hitting the same issue.

@LoopedBard3 LoopedBard3 self-assigned this Apr 16, 2024
@LoopedBard3 LoopedBard3 removed their assignment Apr 16, 2024
@LoopedBard3 LoopedBard3 changed the title Linux PerfTiger machines failing during machine setup Update LTTNG installation to workaround bug Apr 16, 2024
@LoopedBard3
Copy link
Member Author

Updating this issue to track updating how we install the lttng version to get around the bug.

@LoopedBard3 LoopedBard3 reopened this Apr 16, 2024
@LoopedBard3 LoopedBard3 added reenable Issue that specifies something that was disabled and should eventually be reenabled and removed pipeline blocker issue blocks perf pipeline stays green. This be considered high priority. labels Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working reenable Issue that specifies something that was disabled and should eventually be reenabled
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant