Ubuntu 24.04 Noble Numbat Memory Issue

[akoken]

I wanted to try the new Ubuntu 24.04 Noble Numbat image and updated the base image in our CI pipeline to mcr.microsoft.com/dotnet/sdk:8.0-noble. However, it results in an out-of-memory exception. Based on my investigation, the exception occurs specifically when the update-ca-certificates command is executed. It works fine with Jammy, by the way. It also works in Docker Desktop. The problem likely lies with the Noble base image rather than the .NET image. I'm not sure what changes were made between Jammy and Noble, but it appears that updating certificates consumes a lot of memory in Noble. I adjusted some memory settings in our GitLab runner, but it didn't resolve the issue. Is there anyone having this issue?

Noble:

Running with gitlab-runner 16.9.1 (782c6ecb)
  on gitlab-runner 8tQaTuVTa, system ID: s_4b76d26cbd0a
Preparing the "docker" executor
00:01
Using Docker executor with image mcr.microsoft.com/dotnet/sdk:8.0-noble ...
Pulling docker image mcr.microsoft.com/dotnet/sdk:8.0-noble ...
Using docker image sha256:12f966ba0dd3e4c602377c900eb2e78f69294ac7c86d66472baf221a4625eb60 for mcr.microsoft.com/dotnet/sdk:8.0-noble with digest mcr.microsoft.com/dotnet/sdk@sha256:0f814b92f1cc906de4e35234a8d59f349ca8b902ffa8d22d09dcad3010f5b604 ...
Preparing environment
00:00
Running on runner-8tqatuvta-project-24-concurrent-0 via my-server...
Getting source from Git repository
00:01
Fetching changes with git depth set to 1...
Reinitialized existing Git repository in /builds/common/my-project/.git/
Checking out 49fed5c6 as detached HEAD (ref is wip-noble-test)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:12
Using docker image sha256:12f966ba0dd3e4c602377c900eb2e78f69294ac7c86d66472baf221a4625eb60 for mcr.microsoft.com/dotnet/sdk:8.0-noble with digest mcr.microsoft.com/dotnet/sdk@sha256:0f814b92f1cc906de4e35234a8d59f349ca8b902ffa8d22d09dcad3010f5b604 ...
$ free -h # collapsed multi-line command
               total        used        free      shared  buff/cache   available
Mem:            15Gi       1.1Gi        13Gi       896Ki       1.3Gi        14Gi
Swap:          6.0Gi          0B       6.0Gi
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
out of memory
out of memory
......
out of memory
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
out of memory
out of memory
.......
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

Jammy:

Running with gitlab-runner 16.9.1 (782c6ecb)
  on gitlab-runner 8tQaTuVTa, system ID: s_4b76d26cbd0a
Preparing the "docker" executor
00:01
Using Docker executor with image mcr.microsoft.com/dotnet/sdk:8.0.204-jammy-amd64 ...
Pulling docker image mcr.microsoft.com/dotnet/sdk:8.0.204-jammy-amd64 ...
Using docker image sha256:f386a307b38f4d5c3f2d110b6c514a23f806fed0ff5f9316942da69caf021851 for mcr.microsoft.com/dotnet/sdk:8.0.204-jammy-amd64 with digest mcr.microsoft.com/dotnet/sdk@sha256:62df6dbe5d714512467cc4b91e099a7b2babaeff6eb6e6998143f465826db4ff ...
Preparing environment
00:00
Running on runner-8tqatuvta-project-24-concurrent-0 via my-server...
Getting source from Git repository
00:02
Fetching changes with git depth set to 1...
Reinitialized existing Git repository in /builds/common/my-project/.git/
Checking out 2473e0ea as detached HEAD (ref is develop)...
Removing .sonarqube/
Removing coverage.xml
Removing packages/
Removing unit-test-results.trx
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:38
Using docker image sha256:f386a307b38f4d5c3f2d110b6c514a23f806fed0ff5f9316942da69caf021851 for mcr.microsoft.com/dotnet/sdk:8.0.204-jammy-amd64 with digest mcr.microsoft.com/dotnet/sdk@sha256:62df6dbe5d714512467cc4b91e099a7b2babaeff6eb6e6998143f465826db4ff ...
$ free -h # collapsed multi-line command
               total        used        free      shared  buff/cache   available
Mem:            15Gi       855Mi        12Gi       0.0Ki       2.0Gi        14Gi
Swap:          6.0Gi          0B       6.0Gi
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
139 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       835Mi        12Gi       0.0Ki       2.0Gi        14Gi
Swap:          6.0Gi          0B       6.0Gi
$ export DOTNET_ROLL_FORWARD=Major
$ export PATH="$PATH:~/.dotnet/tools"
$ dotnet --version
8.0.204
$ dotnet restore -s $NEXUS_REPOSITORY --no-cache
  Determining projects to restore...
  Restored /builds/common/my-project.UnitTest.csproj (in 15.13 sec).
  Restored /builds/common/my-project.csproj (in 15.13 sec).
$ dotnet build --no-restore /p:RunAnalyzersDuringBuild=true
MSBuild version 17.9.8+b34f75857 for .NET
<redacted>
Build succeeded.
<redacted>
    6 Warning(s)
    0 Error(s)
Time Elapsed 00:00:11.67
Cleaning up project directory and file based variables
00:00
Job succeeded

[mthalman]

Does this repro when using the ubuntu:noble image directly? If so, you're likely to get better traction on resolving this issue by logging a bug at https://help.ubuntu.com/community/ReportingBugs.

[richlander]

Assuming an Ubuntu bug is found and filed, we can send a link to our Canonical contacts.

[akoken]

It's definitely an issue with Ubuntu Noble. According to the bug report link you provided, some tools like ubuntu-bug need to be installed to create dumps, etc. However, my host machine, where the GitLab runner is running, is RHEL. The GitLab runner uses Docker under the hood for pipelines. I need to attach to the container and run those ubuntu-bug commands before the container process gets killed. I wish there were an easy way to file an issue like on GitHub. 😞

Ubuntu Noble:

Running with gitlab-runner 16.9.1 (782c6ecb)
  on gitlab-runner 8tQaTuVTa, system ID: s_4b76d26cbd0a
Preparing the "docker" executor
00:01
Using Docker executor with image ubuntu:noble ...
Pulling docker image ubuntu:noble ...
Using docker image sha256:bf3dc08bfed031182827888bb15977e316ad797ee2ccb63b4c7a57fdfe7eb31d for ubuntu:noble with digest ubuntu@sha256:3f85b7caad41a95462cf5b787d8a04604c8262cdcdf9a472b8c52ef83375fe15 ...
Preparing environment
00:01
Running on runner-8tqatuvta-project-24-concurrent-0 via my-server...
Getting source from Git repository
00:00
Fetching changes with git depth set to 1...
Reinitialized existing Git repository in /builds/common/my-project/.git/
Checking out 49fed5c6 as detached HEAD (ref is wip-noble-test)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:11
Using docker image sha256:bf3dc08bfed031182827888bb15977e316ad797ee2ccb63b4c7a57fdfe7eb31d for ubuntu:noble with digest ubuntu@sha256:3f85b7caad41a95462cf5b787d8a04604c8262cdcdf9a472b8c52ef83375fe15 ...
$ free -h # collapsed multi-line command
               total        used        free      shared  buff/cache   available
Mem:            15Gi       1.1Gi        12Gi       892Ki       2.5Gi        14Gi
Swap:          6.0Gi          0B       6.0Gi
Get:1 http://security.ubuntu.com/ubuntu noble-security InRelease [89.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu noble InRelease [256 kB]
Get:3 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [18.6 kB]
Get:4 http://archive.ubuntu.com/ubuntu noble-updates InRelease [89.7 kB]
Get:5 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [37.2 kB]
Get:6 http://archive.ubuntu.com/ubuntu noble-backports InRelease [89.7 kB]
Get:7 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 Packages [331 kB]
Get:8 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages [1808 kB]
Get:9 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages [19.3 MB]
Get:10 http://archive.ubuntu.com/ubuntu noble/restricted amd64 Packages [117 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [40.8 kB]
Get:12 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [91.3 kB]
Get:13 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Packages [6387 B]
Fetched 22.3 MB in 6s (3846 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libssl3t64 openssl
The following NEW packages will be installed:
  ca-certificates openssl
The following packages will be upgraded:
  libssl3t64
1 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 3102 kB of archives.
After this operation, 2326 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libssl3t64 amd64 3.0.13-0ubuntu3.1 [1939 kB]
Get:2 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 openssl amd64 3.0.13-0ubuntu3.1 [1003 kB]
Get:3 http://archive.ubuntu.com/ubuntu noble/main amd64 ca-certificates all 20240203 [159 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 3102 kB in 1s (5320 kB/s)
(Reading database ... 4368 files and directories currently installed.)
Preparing to unpack .../libssl3t64_3.0.13-0ubuntu3.1_amd64.deb ...
Unpacking libssl3t64:amd64 (3.0.13-0ubuntu3.1) over (3.0.13-0ubuntu3) ...
Setting up libssl3t64:amd64 (3.0.13-0ubuntu3.1) ...
Selecting previously unselected package openssl.
(Reading database ... 4368 files and directories currently installed.)
Preparing to unpack .../openssl_3.0.13-0ubuntu3.1_amd64.deb ...
Unpacking openssl (3.0.13-0ubuntu3.1) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../ca-certificates_20240203_all.deb ...
Unpacking ca-certificates (20240203) ...
Setting up openssl (3.0.13-0ubuntu3.1) ...
Setting up ca-certificates (20240203) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC entries checked: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.38.2 /usr/local/share/perl/5.38.2 /usr/lib/x86_64-linux-gnu/perl5/5.38 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.38 /usr/share/perl/5.38 /usr/local/lib/site_perl) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 8.)
debconf: falling back to frontend: Teletype
Updating certificates in /etc/ssl/certs...
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
......
dpkg: error processing package ca-certificates (--configure):
 installed ca-certificates package post-installation script subprocess returned error exit status 146
Processing triggers for libc-bin (2.39-0ubuntu8.1) ...
Errors were encountered while processing:
 ca-certificates
E: Sub-process /usr/bin/dpkg returned an error code (1)
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

[richlander]

You could file the issue at https://launchpad.net/ubuntu/+source/ca-certificates

[akoken]

Created an issue for the ca-certificates package. Thanks @richlander!