Skip to content

Commit

Permalink
[PROF-9368] Revert Ruby 2.1/2.2/2.3 CI images to older Debian to supp…
Browse files Browse the repository at this point in the history 
…ort mysql gem

**What does this PR do?**

This PR is a slightly tweaked revert of
#3534 .

TL;DR after updating our Ruby 2.1/2.2/2.3 images to a newer debian,
we discovered that some of our CI testing requires the `mysql` gem,
which breaks on newer Debian versions (due to a newer
libmysqlclient/libmariadb).

With these reverts, I was able to locally rebuild the images and install
both mysql and nokogiri:

```
$ docker run -ti -v `pwd`:/working ivoanjo/docker-library:ddtrace_rb_2_3_8_testing_t6  /bin/bash
root@255fe2dc2374:/app# ruby -v
ruby 2.3.8p459 (2018-10-18 revision 65136) [x86_64-linux]
root@255fe2dc2374:/app# docker-compose -v
docker-compose version 1.8.0, build unknown
root@255fe2dc2374:/app# bundle -v
Bundler version 1.17.3
root@255fe2dc2374:/app# gem install mysql
Fetching mysql-2.9.1.gem
Building native extensions. This could take a while...
Successfully installed mysql-2.9.1
1 gem installed
root@255fe2dc2374:/app# gem install nokogiri -v '= 1.10.10'
Fetching nokogiri-1.10.10.gem
Fetching mini_portile2-2.4.0.gem
Successfully installed mini_portile2-2.4.0
Building native extensions. This could take a while...
Successfully installed nokogiri-1.10.10
2 gems installed
```

I was able to reproduce similar successful output on 2.1 and 2.2
as well (using nokogiri 1.9.1; 1.10.10 requires 2.3).

**Motivation:**

**Additional Notes:**

In the `Dockerfile` that #3534 initially changed there was a comment
stating that we were installing `docker-compose 1.21` from
stretch-backports, which confused me initially, since I couldn't
install that version from backports.

I checked an image build log from 2 months ago
(https://github.com/DataDog/dd-trace-rb/actions/runs/7386478002/job/20093153619)
and copied its image sha
`ghcr.io/datadog/dd-trace-rb/ruby:2.3.8-dd@sha256:b940776f41a5993e728f813c92734eaa82070ba74626ad8d2768a0242de269b3`
and got the same `docker-compose 1.8.0` I'm getting now:

```
$ docker run -ti -v `pwd`:/working ghcr.io/datadog/dd-trace-rb/ruby:2.3.8-dd@sha256:b940776f41a5993e728f813c92734eaa82070ba74626ad8d2768a0242de269b3 /bin/bash
root@23481c358c5b:/app# docker-compose -v
docker-compose version 1.8.0, build unknown
```

Unless I'm doing something wrong, either this comment was
never correct OR somehow the package we were using got purged.

Either way, it seems to me that this indicates we can still
use this version, so let's move forward (and hope to really
stop supporting these Rubies soon).

**How to test the change?**

Check CI builds for the images and/or build locally. Note that
our overall CI is a bit undisposed with incorrect images, so I'm
expecting red unrelated CI jobs that will only be fixed after we
publish these new images.
  • Loading branch information
@ivoanjo
ivoanjo committed Mar 20, 2024
1 parent cab98b1 commit 2119ba1
Show file tree
Hide file tree
Showing 3 changed files with 106 additions and 135 deletions.
44 changes: 30 additions & 14 deletions .circleci/images/primary/Dockerfile-2.1.10
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,13 @@

# Note: See the "Publishing updates to images" note in ./README.md for how to publish new builds of this container image

# Here be dragons: Last time we tried to move away from Debian stretch, we ran into issues with two gems that are used in some
# of our CI tests/test apps.
# If you're going to try your luck, make sure that the following work after your changes:
# gem install mysql
#
# taken from https://github.com/docker-library/ruby/blob/c5693b25aa865489fee130e572a3f11bccebd21b/2.1/Dockerfile
FROM buildpack-deps:bookworm AS ruby-2.1.10-bookworm
FROM buildpack-deps:stretch AS ruby-2.1.10-stretch

# skip installing gem documentation
RUN mkdir -p /usr/local/etc \
Expand All @@ -19,10 +24,12 @@ ENV RUBY_DOWNLOAD_SHA256 5be9f8d5d29d252cd7f969ab7550e31bbb001feb4a83532301c0dd3
ENV RUBYGEMS_VERSION 2.7.11
ENV BUNDLER_VERSION 1.17.3

# Pull packages from debian archive, old repos don't work any more
RUN echo "deb [trusted=yes] http://archive.debian.org/debian/ stretch main contrib non-free" | tee /etc/apt/sources.list

RUN set -ex \
\
&& apt-get update \
# ruby 2.1 on stretch can only support libssl1.0-dev (libssl dev from buildpack-deps is way newer)
# ruby 2.1 needs libssl1.0-dev
&& apt-get install -y --no-install-recommends wget \
&& apt-get remove -y libssl-dev libcurl4-openssl-dev \
&& ARCH=$(dpkg --print-architecture) \
Expand Down Expand Up @@ -89,7 +96,7 @@ RUN mkdir -p "$GEM_HOME" "$BUNDLE_BIN" \

CMD [ "irb" ]

FROM ruby-2.1.10-bookworm
FROM ruby-2.1.10-stretch

# Make apt non-interactive
RUN echo 'APT::Get::Assume-Yes "true";' > /etc/apt/apt.conf.d/90circleci \
Expand All @@ -104,7 +111,7 @@ RUN set -ex; \
apt-get install --force-yes --no-install-recommends \
git mercurial xvfb \
locales sudo openssh-client ca-certificates tar gzip parallel \
net-tools netcat-traditional unzip zip bzip2 gnupg curl wget \
net-tools netcat unzip zip bzip2 gnupg curl wget \
tzdata rsync vim less jq iptables; \
rm -rf /var/lib/apt/lists/*;

Expand All @@ -116,16 +123,25 @@ RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en

# Install Docker and Docker Compose
# Install Docker
RUN set -ex \
&& ARCH=$(dpkg --print-architecture) \
&& wget https://download.docker.com/linux/debian/dists/bookworm/pool/stable/${ARCH}/containerd.io_1.6.28-2_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bookworm/pool/stable/${ARCH}/docker-ce_25.0.4-1~debian.12~bookworm_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bookworm/pool/stable/${ARCH}/docker-ce-cli_25.0.4-1~debian.12~bookworm_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bookworm/pool/stable/${ARCH}/docker-buildx-plugin_0.13.0-1~debian.12~bookworm_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bookworm/pool/stable/${ARCH}/docker-compose-plugin_2.24.7-1~debian.12~bookworm_${ARCH}.deb \
&& dpkg -i *.deb \
&& rm -rf *.deb
&& export DOCKER_VERSION=$(curl --silent --fail --retry 3 https://download.docker.com/linux/static/stable/$(arch)/ | grep -o -e 'docker-[.0-9]*-ce\.tgz' | sort -r | head -n 1) \
&& DOCKER_URL="https://download.docker.com/linux/static/stable/$(arch)/${DOCKER_VERSION}" \
&& echo DOCKER_URL: $DOCKER_URL \
&& curl --silent --show-error --location --fail --retry 3 --output /tmp/docker.tgz "${DOCKER_URL}" \
&& ls -lha /tmp/docker.tgz \
&& tar -xz -C /tmp -f /tmp/docker.tgz \
&& mv /tmp/docker/* /usr/bin \
&& rm -rf /tmp/docker /tmp/docker.tgz \
&& which docker \
&& (docker version || true)

# Install Docker Compose (1.8.0, fairly old)
RUN echo 'deb [trusted=yes] http://archive.debian.org/debian/ stretch-backports main' > /etc/apt/sources.list.d/stretch-backports.list \
&& apt-get update \
&& apt-get install -y docker-compose \
&& docker-compose version \
&& rm -rf /var/lib/apt/lists/*

# Install Dockerize
RUN DOCKERIZE_URL="https://github.com/powerman/dockerize/releases/download/v0.17.0/dockerize-$(uname -s | tr '[:upper:]' '[:lower:]')-$(arch | sed 's/aarch64/arm64/')" \
Expand Down
52 changes: 33 additions & 19 deletions .circleci/images/primary/Dockerfile-2.2.10
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@

# Note: See the "Publishing updates to images" note in ./README.md for how to publish new builds of this container image

# taken from https://hub.docker.com/layers/library/ruby/2.2.10/images/sha256-6c8e6f9667b21565284ef1eef81086adda867e936b74cc74a119ac380d9f00f9?context=explore
# NOTE: Why bullseye and not bookworm like 2.1 and 2.3? Something fails with Ruby 2.2 compilation with newer images leading to segfaults and
# errors like `Marshal.load reentered at marshal_load`: https://github.com/ruby/setup-ruby/issues/496#issuecomment-1520667671
# These do not show up with bullseye and ubuntu-20.
FROM buildpack-deps:bullseye AS ruby-2.2.10-bullseye
# Here be dragons: Last time we tried to move away from Debian stretch, we ran into issues with two gems that are used in some
# of our CI tests/test apps.
# If you're going to try your luck, make sure that the following work after your changes:
# gem install mysql
#
# taken from https://github.com/docker-library/ruby/blob/b5ef401d348ca9b1d9f6a5cb4b25f32bf013daca/2.2/jessie/Dockerfile
FROM buildpack-deps:stretch AS ruby-2.2.10-stretch

# skip installing gem documentation
RUN mkdir -p /usr/local/etc \
Expand All @@ -20,14 +22,17 @@ RUN mkdir -p /usr/local/etc \
ENV RUBY_MAJOR 2.2
ENV RUBY_VERSION 2.2.10
ENV RUBY_DOWNLOAD_SHA256 bf77bcb7e6666ccae8d0882ea12b05f382f963f0a9a5285a328760c06a9ab650
ENV RUBYGEMS_VERSION 2.7.6
ENV BUNDLER_VERSION 1.16.1
ENV RUBYGEMS_VERSION 2.7.11
ENV BUNDLER_VERSION 1.17.3

# Pull packages from debian archive, old repos don't work any more
RUN echo "deb [trusted=yes] http://archive.debian.org/debian/ stretch main contrib non-free" | tee /etc/apt/sources.list

# some of ruby's build scripts are written in ruby
# we purge system ruby later to make sure our final image uses what we just built
RUN set -ex \
&& apt-get update \
# ruby 2.2 on stretch can only support libssl1.0-dev (libssl dev from buildpack-deps is way newer)
# ruby 2.2 needs libssl1.0-dev
&& apt-get install -y --no-install-recommends wget \
&& apt-get remove -y libssl-dev libcurl4-openssl-dev \
&& ARCH=$(dpkg --print-architecture) \
Expand Down Expand Up @@ -95,7 +100,7 @@ RUN mkdir -p "$GEM_HOME" "$BUNDLE_BIN" \

CMD [ "irb" ]

FROM ruby-2.2.10-bullseye
FROM ruby-2.2.10-stretch

# Make apt non-interactive
RUN echo 'APT::Get::Assume-Yes "true";' > /etc/apt/apt.conf.d/90circleci \
Expand All @@ -110,7 +115,7 @@ RUN set -ex; \
apt-get install --force-yes --no-install-recommends \
git mercurial xvfb \
locales sudo openssh-client ca-certificates tar gzip parallel \
net-tools netcat-traditional unzip zip bzip2 gnupg curl wget \
net-tools netcat unzip zip bzip2 gnupg curl wget \
tzdata rsync vim less jq iptables; \
rm -rf /var/lib/apt/lists/*;

Expand All @@ -122,16 +127,25 @@ RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en

# Install Docker and Docker Compose
# Install Docker
RUN set -ex \
&& ARCH=$(dpkg --print-architecture) \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/containerd.io_1.6.28-2_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/docker-ce_25.0.4-1~debian.11~bullseye_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/docker-ce-cli_25.0.4-1~debian.11~bullseye_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/docker-buildx-plugin_0.13.0-1~debian.11~bullseye_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/docker-compose-plugin_2.24.7-1~debian.11~bullseye_${ARCH}.deb \
&& dpkg -i *.deb \
&& rm -rf *.deb
&& export DOCKER_VERSION=$(curl --silent --fail --retry 3 https://download.docker.com/linux/static/stable/$(arch)/ | grep -o -e 'docker-[.0-9]*-ce\.tgz' | sort -r | head -n 1) \
&& DOCKER_URL="https://download.docker.com/linux/static/stable/$(arch)/${DOCKER_VERSION}" \
&& echo DOCKER_URL: $DOCKER_URL \
&& curl --silent --show-error --location --fail --retry 3 --output /tmp/docker.tgz "${DOCKER_URL}" \
&& ls -lha /tmp/docker.tgz \
&& tar -xz -C /tmp -f /tmp/docker.tgz \
&& mv /tmp/docker/* /usr/bin \
&& rm -rf /tmp/docker /tmp/docker.tgz \
&& which docker \
&& (docker version || true)

# Install Docker Compose (1.8.0, fairly old)
RUN echo 'deb [trusted=yes] http://archive.debian.org/debian/ stretch-backports main' > /etc/apt/sources.list.d/stretch-backports.list \
&& apt-get update \
&& apt-get install -y docker-compose \
&& docker-compose version \
&& rm -rf /var/lib/apt/lists/*

# Install Dockerize
RUN DOCKERIZE_URL="https://github.com/powerman/dockerize/releases/download/v0.17.0/dockerize-$(uname -s | tr '[:upper:]' '[:lower:]')-$(arch | sed 's/aarch64/arm64/')" \
Expand Down
145 changes: 43 additions & 102 deletions .circleci/images/primary/Dockerfile-2.3.8
View file
Original file line number Diff line number Diff line change
@@ -1,112 +1,31 @@
# Note: See the "Publishing updates to images" note in ./README.md for how to publish new builds of this container image

# Based on https://hub.docker.com/layers/library/ruby/2.3.8-jessie/images/sha256-cff37ff7e88348a8dd630f9b2d1c679fe9e7b597cfb51a30c42bf2c2afb2eca2?context=explore
#
# Important note: In https://github.com/DataDog/dd-trace-rb/pull/3534 we actually tried to use debian 12 (bookworm) here
# but ran into issues because one of the test apps we use in system-tests uses a version of nokogiri that fails to
# build on that debian version. If you change the base image, make sure that `gem install nokogiri -v '= 1.10.10'`
# successfully installs.
FROM buildpack-deps:bullseye AS ruby-2.3.8-bullseye

# skip installing gem documentation
RUN mkdir -p /usr/local/etc \
&& { \
echo 'install: --no-document'; \
echo 'update: --no-document'; \
} >> /usr/local/etc/gemrc

ENV RUBY_MAJOR 2.3
ENV RUBY_VERSION 2.3.8
ENV RUBY_DOWNLOAD_SHA256 910f635d84fd0d81ac9bdee0731279e6026cb4cd1315bbbb5dfb22e09c5c1dfe
ENV RUBYGEMS_VERSION 2.7.11

# some of ruby's build scripts are written in ruby
# we purge system ruby later to make sure our final image uses what we just built
RUN set -ex \
&& apt-get update \
# ruby 2.3 on stretch can only support libssl1.0-dev (libssl dev from buildpack-deps is way newer)
&& apt-get install -y --no-install-recommends wget \
&& apt-get remove -y libssl-dev libcurl4-openssl-dev \
&& ARCH=$(dpkg --print-architecture) \
&& wget "https://snapshot.debian.org/archive/debian-security/20220317T093342Z/pool/updates/main/o/openssl1.0/libssl1.0.2_1.0.2u-1~deb9u7_${ARCH}.deb" \
&& dpkg -i libssl1.0.2*.deb \
&& rm -rf libssl1.0.2*.deb \
&& wget "https://snapshot.debian.org/archive/debian-security/20220317T093342Z/pool/updates/main/o/openssl1.0/libssl1.0-dev_1.0.2u-1~deb9u7_${ARCH}.deb" \
&& dpkg -i libssl1.0-dev*.deb \
&& rm -rf libssl1.0-dev*.deb \
# some of ruby's build scripts are written in ruby
# we purge system ruby later to make sure our final image uses what we just built
&& buildDeps=' \
bison \
dpkg-dev \
libgdbm-dev \
ruby \
' \
&& apt-get install -y --no-install-recommends $buildDeps \
&& rm -rf /var/lib/apt/lists/* \
\
&& wget -O ruby.tar.xz "https://cache.ruby-lang.org/pub/ruby/${RUBY_MAJOR%-rc}/ruby-$RUBY_VERSION.tar.xz" \
&& echo "$RUBY_DOWNLOAD_SHA256 *ruby.tar.xz" | sha256sum -c - \
\
&& mkdir -p /usr/src/ruby \
&& tar -xJf ruby.tar.xz -C /usr/src/ruby --strip-components=1 \
&& rm ruby.tar.xz \
\
&& cd /usr/src/ruby \
\
# hack in "ENABLE_PATH_CHECK" disabling to suppress:
# warning: Insecure world writable dir
&& { \
echo '#define ENABLE_PATH_CHECK 0'; \
echo; \
cat file.c; \
} > file.c.new \
&& mv file.c.new file.c \
\
&& autoconf \
&& gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
&& ./configure \
--build="$gnuArch" \
--disable-install-doc \
--enable-shared \
&& make -j "$(nproc)" \
&& make install \
\
&& apt-get purge -y --auto-remove $buildDeps \
&& cd / \
&& rm -r /usr/src/ruby \
\
&& gem update --system "$RUBYGEMS_VERSION" \
&& rm -r /root/.gem/

# install things globally, for great justice
# and don't create ".bundle" in all our apps
ENV GEM_HOME /usr/local/bundle
ENV BUNDLE_BIN="$GEM_HOME/bin" \
BUNDLE_SILENCE_ROOT_WARNING=1 \
BUNDLE_APP_CONFIG="$GEM_HOME"
ENV PATH $BUNDLE_BIN:$PATH
RUN mkdir -p "$GEM_HOME" "$BUNDLE_BIN" \
&& chmod 777 "$GEM_HOME" "$BUNDLE_BIN"

CMD [ "irb" ]

FROM ruby-2.3.8-bullseye
# Here be dragons: Last time we tried to move away from Debian stretch, we ran into issues with two gems that are used in some
# of our CI tests/test apps.
# If you're going to try your luck, make sure that the following work after your changes:
# gem install nokogiri -v '= 1.10.10'
# gem install mysql
#
# Last version: https://github.com/docker-library/ruby/blob/31f66490fdb837ddcc5896e3275f2188f2b7b6dd/2.3/stretch/Dockerfile
FROM ruby:2.3.8-stretch

# Make apt non-interactive
RUN echo 'APT::Get::Assume-Yes "true";' > /etc/apt/apt.conf.d/90circleci \
&& echo 'DPkg::Options "--force-confnew";' >> /etc/apt/apt.conf.d/90circleci

ENV DEBIAN_FRONTEND=noninteractive

# Pull packages from debian archive, old repos don't work any more
RUN echo "deb [trusted=yes] http://archive.debian.org/debian/ stretch main contrib non-free" | tee /etc/apt/sources.list

# Install required packages
RUN set -ex; \
apt-get update; \
mkdir -p /usr/share/man/man1; \
apt-get install --force-yes --no-install-recommends \
git mercurial xvfb \
locales sudo openssh-client ca-certificates tar gzip parallel \
net-tools netcat-traditional unzip zip bzip2 gnupg curl wget \
net-tools netcat unzip zip bzip2 gnupg curl wget \
tzdata rsync vim less jq iptables; \
rm -rf /var/lib/apt/lists/*;

Expand All @@ -118,16 +37,25 @@ RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en

# Install Docker and Docker Compose
# Install Docker
RUN set -ex \
&& ARCH=$(dpkg --print-architecture) \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/containerd.io_1.6.28-2_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/docker-ce_25.0.4-1~debian.11~bullseye_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/docker-ce-cli_25.0.4-1~debian.11~bullseye_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/docker-buildx-plugin_0.13.0-1~debian.11~bullseye_${ARCH}.deb \
&& wget https://download.docker.com/linux/debian/dists/bullseye/pool/stable/${ARCH}/docker-compose-plugin_2.24.7-1~debian.11~bullseye_${ARCH}.deb \
&& dpkg -i *.deb \
&& rm -rf *.deb
&& export DOCKER_VERSION=$(curl --silent --fail --retry 3 https://download.docker.com/linux/static/stable/$(arch)/ | grep -o -e 'docker-[.0-9]*-ce\.tgz' | sort -r | head -n 1) \
&& DOCKER_URL="https://download.docker.com/linux/static/stable/$(arch)/${DOCKER_VERSION}" \
&& echo DOCKER_URL: $DOCKER_URL \
&& curl --silent --show-error --location --fail --retry 3 --output /tmp/docker.tgz "${DOCKER_URL}" \
&& ls -lha /tmp/docker.tgz \
&& tar -xz -C /tmp -f /tmp/docker.tgz \
&& mv /tmp/docker/* /usr/bin \
&& rm -rf /tmp/docker /tmp/docker.tgz \
&& which docker \
&& (docker version || true)

# Install Docker Compose (1.8.0, fairly old)
RUN echo 'deb [trusted=yes] http://archive.debian.org/debian/ stretch-backports main' > /etc/apt/sources.list.d/stretch-backports.list \
&& apt-get update \
&& apt-get install -y docker-compose \
&& docker-compose version \
&& rm -rf /var/lib/apt/lists/*

# Install Dockerize
RUN DOCKERIZE_URL="https://github.com/powerman/dockerize/releases/download/v0.17.0/dockerize-$(uname -s | tr '[:upper:]' '[:lower:]')-$(arch | sed 's/aarch64/arm64/')" \
Expand All @@ -136,6 +64,19 @@ RUN DOCKERIZE_URL="https://github.com/powerman/dockerize/releases/download/v0.17
&& chmod +x /usr/local/bin/dockerize \
&& dockerize --version

# Install RubyGems
RUN mkdir -p "$GEM_HOME" && chmod -R 777 "$GEM_HOME"

# Install RubyGems and Bundler
# NOTE: Rubygems 3.0.6 is the last version that seems to work fine in this image AND not drag bundler 2 along.
# Later versions are either broken or they force the use of bundler 2, which we can't use because some of our
# dependencies (e.g. rails 3.0.20) don't like it.
RUN gem update --system '3.0.6'
# Ruby 2.3 can support Bundler 2+
# But hold back to < 2 for now, because some dependencies require it.
RUN gem install bundler -v '1.17.3'
ENV BUNDLE_SILENCE_ROOT_WARNING 1

RUN mkdir /app
WORKDIR /app

Expand Down

0 comments on commit 2119ba1

Please sign in to comment.