Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: upgrade to openssl@3 #13746

Merged
merged 14 commits into from
Dec 4, 2023
Merged

build: upgrade to openssl@3 #13746

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
b1a9440
build: try upgrade openssl
xxchan Nov 30, 2023
dcf3acd
fix
xxchan Nov 30, 2023
77e35fe
fix
xxchan Nov 30, 2023
8811800
try install lld from source
xxchan Nov 30, 2023
4f1558f
make -j
xxchan Nov 30, 2023
c741ef8
install pre-built lld
xxchan Nov 30, 2023
116fb03
fix
xxchan Nov 30, 2023
f39edab
fix
xxchan Nov 30, 2023
efebb49
try debug build
xxchan Nov 30, 2023
8f3992d
Revert "try debug build"
xxchan Nov 30, 2023
e5beed6
add a note
xxchan Dec 3, 2023
3f9aa3c
Merge branch 'main' into xxchan/openssl
xxchan Dec 4, 2023
925b3a3
also upgrade openssl
xxchan Dec 4, 2023
ad144d7
fix trailing space
xxchan Dec 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions ci/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,16 @@ services:
- ..:/risingwave

release-env:
# build binaries on a earlier Linux distribution (therefore with earlier version GLIBC)
# See https://github.com/risingwavelabs/risingwave/issues/4556 for more details.
#
# GLIBC versions on some systems:
# Amazon Linux 2: 2.26 (EOL 2025-06-30) (We will definitely want to support this)
# AL2023: 2.34
# Ubuntu 18.04: 2.27 (Already EOL 2023-05-31)
# Ubuntu 20.04: 2.31
#
# manylinux2014: CentOS 7 (EOL 2024-06-30), GLIBC 2.17
Copy link
Contributor

@xiangjinwu xiangjinwu Dec 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just adding a link for future (in 7 months) reference:
https://github.com/pypa/manylinux

The goal of the manylinux project is to provide a convenient way to distribute binary Python extensions as wheels on Linux. This effort has produced PEP 513 (manylinux1), PEP 571 (manylinux2010), PEP 599 (manylinux2014) and PEP 600 (manylinux_x_y).

PEP 599 defines the following platform tags:

  • manylinux2014_x86_64
  • ...

Wheels are built on CentOS 7 which will reach End of Life (EOL) on June 30th, 2024.

PEP 600 has been designed to be "future-proof" and does not enforce specific symbols and a specific distro to build. It only states that a wheel tagged manylinux_x_y shall work on any distro based on glibc>=x.y. The manylinux project supports:

  • manylinux_2_24 images for x86_64, i686, aarch64, ppc64le and s390x.
  • manylinux_2_28 images for x86_64, aarch64, ppc64le and s390x.

manylinux_2_28 (AlmaLinux 8 based)

manylinux_2_24 (Debian 9 based) - EOL

Support for manylinux_2_24 has ended on January 1st, 2023.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very unfortunately 2_24 is already EOL. When 2014 (2_17) is EOL, we won’t have a nice alternative immediately. 🤡

2_28 is larger than Amazon Linux. And... Amazon Linux‘s EOL is sooooo long

image: quay.io/pypa/manylinux2014_x86_64
working_dir: /mnt
volumes:
Expand Down
24 changes: 15 additions & 9 deletions ci/scripts/release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,21 @@ if [ "${BUILDKITE_SOURCE}" != "schedule" ] && [ "${BUILDKITE_SOURCE}" != "webhoo
exit 0
fi

echo "--- Install aws cli"
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -q awscliv2.zip && ./aws/install && mv /usr/local/bin/aws /bin/aws

echo "--- Install lld"
# The lld in the CentOS 7 repository is too old and contains a bug that causes a linker error.
# So we install a newer version here. (17.0.6, latest version at the time of writing)
# It is manually built in the same environent and uploaded to S3.
aws s3 cp s3://ci-deps-dist/llvm-lld-manylinux2014_x86_64.tar.gz .
tar -zxvf llvm-lld-manylinux2014_x86_64.tar.gz --directory=/usr/local
ld.lld --version

echo "--- Install dependencies for openssl"
yum install -y perl-core

echo "--- Install java and maven"
yum install -y java-11-openjdk java-11-openjdk-devel wget python3 cyrus-sasl-devel
pip3 install toml-cli
Expand All @@ -30,15 +45,6 @@ unzip -o protoc-3.15.8-linux-x86_64.zip -d protoc
mv ./protoc/bin/protoc /usr/local/bin/
mv ./protoc/include/* /usr/local/include/

echo "--- Install lld"
yum install -y centos-release-scl-rh
yum install -y llvm-toolset-7.0-lld
source /opt/rh/llvm-toolset-7.0/enable

echo "--- Install aws cli"
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -q awscliv2.zip && ./aws/install && mv /usr/local/bin/aws /bin/aws

echo "--- Check risingwave release version"
if [[ -n "${BUILDKITE_TAG}" ]]; then
CARGO_PKG_VERSION="$(toml get --toml-path Cargo.toml workspace.package.version)"
Expand Down
2 changes: 1 addition & 1 deletion src/utils/pgwire/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ byteorder = "1.5"
bytes = "1"
futures = { version = "0.3", default-features = false, features = ["alloc"] }
itertools = "0.12"
openssl = "0.10.57"
openssl = "0.10.60"
panic-message = "0.3"
risingwave_common = { workspace = true }
risingwave_sqlparser = { workspace = true }
Expand Down
6 changes: 2 additions & 4 deletions src/utils/workspace-config/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,7 @@ rw-dynamic-link = ["zstd-sys"]

[dependencies]
log = { version = "0.4", features = ["release_max_level_debug"] }
# FIXME: 0.9.93 upgrades openssl-src to openssl@3, but we failed to build it.
# fix it later https://github.com/risingwavelabs/risingwave/pull/12198
openssl-sys = { version = "=0.9.92", optional = true, features = ["vendored"] }
openssl-sys = { version = "0.9.96", optional = true, features = ["vendored"] }
sasl2-sys = { version = "0.1", optional = true, features = ["gssapi-vendored"] }
tracing = { version = "0.1", features = ["release_max_level_debug"] }
zstd-sys = { version = "2", optional = true, default-features = false, features = ["pkg-config"] }
Expand All @@ -28,7 +26,7 @@ zstd-sys = { version = "2", optional = true, default-features = false, features
# FIXME(xxchan): This is a temporary fix due to how cargo and hakari works. See related PR for more details.
# We will revisit how to handle workspace-hack and build-dependency issues later.
[build-dependencies]
openssl-sys = { version = "=0.9.92", optional = true, features = ["vendored"] }
openssl-sys = { version = "0.9.96", optional = true, features = ["vendored"] }

[lints]
workspace = true