Skip to content

Commit fa275f1

Browse files
parthchandraandygrove
parthchandra
and
andygrove
authored
feat: implement scripts for binary release build (#932)
* feat: implement scripts for binary release build * Install to temp local maven repo and updates for MacOS * newline * Use independent docker images for different architectures instead of a multi-arch image * update docs and cleanup * remove unused code * fail build script on error * Build all profiles * remove duplicate target from makefile --------- Co-authored-by: Andy Grove <[email protected]>
1 parent 9dfd6d1 commit fa275f1

File tree

6 files changed

+422
-0
lines changed

6 files changed

+422
-0
lines changed

‎Makefile

+16
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,22 @@ format:
4646
./mvnw compile test-compile scalafix:scalafix -Psemanticdb $(PROFILES)
4747
./mvnw spotless:apply $(PROFILES)
4848

49+
# build native libs for amd64 architecture Linux/MacOS on a Linux/amd64 machine/container
50+
core-amd64-libs:
51+
cd native && cargo build -j 2 --release
52+
ifdef HAS_OSXCROSS
53+
rustup target add x86_64-apple-darwin
54+
cd native && cargo build -j 2 --target x86_64-apple-darwin --release
55+
endif
56+
57+
# build native libs for arm64 architecture Linux/MacOS on a Linux/arm64 machine/container
58+
core-arm64-libs:
59+
cd native && cargo build -j 2 --release
60+
ifdef HAS_OSXCROSS
61+
rustup target add aarch64-apple-darwin
62+
cd native && cargo build -j 2 --target aarch64-apple-darwin --release
63+
endif
64+
4965
core-amd64:
5066
rustup target add x86_64-apple-darwin
5167
cd native && RUSTFLAGS="-Ctarget-cpu=skylake -Ctarget-feature=-prefer-256-bit" CC=o64-clang CXX=o64-clang++ cargo build --target x86_64-apple-darwin --release

‎dev/release/README.md

+53
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,50 @@ python3 generate-changelog.py 0.0.0 HEAD 0.1.0 > ../changelog/0.1.0.md
8181
Create a PR against the _main_ branch to add this change log and once this is approved and merged, cherry-pick the
8282
commit into the release branch.
8383

84+
### Build the jars
85+
86+
#### Setup to do the build
87+
The build process requires Docker. Download the latest Docker Desktop from https://www.docker.com/products/docker-desktop/.
88+
If you have multiple docker contexts running switch to the context of the Docker Desktop. For example -
89+
90+
```shell
91+
$ docker context ls
92+
NAME DESCRIPTION DOCKER ENDPOINT ERROR
93+
default Current DOCKER_HOST based configuration unix:///var/run/docker.sock
94+
desktop-linux Docker Desktop unix:///Users/parth/.docker/run/docker.sock
95+
my_custom_context * tcp://192.168.64.2:2376
96+
97+
$ docker context use desktop-linux
98+
```
99+
#### Run the build script
100+
The `build-release-comet.sh` script will create a docker image for each architecture and use the image
101+
to build the platform specific binaries. These builder images are created every time this script is run.
102+
The script optionally allows overriding of the repository and branch to build the binaries from (Note that
103+
the local git repo is not used in the building of the binaries, but it is used to build the final uber jar).
104+
105+
```shell
106+
Usage: build-release-comet.sh [options]
107+
108+
This script builds comet native binaries inside a docker image. The image is named
109+
"comet-rm" and will be generated by this script
110+
111+
Options are:
112+
113+
-r [repo] : git repo (default: https://github.com/apache/datafusion-comet.git)
114+
-b [branch] : git branch (default: release)
115+
-t [tag] : tag for the spark-rm docker image to use for building (default: "latest").
116+
```
117+
118+
Example:
119+
120+
```shell
121+
cd dev/release && ./build-release-comet.sh && cd ../..
122+
```
123+
124+
#### Build output
125+
The build output is installed to a temporary local maven repository. The build script will print the name of the repository
126+
location at the end. This location will be required at the time of deploying the artifacts to a staging repository
127+
84128
### Tag the Release Candidate
85129

86130
Tag the release branch with `0.1.0-rc1` and push to the `apache` repo
@@ -105,6 +149,15 @@ Run the create-tarball script on the release candidate tag (`0.1.0-rc1`) to crea
105149
GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 0.1.0 1
106150
```
107151

152+
### Publish the maven artifacts
153+
#### Setup maven
154+
##### One time project setup
155+
Setting up your project in the ASF Nexus Repository from here: https://infra.apache.org/publishing-maven-artifacts.html
156+
##### Release Manager Setup
157+
Set up your development environment from here: https://infra.apache.org/publishing-maven-artifacts.html
158+
159+
TODO: build and publish a release candidate to nexus.
160+
108161
### Start an Email Voting Thread
109162

110163
Send the email that is generated in the previous step to `[email protected]`.

‎dev/release/build-release-comet.sh

+202
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
21+
set -e
22+
23+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
24+
COMET_HOME_DIR=$SCRIPT_DIR/../..
25+
26+
function usage {
27+
local NAME=$(basename $0)
28+
cat <<EOF
29+
Usage: $NAME [options]
30+
31+
This script builds comet native binaries inside a docker image. The image is named
32+
"comet-rm" and will be generated by this script
33+
34+
Options are:
35+
36+
-r [repo] : git repo (default: ${REPO})
37+
-b [branch] : git branch (default: ${BRANCH})
38+
-t [tag] : tag for the spark-rm docker image to use for building (default: "latest").
39+
EOF
40+
exit 1
41+
}
42+
43+
function cleanup()
44+
{
45+
if [ $CLEANUP != 0 ]
46+
then
47+
echo Cleaning up ...
48+
if [ "$(docker ps -a | grep comet-arm64-builder-container)" != "" ]
49+
then
50+
docker rm comet-arm64-builder-container
51+
fi
52+
if [ "$(docker ps -a | grep comet-amd64-builder-container)" != "" ]
53+
then
54+
docker rm comet-amd64-builder-container
55+
fi
56+
CLEANUP=0
57+
fi
58+
}
59+
60+
trap cleanup SIGINT SIGTERM EXIT
61+
62+
CLEANUP=1
63+
64+
REPO="https://github.com/apache/datafusion-comet.git"
65+
BRANCH="release"
66+
MACOS_SDK=
67+
HAS_MACOS_SDK="false"
68+
IMGTAG=latest
69+
70+
while getopts "b:hr:t:" opt; do
71+
case $opt in
72+
r) REPO="$OPTARG";;
73+
b) BRANCH="$OPTARG";;
74+
t) IMGTAG="$OPTARG" ;;
75+
h) usage ;;
76+
\?) error "Invalid option. Run with -h for help." ;;
77+
esac
78+
done
79+
80+
echo "Building binaries from $REPO/$BRANCH"
81+
82+
WORKING_DIR="$SCRIPT_DIR/comet-rm/workdir"
83+
cp $SCRIPT_DIR/../cargo.config $WORKING_DIR
84+
85+
# TODO: Search for Xcode (Once building macos binaries works)
86+
#PS3="Select Xcode:"
87+
#select xcode_path in `find . -name "${MACOS_SDK}"`
88+
#do
89+
# echo "found Xcode in $xcode_path"
90+
# cp $xcode_path $WORKING_DIR
91+
# break
92+
#done
93+
94+
if [ -f "${WORKING_DIR}/${MACOS_SDK}" ]
95+
then
96+
HAS_MACOS_SDK="true"
97+
fi
98+
99+
BUILDER_IMAGE_ARM64="comet-rm-arm64:$IMGTAG"
100+
BUILDER_IMAGE_AMD64="comet-rm-amd64:$IMGTAG"
101+
102+
# Build the docker image in which we will do the build
103+
docker build \
104+
--platform=linux/arm64 \
105+
-t "$BUILDER_IMAGE_ARM64" \
106+
--build-arg HAS_MACOS_SDK=${HAS_MACOS_SDK} \
107+
--build-arg MACOS_SDK=${MACOS_SDK} \
108+
"$SCRIPT_DIR/comet-rm"
109+
110+
docker build \
111+
--platform=linux/amd64 \
112+
-t "$BUILDER_IMAGE_AMD64" \
113+
--build-arg HAS_MACOS_SDK=${HAS_MACOS_SDK} \
114+
--build-arg MACOS_SDK=${MACOS_SDK} \
115+
"$SCRIPT_DIR/comet-rm"
116+
117+
# Clean previous Java build
118+
pushd $COMET_HOME_DIR && ./mvnw clean && popd
119+
120+
# Run the builder container for each architecture. The entrypoint script will build the binaries
121+
122+
# AMD64
123+
echo "Building amd64 binary"
124+
docker run \
125+
--name comet-amd64-builder-container \
126+
--memory 24g \
127+
--cpus 6 \
128+
-it \
129+
--platform linux/amd64 \
130+
$BUILDER_IMAGE_AMD64 "${REPO}" "${BRANCH}" amd64
131+
132+
if [ $? != 0 ]
133+
then
134+
echo "Building amd64 binary failed."
135+
exit 1
136+
fi
137+
138+
# ARM64
139+
echo "Building arm64 binary"
140+
docker run \
141+
--name comet-arm64-builder-container \
142+
--memory 24g \
143+
--cpus 6 \
144+
-it \
145+
--platform linux/arm64 \
146+
$BUILDER_IMAGE_ARM64 "${REPO}" "${BRANCH}" arm64
147+
148+
if [ $? != 0 ]
149+
then
150+
echo "Building arm64 binary failed."
151+
exit 1
152+
fi
153+
154+
echo "Building binaries completed"
155+
echo "Copying to java build directories"
156+
157+
JVM_TARGET_DIR=$COMET_HOME_DIR/common/target/classes/org/apache/comet
158+
mkdir -p $JVM_TARGET_DIR
159+
160+
mkdir -p $JVM_TARGET_DIR/linux/amd64
161+
docker cp \
162+
comet-amd64-builder-container:"/opt/comet-rm/comet/native/target/release/libcomet.so" \
163+
$JVM_TARGET_DIR/linux/amd64/
164+
165+
if [ "$HAS_MACOS_SDK" == "true" ]
166+
then
167+
mkdir -p $JVM_TARGET_DIR/darwin/x86_64
168+
docker cp \
169+
comet-amd64-builder-container:"/opt/comet-rm/comet/native/target/x86_64-apple-darwin/release/libcomet.dylib" \
170+
$JVM_TARGET_DIR/darwin/x86_64/
171+
fi
172+
173+
mkdir -p $JVM_TARGET_DIR/linux/aarch64
174+
docker cp \
175+
comet-arm64-builder-container:"/opt/comet-rm/comet/native/target/release/libcomet.so" \
176+
$JVM_TARGET_DIR/linux/aarch64/
177+
178+
if [ "$HAS_MACOS_SDK" == "true" ]
179+
then
180+
mkdir -p $JVM_TARGET_DIR/linux/aarch64
181+
docker cp \
182+
comet-arm64-builder-container:"/opt/comet-rm/comet/native/target/aarch64-apple-darwin/release/libcomet.dylib" \
183+
$JVM_TARGET_DIR/darwin/aarch64/
184+
fi
185+
186+
# Build final jar
187+
echo "Building uber jar and publishing it locally"
188+
pushd $COMET_HOME_DIR
189+
190+
GIT_HASH=$(git rev-parse --short HEAD)
191+
LOCAL_REPO=$(mktemp -d /tmp/comet-staging-repo-XXXXX)
192+
193+
./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.4 -P scala-2.12 -DskipTests install
194+
./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.4 -P scala-2.13 -DskipTests install
195+
./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.3 -P scala-2.12 -DskipTests install
196+
./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.3 -P scala-2.13 -DskipTests install
197+
./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.5 -P scala-2.12 -DskipTests install
198+
./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.5 -P scala-2.13 -DskipTests install
199+
200+
echo "Installed to local repo: ${LOCAL_REPO}"
201+
202+
popd

‎dev/release/comet-rm/Dockerfile

+91
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
ARG HAS_MACOS_SDK="false"
18+
19+
FROM ubuntu:20.04 AS base
20+
21+
USER root
22+
23+
# For apt to be noninteractive
24+
ENV DEBIAN_FRONTEND=noninteractive
25+
ENV DEBCONF_NONINTERACTIVE_SEEN=true
26+
27+
ENV LC_ALL=C
28+
# Install pr-requisites for rust
29+
RUN export LC_ALL=C \
30+
&& apt-get update \
31+
&& apt-get install --no-install-recommends -y \
32+
ca-certificates \
33+
build-essential \
34+
curl \
35+
wget \
36+
git \
37+
llvm \
38+
clang \
39+
libssl-dev \
40+
lzma-dev \
41+
liblzma-dev \
42+
openssh-client \
43+
cmake \
44+
cpio \
45+
libxml2-dev \
46+
patch \
47+
bzip2 \
48+
libbz2-dev \
49+
zlib1g-dev
50+
51+
52+
# Install rust
53+
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
54+
ENV PATH="/root/.cargo/bin:${PATH}"
55+
RUN cargo install cargo2junit
56+
57+
# Stage to add OSXCross if MacOSSDK is provided
58+
FROM base AS with-macos-sdk-true
59+
ARG MACOS_SDK
60+
61+
COPY workdir/$MACOS_SDK /opt/xcode/
62+
63+
RUN if [ "$TARGETPLATFORM" = "linux/arm64" ]; then \
64+
rustup target add aarch64-apple-darwin; \
65+
elif [ "$TARGETPLATFORM" = "linux/amd64" ]; then \
66+
rustup target add x86_64-apple-darwin; \
67+
fi
68+
69+
# Build OSXCross
70+
RUN cd /opt && git clone --depth 1 https://github.com/tpoechtrager/osxcross.git \
71+
&& cd /opt/osxcross \
72+
&& ./tools/gen_sdk_package_pbzx.sh /opt/xcode/${MACOS_SDK} \
73+
&& cd .. \
74+
&& cp /opt/osxcross/*.tar.xz tarballs \
75+
&& UNATTENDED=1 ./build.sh
76+
ENV PATH="/opt/osxcross/target/bin:${PATH}"
77+
# Use osxcross toolchain for cargo
78+
COPY workdir/cargo.config /root/.cargo/config
79+
ENV HAS_OSXCROSS="true"
80+
81+
# Placeholder Stage if MacOSSDK is not provided
82+
FROM base AS with-macos-sdk-false
83+
RUN echo "Building without MacOS"
84+
85+
86+
FROM with-macos-sdk-${HAS_MACOS_SDK} AS final
87+
88+
COPY build-comet-native-libs.sh /opt/comet-rm/build-comet-native-libs.sh
89+
WORKDIR /opt/comet-rm
90+
91+
ENTRYPOINT [ "/opt/comet-rm/build-comet-native-libs.sh"]

0 commit comments

Comments
 (0)