Skip to content

chore: Update release scripts to reflect move to TLP #10235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 35 additions & 46 deletions dev/release/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ patch release:
- Created a personal access token in GitHub for changelog automation script.
- Github PAT should be created with `repo` access
- Make sure your signing key is added to the following files in SVN:
- https://dist.apache.org/repos/dist/dev/arrow/KEYS
- https://dist.apache.org/repos/dist/release/arrow/KEYS
- https://dist.apache.org/repos/dist/dev/datafusion/KEYS
- https://dist.apache.org/repos/dist/release/datafusion/KEYS

### How to add signing key

Expand All @@ -58,8 +58,8 @@ See instructions at https://infra.apache.org/release-signing.html#generate for g
Committers can add signing keys in Subversion client with their ASF account. e.g.:

```bash
$ svn co https://dist.apache.org/repos/dist/dev/arrow
$ cd arrow
$ svn co https://dist.apache.org/repos/dist/dev/datafusion
$ cd datafusion
$ editor KEYS
$ svn ci KEYS
```
Expand Down Expand Up @@ -128,7 +128,7 @@ release.

See [#9697](https://github.com/apache/datafusion/pull/9697) for an example.

Here are the commands that could be used to prepare the `5.1.0` release:
Here are the commands that could be used to prepare the `38.0.0` release:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to get rid of hardcoded versions somehow....

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should really try and automate more of the release process so that we just run scripts and pass in the version number

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One way to to so might be to make a script that just does all of this (rather than having manual instructions). And then the script could have $VERSION or whatever


### Update Version

Expand All @@ -139,10 +139,10 @@ git fetch apache
git checkout apache/main
```

Update datafusion version in `datafusion/Cargo.toml` to `5.1.0`:
Update datafusion version in `datafusion/Cargo.toml` to `38.0.0`:

```
./dev/update_datafusion_versions.py 5.1.0
./dev/update_datafusion_versions.py 38.0.0
```

Lastly commit the version change:
Expand All @@ -167,7 +167,7 @@ Pick numbers in sequential order, with `0` for `rc0`, `1` for `rc1`, etc.
While the official release artifacts are signed tarballs and zip files, we also
tag the commit it was created for convenience and code archaeology.

Using a string such as `5.1.0` as the `<version>`, create and push the tag by running these commands:
Using a string such as `38.0.0` as the `<version>`, create and push the tag by running these commands:

```shell
git fetch apache
Expand All @@ -181,29 +181,29 @@ git push apache <version>
Run `create-tarball.sh` with the `<version>` tag and `<rc>` and you found in previous steps:

```shell
GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 5.1.0 0
GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 38.0.0 0
```

The `create-tarball.sh` script

1. creates and uploads all release candidate artifacts to the [arrow
dev](https://dist.apache.org/repos/dist/dev/arrow) location on the
1. creates and uploads all release candidate artifacts to the [datafusion
dev](https://dist.apache.org/repos/dist/dev/datafusion) location on the
apache distribution svn server

2. provide you an email template to
send to dev@arrow.apache.org for release voting.
send to dev@datafusion.apache.org for release voting.

### Vote on Release Candidate artifacts

Send the email output from the script to dev@arrow.apache.org. The email should look like
Send the email output from the script to dev@datafusion.apache.org. The email should look like

```
To: dev@arrow.apache.org
Subject: [VOTE][DataFusion] Release Apache DataFusion 5.1.0 RC0
To: dev@datafusion.apache.org
Subject: [VOTE] Release Apache DataFusion 38.0.0 RC1

Hi,

I would like to propose a release of Apache DataFusion version 5.1.0.
I would like to propose a release of Apache DataFusion version 38.0.0.

This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1]
The proposed release artifacts and signatures are hosted at [2].
Expand All @@ -214,16 +214,16 @@ and vote on the release.

The vote will be open for at least 72 hours.

[ ] +1 Release this as Apache DataFusion 5.1.0
[ ] +1 Release this as Apache DataFusion 38.0.0
[ ] +0
[ ] -1 Do not release this as Apache DataFusion 5.1.0 because...
[ ] -1 Do not release this as Apache DataFusion 38.0.0 because...

Here is my vote:

+1

[1]: https://github.com/apache/datafusion/tree/a5dd428f57e62db20a945e8b1895de91405958c4
[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-datafusion-5.1.0
[2]: https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-38.0.0
[3]: https://github.com/apache/datafusion/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md
```

Expand All @@ -234,7 +234,7 @@ For the release to become "official" it needs at least three PMC members to vote
The `dev/release/verify-release-candidate.sh` is a script in this repository that can assist in the verification process. Run it like:

```
./dev/release/verify-release-candidate.sh 5.1.0 0
./dev/release/verify-release-candidate.sh 38.0.0 0
```

#### If the release is not approved
Expand All @@ -249,11 +249,11 @@ NOTE: steps in this section can only be done by PMC members.
### After the release is approved

Move artifacts to the release location in SVN, e.g.
https://dist.apache.org/repos/dist/release/datafusion/datafusion-5.1.0/, using
https://dist.apache.org/repos/dist/release/datafusion/datafusion-38.0.0/, using
the `release-tarball.sh` script:

```shell
./dev/release/release-tarball.sh 5.1.0 0
./dev/release/release-tarball.sh 38.0.0 0
```

Congratulations! The release is now official!
Expand All @@ -263,9 +263,9 @@ Congratulations! The release is now official!
Tag the same release candidate commit with the final release tag

```
git co apache/5.1.0-rc0
git tag 5.1.0
git push apache 5.1.0
git co apache/38.0.0-rc0
git tag 38.0.0
git push apache 38.0.0
```

### Publish on Crates.io
Expand Down Expand Up @@ -300,7 +300,7 @@ of the following crates:
Download and unpack the official release tarball

Verify that the Cargo.toml in the tarball contains the correct version
(e.g. `version = "5.1.0"`) and then publish the crates by running the script `release-crates.sh`
(e.g. `version = "38.0.0"`) and then publish the crates by running the script `release-crates.sh`
in a directory extracted from the source tarball that was voted on. Note that this script doesn't
work if run in a Git repo.

Expand Down Expand Up @@ -413,10 +413,9 @@ https://crates.io/crates/datafusion-substrait/28.0.0

### Add the release to Apache Reporter

Add the release to https://reporter.apache.org/addrelease.html?arrow with a version name prefixed with `RS-DATAFUSION-`,
for example `RS-DATAFUSION-14.0.0`.
Add the release to https://reporter.apache.org/addrelease.html?datafusion using the version number e.g. 38.0.0.

The release information is used to generate a template for a board report (see example
The release information is used to generate a template for a board report (see example from Apache Arrow project
[here](https://github.com/apache/arrow/pull/14357)).

### Delete old RCs and Releases
Expand All @@ -431,13 +430,13 @@ Release candidates should be deleted once the release is published.
Get a list of DataFusion release candidates:

```bash
svn ls https://dist.apache.org/repos/dist/dev/arrow | grep datafusion
svn ls https://dist.apache.org/repos/dist/dev/datafusion
```

Delete a release candidate:

```bash
svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-7.1.0-rc1/
svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-38.0.0-rc1/
```

#### Deleting old releases from `release` svn
Expand All @@ -447,35 +446,25 @@ Only the latest release should be available. Delete old releases after publishin
Get a list of DataFusion releases:

```bash
svn ls https://dist.apache.org/repos/dist/release/arrow | grep datafusion
svn ls https://dist.apache.org/repos/dist/release/datafusion
```

Delete a release:

```bash
svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-7.0.0
svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-37.0.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-37.0.0
svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-38.0.0

```

### Publish the User Guide to the Arrow Site

- Run the `build.sh` in the `docs` directory from the release tarball.
- Clone the [arrow-site](https://github.com/apache/arrow-site) repository
- Checkout the `asf-site` branch
- Copy content from `docs/build/html/*` to the `datafusion` directory in arrow-site
- Create a PR against the `asf-site` branch ([example](https://github.com/apache/arrow-site/pull/237))
- Once the PR is merged, the content will be published to https://datafusion.apache.org/ by GitHub Pages (this
can take some time).

### Optional: Write a blog post announcing the release

We typically crowdsource release announcements by collaborating on a Google document, usually starting
We typically crowd source release announcements by collaborating on a Google document, usually starting
with a copy of the previous release announcement.

Run the following commands to get the number of commits and number of unique contributors for inclusion in the blog post.

```bash
git log --pretty=oneline 10.0.0..11.0.0 datafusion datafusion-cli datafusion-examples | wc -l
git shortlog -sn 10.0.0..11.0.0 datafusion datafusion-cli datafusion-examples | wc -l
git log --pretty=oneline 37.0.0..38.0.0 datafusion datafusion-cli datafusion-examples | wc -l
git shortlog -sn 37.0.0..38.0.0 datafusion datafusion-cli datafusion-examples | wc -l
```

Once there is consensus on the contents of the post, create a PR to add a blog post to the
Expand Down
20 changes: 10 additions & 10 deletions dev/release/create-tarball.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@
# Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/create-tarball.sh

# This script creates a signed tarball in
# dev/dist/apache-arrow-datafusion-<version>-<sha>.tar.gz and uploads it to
# the "dev" area of the dist.apache.arrow repository and prepares an
# email for sending to the dev@arrow.apache.org list for a formal
# dev/dist/apache-datafusion-<version>-<sha>.tar.gz and uploads it to
# the "dev" area of the dist.apache.datafusion repository and prepares an
# email for sending to the dev@datafusion.apache.org list for a formal
# vote.
#
# See release/README.md for full release instructions
Expand Down Expand Up @@ -65,21 +65,21 @@ tag="${version}-rc${rc}"
echo "Attempting to create ${tarball} from tag ${tag}"
release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag})

release=apache-arrow-datafusion-${version}
release=apache-datafusion-${version}
distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc}
tarname=${release}.tar.gz
tarball=${distdir}/${tarname}
url="https://dist.apache.org/repos/dist/dev/arrow/${release}-rc${rc}"
url="https://dist.apache.org/repos/dist/dev/datafusion/${release}-rc${rc}"

if [ -z "$release_hash" ]; then
echo "Cannot continue: unknown git tag: ${tag}"
fi

echo "Draft email for dev@arrow.apache.org mailing list"
echo "Draft email for dev@datafusion.apache.org mailing list"
echo ""
echo "---------------------------------------------------------"
cat <<MAIL
To: dev@arrow.apache.org
To: dev@datafusion.apache.org
Subject: [VOTE][RUST][DataFusion] Release Apache DataFusion ${version} RC${rc}
Hi,

Expand Down Expand Up @@ -124,12 +124,12 @@ echo "Signing tarball and creating checksums"
gpg --armor --output ${tarball}.asc --detach-sig ${tarball}
# create signing with relative path of tarball
# so that they can be verified with a command such as
# shasum --check apache-arrow-datafusion-4.1.0-rc2.tar.gz.sha512
# shasum --check apache-datafusion-38.0.0-rc1.tar.gz.sha512
(cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256
(cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512


echo "Uploading to apache dist/dev to ${url}"
svn co --depth=empty https://dist.apache.org/repos/dist/dev/arrow ${SOURCE_TOP_DIR}/dev/dist
echo "Uploading to datafusion dist/dev to ${url}"
svn co --depth=empty https://dist.apache.org/repos/dist/dev/datafusion ${SOURCE_TOP_DIR}/dev/dist
svn add ${distdir}
svn ci -m "Apache DataFusion ${version} ${rc}" ${distdir}
6 changes: 3 additions & 3 deletions dev/release/publish_homebrew.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ else # Fallback
num_processing_units=1
fi

url="https://www.apache.org/dyn/closer.lua?path=arrow/arrow-datafusion-${version}/apache-arrow-datafusion-${version}.tar.gz"
sha256="$(curl https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-${version}/apache-arrow-datafusion-${version}.tar.gz.sha256 | cut -d' ' -f1)"
url="https://www.apache.org/dyn/closer.lua?path=datafusion/datafusion-${version}/apache-datafusion-${version}.tar.gz"
sha256="$(curl https://dist.apache.org/repos/dist/release/datafusion/datafusion-${version}/apache-datafusion-${version}.tar.gz.sha256 | cut -d' ' -f1)"

pushd "$(brew --repository homebrew/core)"

Expand All @@ -52,7 +52,7 @@ fi
echo "Updating working copy"
git fetch --all --prune --tags --force -j$num_processing_units

branch=apache-arrow-datafusion-${version}
branch=apache-datafusion-${version}
echo "Creating branch: ${branch}"
git branch -D ${branch} || :
git checkout -b ${branch} origin/master
Expand Down
78 changes: 0 additions & 78 deletions dev/release/rat_exclude_files.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,84 +15,8 @@ ci/etc/*.patch
ci/vcpkg/*.patch
CHANGELOG.md
datafusion/CHANGELOG.md
python/CHANGELOG.md
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

conbench/benchmarks.json
conbench/requirements.txt
conbench/requirements-test.txt
conbench/.flake8
conbench/.isort.cfg
dev/requirements*.txt
dev/archery/MANIFEST.in
dev/archery/requirements*.txt
dev/archery/archery/tests/fixtures/*
dev/archery/archery/crossbow/tests/fixtures/*
dev/release/rat_exclude_files.txt
dev/tasks/homebrew-formulae/apache-arrow.rb
dev/tasks/linux-packages/apache-arrow-apt-source/debian/apache-arrow-apt-source.install
dev/tasks/linux-packages/apache-arrow-apt-source/debian/compat
dev/tasks/linux-packages/apache-arrow-apt-source/debian/control
dev/tasks/linux-packages/apache-arrow-apt-source/debian/rules
dev/tasks/linux-packages/apache-arrow-apt-source/debian/source/format
dev/tasks/linux-packages/apache-arrow/debian/compat
dev/tasks/linux-packages/apache-arrow/debian/control.in
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-arrow-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-arrow-cuda-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-arrow-dataset-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-gandiva-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-parquet-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-plasma-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-flight-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-flight400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python-flight-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python-flight400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow400.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva400.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet400.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma400.install
dev/tasks/linux-packages/apache-arrow/debian/patches/series
dev/tasks/linux-packages/apache-arrow/debian/plasma-store-server.install
dev/tasks/linux-packages/apache-arrow/debian/rules
dev/tasks/linux-packages/apache-arrow/debian/source/format
dev/tasks/linux-packages/apache-arrow/debian/watch
dev/tasks/requirements*.txt
dev/tasks/conda-recipes/*
pax_global_header
MANIFEST.in
__init__.pxd
Expand All @@ -109,8 +33,6 @@ requirements.txt
.gitattributes
rust-toolchain
benchmarks/queries/q*.sql
python/rust-toolchain
python/requirements*.txt
**/testdata/*
benchmarks/queries/*
benchmarks/expected-plans/*
Expand Down
Loading
Loading