Skip to content

Commit 44f1147

Browse files
andygroveandygrove
authored
Update release scripts now that DataFusion is TLP (#10235)
1 parent 4ec3d51 commit 44f1147

8 files changed

+61
-150
lines changed

‎dev/release/README.md

Lines changed: 35 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -48,8 +48,8 @@ patch release:
4848
- Created a personal access token in GitHub for changelog automation script.
4949
- Github PAT should be created with `repo` access
5050
- Make sure your signing key is added to the following files in SVN:
51-
- https://dist.apache.org/repos/dist/dev/arrow/KEYS
52-
- https://dist.apache.org/repos/dist/release/arrow/KEYS
51+
- https://dist.apache.org/repos/dist/dev/datafusion/KEYS
52+
- https://dist.apache.org/repos/dist/release/datafusion/KEYS
5353

5454
### How to add signing key
5555

@@ -58,8 +58,8 @@ See instructions at https://infra.apache.org/release-signing.html#generate for g
5858
Committers can add signing keys in Subversion client with their ASF account. e.g.:
5959

6060
```bash
61-
$ svn co https://dist.apache.org/repos/dist/dev/arrow
62-
$ cd arrow
61+
$ svn co https://dist.apache.org/repos/dist/dev/datafusion
62+
$ cd datafusion
6363
$ editor KEYS
6464
$ svn ci KEYS
6565
```
@@ -128,7 +128,7 @@ release.
128128

129129
See [#9697](https://github.com/apache/datafusion/pull/9697) for an example.
130130

131-
Here are the commands that could be used to prepare the `5.1.0` release:
131+
Here are the commands that could be used to prepare the `38.0.0` release:
132132

133133
### Update Version
134134

@@ -139,10 +139,10 @@ git fetch apache
139139
git checkout apache/main
140140
```
141141

142-
Update datafusion version in `datafusion/Cargo.toml` to `5.1.0`:
142+
Update datafusion version in `datafusion/Cargo.toml` to `38.0.0`:
143143

144144
```
145-
./dev/update_datafusion_versions.py 5.1.0
145+
./dev/update_datafusion_versions.py 38.0.0
146146
```
147147

148148
Lastly commit the version change:
@@ -167,7 +167,7 @@ Pick numbers in sequential order, with `0` for `rc0`, `1` for `rc1`, etc.
167167
While the official release artifacts are signed tarballs and zip files, we also
168168
tag the commit it was created for convenience and code archaeology.
169169

170-
Using a string such as `5.1.0` as the `<version>`, create and push the tag by running these commands:
170+
Using a string such as `38.0.0` as the `<version>`, create and push the tag by running these commands:
171171

172172
```shell
173173
git fetch apache
@@ -181,29 +181,29 @@ git push apache <version>
181181
Run `create-tarball.sh` with the `<version>` tag and `<rc>` and you found in previous steps:
182182

183183
```shell
184-
GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 5.1.0 0
184+
GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 38.0.0 0
185185
```
186186

187187
The `create-tarball.sh` script
188188

189-
1. creates and uploads all release candidate artifacts to the [arrow
190-
dev](https://dist.apache.org/repos/dist/dev/arrow) location on the
189+
1. creates and uploads all release candidate artifacts to the [datafusion
190+
dev](https://dist.apache.org/repos/dist/dev/datafusion) location on the
191191
apache distribution svn server
192192

193193
2. provide you an email template to
194-
send to dev@arrow.apache.org for release voting.
194+
send to dev@datafusion.apache.org for release voting.
195195

196196
### Vote on Release Candidate artifacts
197197

198-
Send the email output from the script to dev@arrow.apache.org. The email should look like
198+
Send the email output from the script to dev@datafusion.apache.org. The email should look like
199199

200200
```
201-
To: dev@arrow.apache.org
202-
Subject: [VOTE][DataFusion] Release Apache DataFusion 5.1.0 RC0
201+
To: dev@datafusion.apache.org
202+
Subject: [VOTE] Release Apache DataFusion 38.0.0 RC1
203203
204204
Hi,
205205
206-
I would like to propose a release of Apache DataFusion version 5.1.0.
206+
I would like to propose a release of Apache DataFusion version 38.0.0.
207207
208208
This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1]
209209
The proposed release artifacts and signatures are hosted at [2].
@@ -214,16 +214,16 @@ and vote on the release.
214214
215215
The vote will be open for at least 72 hours.
216216
217-
[ ] +1 Release this as Apache DataFusion 5.1.0
217+
[ ] +1 Release this as Apache DataFusion 38.0.0
218218
[ ] +0
219-
[ ] -1 Do not release this as Apache DataFusion 5.1.0 because...
219+
[ ] -1 Do not release this as Apache DataFusion 38.0.0 because...
220220
221221
Here is my vote:
222222
223223
+1
224224
225225
[1]: https://github.com/apache/datafusion/tree/a5dd428f57e62db20a945e8b1895de91405958c4
226-
[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-datafusion-5.1.0
226+
[2]: https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-38.0.0
227227
[3]: https://github.com/apache/datafusion/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md
228228
```
229229

@@ -234,7 +234,7 @@ For the release to become "official" it needs at least three PMC members to vote
234234
The `dev/release/verify-release-candidate.sh` is a script in this repository that can assist in the verification process. Run it like:
235235

236236
```
237-
./dev/release/verify-release-candidate.sh 5.1.0 0
237+
./dev/release/verify-release-candidate.sh 38.0.0 0
238238
```
239239

240240
#### If the release is not approved
@@ -249,11 +249,11 @@ NOTE: steps in this section can only be done by PMC members.
249249
### After the release is approved
250250

251251
Move artifacts to the release location in SVN, e.g.
252-
https://dist.apache.org/repos/dist/release/datafusion/datafusion-5.1.0/, using
252+
https://dist.apache.org/repos/dist/release/datafusion/datafusion-38.0.0/, using
253253
the `release-tarball.sh` script:
254254

255255
```shell
256-
./dev/release/release-tarball.sh 5.1.0 0
256+
./dev/release/release-tarball.sh 38.0.0 0
257257
```
258258

259259
Congratulations! The release is now official!
@@ -263,9 +263,9 @@ Congratulations! The release is now official!
263263
Tag the same release candidate commit with the final release tag
264264

265265
```
266-
git co apache/5.1.0-rc0
267-
git tag 5.1.0
268-
git push apache 5.1.0
266+
git co apache/38.0.0-rc0
267+
git tag 38.0.0
268+
git push apache 38.0.0
269269
```
270270

271271
### Publish on Crates.io
@@ -300,7 +300,7 @@ of the following crates:
300300
Download and unpack the official release tarball
301301

302302
Verify that the Cargo.toml in the tarball contains the correct version
303-
(e.g. `version = "5.1.0"`) and then publish the crates by running the script `release-crates.sh`
303+
(e.g. `version = "38.0.0"`) and then publish the crates by running the script `release-crates.sh`
304304
in a directory extracted from the source tarball that was voted on. Note that this script doesn't
305305
work if run in a Git repo.
306306

@@ -413,10 +413,9 @@ https://crates.io/crates/datafusion-substrait/28.0.0
413413

414414
### Add the release to Apache Reporter
415415

416-
Add the release to https://reporter.apache.org/addrelease.html?arrow with a version name prefixed with `RS-DATAFUSION-`,
417-
for example `RS-DATAFUSION-14.0.0`.
416+
Add the release to https://reporter.apache.org/addrelease.html?datafusion using the version number e.g. 38.0.0.
418417

419-
The release information is used to generate a template for a board report (see example
418+
The release information is used to generate a template for a board report (see example from Apache Arrow project
420419
[here](https://github.com/apache/arrow/pull/14357)).
421420

422421
### Delete old RCs and Releases
@@ -431,13 +430,13 @@ Release candidates should be deleted once the release is published.
431430
Get a list of DataFusion release candidates:
432431

433432
```bash
434-
svn ls https://dist.apache.org/repos/dist/dev/arrow | grep datafusion
433+
svn ls https://dist.apache.org/repos/dist/dev/datafusion
435434
```
436435

437436
Delete a release candidate:
438437

439438
```bash
440-
svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-7.1.0-rc1/
439+
svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-38.0.0-rc1/
441440
```
442441

443442
#### Deleting old releases from `release` svn
@@ -447,35 +446,25 @@ Only the latest release should be available. Delete old releases after publishin
447446
Get a list of DataFusion releases:
448447

449448
```bash
450-
svn ls https://dist.apache.org/repos/dist/release/arrow | grep datafusion
449+
svn ls https://dist.apache.org/repos/dist/release/datafusion
451450
```
452451

453452
Delete a release:
454453

455454
```bash
456-
svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-7.0.0
455+
svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-37.0.0
457456
```
458457

459-
### Publish the User Guide to the Arrow Site
460-
461-
- Run the `build.sh` in the `docs` directory from the release tarball.
462-
- Clone the [arrow-site](https://github.com/apache/arrow-site) repository
463-
- Checkout the `asf-site` branch
464-
- Copy content from `docs/build/html/*` to the `datafusion` directory in arrow-site
465-
- Create a PR against the `asf-site` branch ([example](https://github.com/apache/arrow-site/pull/237))
466-
- Once the PR is merged, the content will be published to https://datafusion.apache.org/ by GitHub Pages (this
467-
can take some time).
468-
469458
### Optional: Write a blog post announcing the release
470459

471-
We typically crowdsource release announcements by collaborating on a Google document, usually starting
460+
We typically crowd source release announcements by collaborating on a Google document, usually starting
472461
with a copy of the previous release announcement.
473462

474463
Run the following commands to get the number of commits and number of unique contributors for inclusion in the blog post.
475464

476465
```bash
477-
git log --pretty=oneline 10.0.0..11.0.0 datafusion datafusion-cli datafusion-examples | wc -l
478-
git shortlog -sn 10.0.0..11.0.0 datafusion datafusion-cli datafusion-examples | wc -l
466+
git log --pretty=oneline 37.0.0..38.0.0 datafusion datafusion-cli datafusion-examples | wc -l
467+
git shortlog -sn 37.0.0..38.0.0 datafusion datafusion-cli datafusion-examples | wc -l
479468
```
480469

481470
Once there is consensus on the contents of the post, create a PR to add a blog post to the

‎dev/release/create-tarball.sh

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,9 @@
2121
# Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/create-tarball.sh
2222

2323
# This script creates a signed tarball in
24-
# dev/dist/apache-arrow-datafusion-<version>-<sha>.tar.gz and uploads it to
25-
# the "dev" area of the dist.apache.arrow repository and prepares an
26-
# email for sending to the dev@arrow.apache.org list for a formal
24+
# dev/dist/apache-datafusion-<version>-<sha>.tar.gz and uploads it to
25+
# the "dev" area of the dist.apache.datafusion repository and prepares an
26+
# email for sending to the dev@datafusion.apache.org list for a formal
2727
# vote.
2828
#
2929
# See release/README.md for full release instructions
@@ -65,21 +65,21 @@ tag="${version}-rc${rc}"
6565
echo "Attempting to create ${tarball} from tag ${tag}"
6666
release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag})
6767

68-
release=apache-arrow-datafusion-${version}
68+
release=apache-datafusion-${version}
6969
distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc}
7070
tarname=${release}.tar.gz
7171
tarball=${distdir}/${tarname}
72-
url="https://dist.apache.org/repos/dist/dev/arrow/${release}-rc${rc}"
72+
url="https://dist.apache.org/repos/dist/dev/datafusion/${release}-rc${rc}"
7373

7474
if [ -z "$release_hash" ]; then
7575
echo "Cannot continue: unknown git tag: ${tag}"
7676
fi
7777

78-
echo "Draft email for dev@arrow.apache.org mailing list"
78+
echo "Draft email for dev@datafusion.apache.org mailing list"
7979
echo ""
8080
echo "---------------------------------------------------------"
8181
cat <<MAIL
82-
To: dev@arrow.apache.org
82+
To: dev@datafusion.apache.org
8383
Subject: [VOTE][RUST][DataFusion] Release Apache DataFusion ${version} RC${rc}
8484
Hi,
8585
@@ -124,12 +124,12 @@ echo "Signing tarball and creating checksums"
124124
gpg --armor --output ${tarball}.asc --detach-sig ${tarball}
125125
# create signing with relative path of tarball
126126
# so that they can be verified with a command such as
127-
# shasum --check apache-arrow-datafusion-4.1.0-rc2.tar.gz.sha512
127+
# shasum --check apache-datafusion-38.0.0-rc1.tar.gz.sha512
128128
(cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256
129129
(cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512
130130

131131

132-
echo "Uploading to apache dist/dev to ${url}"
133-
svn co --depth=empty https://dist.apache.org/repos/dist/dev/arrow ${SOURCE_TOP_DIR}/dev/dist
132+
echo "Uploading to datafusion dist/dev to ${url}"
133+
svn co --depth=empty https://dist.apache.org/repos/dist/dev/datafusion ${SOURCE_TOP_DIR}/dev/dist
134134
svn add ${distdir}
135135
svn ci -m "Apache DataFusion ${version} ${rc}" ${distdir}

‎dev/release/publish_homebrew.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,8 @@ else # Fallback
3939
num_processing_units=1
4040
fi
4141

42-
url="https://www.apache.org/dyn/closer.lua?path=arrow/arrow-datafusion-${version}/apache-arrow-datafusion-${version}.tar.gz"
43-
sha256="$(curl https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-${version}/apache-arrow-datafusion-${version}.tar.gz.sha256 | cut -d' ' -f1)"
42+
url="https://www.apache.org/dyn/closer.lua?path=datafusion/datafusion-${version}/apache-datafusion-${version}.tar.gz"
43+
sha256="$(curl https://dist.apache.org/repos/dist/release/datafusion/datafusion-${version}/apache-datafusion-${version}.tar.gz.sha256 | cut -d' ' -f1)"
4444

4545
pushd "$(brew --repository homebrew/core)"
4646

@@ -52,7 +52,7 @@ fi
5252
echo "Updating working copy"
5353
git fetch --all --prune --tags --force -j$num_processing_units
5454

55-
branch=apache-arrow-datafusion-${version}
55+
branch=apache-datafusion-${version}
5656
echo "Creating branch: ${branch}"
5757
git branch -D ${branch} || :
5858
git checkout -b ${branch} origin/master

‎dev/release/rat_exclude_files.txt

Lines changed: 0 additions & 78 deletions
Original file line numberDiff line numberDiff line change
@@ -15,84 +15,8 @@ ci/etc/*.patch
1515
ci/vcpkg/*.patch
1616
CHANGELOG.md
1717
datafusion/CHANGELOG.md
18-
python/CHANGELOG.md
19-
conbench/benchmarks.json
20-
conbench/requirements.txt
21-
conbench/requirements-test.txt
22-
conbench/.flake8
23-
conbench/.isort.cfg
2418
dev/requirements*.txt
25-
dev/archery/MANIFEST.in
26-
dev/archery/requirements*.txt
27-
dev/archery/archery/tests/fixtures/*
28-
dev/archery/archery/crossbow/tests/fixtures/*
2919
dev/release/rat_exclude_files.txt
30-
dev/tasks/homebrew-formulae/apache-arrow.rb
31-
dev/tasks/linux-packages/apache-arrow-apt-source/debian/apache-arrow-apt-source.install
32-
dev/tasks/linux-packages/apache-arrow-apt-source/debian/compat
33-
dev/tasks/linux-packages/apache-arrow-apt-source/debian/control
34-
dev/tasks/linux-packages/apache-arrow-apt-source/debian/rules
35-
dev/tasks/linux-packages/apache-arrow-apt-source/debian/source/format
36-
dev/tasks/linux-packages/apache-arrow/debian/compat
37-
dev/tasks/linux-packages/apache-arrow/debian/control.in
38-
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-arrow-1.0.install
39-
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-arrow-cuda-1.0.install
40-
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-arrow-dataset-1.0.install
41-
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-gandiva-1.0.install
42-
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-parquet-1.0.install
43-
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-plasma-1.0.install
44-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dev.install
45-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-dev.install
46-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-doc.doc-base
47-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-doc.install
48-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-doc.links
49-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib400.install
50-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda-dev.install
51-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda-glib-dev.install
52-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda-glib400.install
53-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda400.install
54-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-dev.install
55-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-dev.install
56-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-doc.doc-base
57-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-doc.install
58-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-doc.links
59-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib400.install
60-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset400.install
61-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-flight-dev.install
62-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-flight400.install
63-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python-dev.install
64-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python-flight-dev.install
65-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python-flight400.install
66-
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python400.install
67-
dev/tasks/linux-packages/apache-arrow/debian/libarrow400.install
68-
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-dev.install
69-
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-dev.install
70-
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-doc.doc-base
71-
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-doc.install
72-
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-doc.links
73-
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib400.install
74-
dev/tasks/linux-packages/apache-arrow/debian/libgandiva400.install
75-
dev/tasks/linux-packages/apache-arrow/debian/libparquet-dev.install
76-
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-dev.install
77-
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-doc.doc-base
78-
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-doc.install
79-
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-doc.links
80-
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib400.install
81-
dev/tasks/linux-packages/apache-arrow/debian/libparquet400.install
82-
dev/tasks/linux-packages/apache-arrow/debian/libplasma-dev.install
83-
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-dev.install
84-
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-doc.doc-base
85-
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-doc.install
86-
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-doc.links
87-
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib400.install
88-
dev/tasks/linux-packages/apache-arrow/debian/libplasma400.install
89-
dev/tasks/linux-packages/apache-arrow/debian/patches/series
90-
dev/tasks/linux-packages/apache-arrow/debian/plasma-store-server.install
91-
dev/tasks/linux-packages/apache-arrow/debian/rules
92-
dev/tasks/linux-packages/apache-arrow/debian/source/format
93-
dev/tasks/linux-packages/apache-arrow/debian/watch
94-
dev/tasks/requirements*.txt
95-
dev/tasks/conda-recipes/*
9620
pax_global_header
9721
MANIFEST.in
9822
__init__.pxd
@@ -109,8 +33,6 @@ requirements.txt
10933
.gitattributes
11034
rust-toolchain
11135
benchmarks/queries/q*.sql
112-
python/rust-toolchain
113-
python/requirements*.txt
11436
**/testdata/*
11537
benchmarks/queries/*
11638
benchmarks/expected-plans/*

0 commit comments

Comments
 (0)