Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update release-process.md for pypi|cran|docker instructions #548

Merged
merged 2 commits into from
Aug 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 6 additions & 18 deletions release-process.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ The release manager role in Spark means you are responsible for a few different
- [Finalizing and posting a release](#finalize-the-release)
- [Upload to Apache release directory](#upload-to-apache-release-directory)
- [Upload to PyPI](#upload-to-pypi)
- [Publish to CRAN](#publish-to-cran)
- [Remove RC artifacts from repositories](#remove-rc-artifacts-from-repositories)
- [Remove old releases from Mirror Network](#remove-old-releases-from-mirror-network)
- [Update the Apache Spark<sup>TM</sup> repository](#update-the-apache-spark-repository)
Expand Down Expand Up @@ -208,7 +207,6 @@ that looks something like `[VOTE][RESULT] ...`.
<h2 id="finalize-the-release">Finalize the release</h2>

Note that `dev/create-release/do-release-docker.sh` script (`finalize` step ) automates most of the following steps **except** for:
- Publish to CRAN
- Update the configuration of Algolia Crawler
- Remove old releases from Mirror Network
- Update the rest of the Spark website
Expand Down Expand Up @@ -259,21 +257,16 @@ You'll need your own PyPI account. If you do not have a PyPI account that has ac
The artifacts can be uploaded using <a href="https://pypi.org/project/twine/">twine</a>. Just run:

```
twine upload --repository-url https://upload.pypi.org/legacy/ pyspark-{version}.tar.gz pyspark-{version}.tar.gz.asc
twine upload -u __token__ -p $PYPI_API_TOKEN \
--repository-url https://upload.pypi.org/legacy/ \
"pyspark-$PYSPARK_VERSION.tar.gz" \
"pyspark-$PYSPARK_VERSION.tar.gz.asc"
```

Adjusting the command for the files that match the new release. If for some reason the twine upload
is incorrect (e.g. http failure or other issue), you can rename the artifact to
`pyspark-version.post0.tar.gz`, delete the old artifact from PyPI and re-upload.


<p align="right"><a href="#top">Return to top</a></p>
<h3 id="publish-to-cran">Publish to CRAN</h3>

Publishing to CRAN is done using <a href="https://cran.r-project.org/submit.html">this form</a>.
Since it requires further manual steps, please also contact the <a href="mailto:[email protected]">PMC</a>.


<p align="right"><a href="#top">Return to top</a></p>
<h3 id="remove-rc-artifacts-from-repositories">Remove RC artifacts from repositories</h3>

Expand Down Expand Up @@ -436,13 +429,8 @@ $ git log v1.1.1 --grep "$expr" --shortstat --oneline | grep -B 1 -e "[3-9][0-9]
<p align="right"><a href="#top">Return to top</a></p>
<h3 id="create-and-upload-spark-docker-images">Create and upload Spark Docker Images</h3>

The apache/spark-docker provides dockerfiles and Github Action for Spark Docker images publish.
1. Upload Spark Dockerfiles to apache/spark-docker repository, please refer to [link](https://github.com/apache/spark-docker/pull/33).
2. Publish Spark Docker Images:
1. Enter [publish page](https://github.com/apache/spark-docker/actions/workflows/publish.yml).
2. Click "Run workflow".
3. Select "The Spark version of Spark image", click "Publish the image or not", select "apache" as target registry.
4. Click "Run workflow" button to publish the image to Apache dockerhub.
The apache/spark-docker provides Dockerfiles and GitHub Action for Spark Docker images published, please follow the [instructions](https://github.com/apache/spark-docker?tab=readme-ov-file#create-a-new-version) to create and upload the docker images.


<p align="right"><a href="#top">Return to top</a></p>
<h3 id="create-an-announcement">Create an announcement</h3>
Expand Down
26 changes: 5 additions & 21 deletions site/release-process.html
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,6 @@ <h1>Preparing Spark releases</h1>
<ul>
<li><a href="#upload-to-apache-release-directory">Upload to Apache release directory</a></li>
<li><a href="#upload-to-pypi">Upload to PyPI</a></li>
<li><a href="#publish-to-cran">Publish to CRAN</a></li>
<li><a href="#remove-rc-artifacts-from-repositories">Remove RC artifacts from repositories</a></li>
<li><a href="#remove-old-releases-from-mirror-network">Remove old releases from Mirror Network</a></li>
<li><a href="#update-the-apache-spark-repository">Update the Apache Spark<sup>TM</sup> repository</a></li>
Expand Down Expand Up @@ -371,7 +370,6 @@ <h2 id="finalize-the-release">Finalize the release</h2>

<p>Note that <code class="language-plaintext highlighter-rouge">dev/create-release/do-release-docker.sh</code> script (<code class="language-plaintext highlighter-rouge">finalize</code> step ) automates most of the following steps <strong>except</strong> for:</p>
<ul>
<li>Publish to CRAN</li>
<li>Update the configuration of Algolia Crawler</li>
<li>Remove old releases from Mirror Network</li>
<li>Update the rest of the Spark website</li>
Expand Down Expand Up @@ -419,19 +417,16 @@ <h3 id="upload-to-pypi">Upload to PyPI</h3>

<p>The artifacts can be uploaded using <a href="https://pypi.org/project/twine/">twine</a>. Just run:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>twine upload --repository-url https://upload.pypi.org/legacy/ pyspark-{version}.tar.gz pyspark-{version}.tar.gz.asc
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>twine upload -u __token__ -p $PYPI_API_TOKEN \
--repository-url https://upload.pypi.org/legacy/ \
"pyspark-$PYSPARK_VERSION.tar.gz" \
"pyspark-$PYSPARK_VERSION.tar.gz.asc"
</code></pre></div></div>

<p>Adjusting the command for the files that match the new release. If for some reason the twine upload
is incorrect (e.g. http failure or other issue), you can rename the artifact to
<code class="language-plaintext highlighter-rouge">pyspark-version.post0.tar.gz</code>, delete the old artifact from PyPI and re-upload.</p>

<p align="right"><a href="#top">Return to top</a></p>
<h3 id="publish-to-cran">Publish to CRAN</h3>

<p>Publishing to CRAN is done using <a href="https://cran.r-project.org/submit.html">this form</a>.
Since it requires further manual steps, please also contact the <a href="mailto:[email protected]">PMC</a>.</p>

<p align="right"><a href="#top">Return to top</a></p>
<h3 id="remove-rc-artifacts-from-repositories">Remove RC artifacts from repositories</h3>

Expand Down Expand Up @@ -587,18 +582,7 @@ <h4 id="update-the-rest-of-the-spark-website">Update the rest of the Spark websi
<p align="right"><a href="#top">Return to top</a></p>
<h3 id="create-and-upload-spark-docker-images">Create and upload Spark Docker Images</h3>

<p>The apache/spark-docker provides dockerfiles and Github Action for Spark Docker images publish.</p>
<ol>
<li>Upload Spark Dockerfiles to apache/spark-docker repository, please refer to <a href="https://github.com/apache/spark-docker/pull/33">link</a>.</li>
<li>Publish Spark Docker Images:
<ol>
<li>Enter <a href="https://github.com/apache/spark-docker/actions/workflows/publish.yml">publish page</a>.</li>
<li>Click &#8220;Run workflow&#8221;.</li>
<li>Select &#8220;The Spark version of Spark image&#8221;, click &#8220;Publish the image or not&#8221;, select &#8220;apache&#8221; as target registry.</li>
<li>Click &#8220;Run workflow&#8221; button to publish the image to Apache dockerhub.</li>
</ol>
</li>
</ol>
<p>The apache/spark-docker provides Dockerfiles and GitHub Action for Spark Docker images published, please follow the <a href="https://github.com/apache/spark-docker?tab=readme-ov-file#create-a-new-version">instructions</a> to create and upload the docker images.</p>

<p align="right"><a href="#top">Return to top</a></p>
<h3 id="create-an-announcement">Create an announcement</h3>
Expand Down
Loading