-
Notifications
You must be signed in to change notification settings - Fork 84
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refactor wheel upload job to a separate job running on GH ephemeral r…
…unner (#4877) To run the upload part in a separate upload job on GH ephemeral runners, we need: 1. Specific artifact name for each binary, so the upload job could find the correct one. 2. Create a new GHA `setup-binary-upload` to: 1. Download the artifacts from GitHub 2. Running `pkg-helpers` is needed to figure out the correct S3 bucket and path to upload to. 3. Create a new GHA reusable workflow `_binary_upload` to upload the artifacts to S3. 1. Run on GH ephemeral runner `ubuntu-22.04`. 2. Only this job has access to the credential, the build job doesn't have that privilege anymore. A small caveat here is that the upload job will depend on the build job with all its configuration matrix, so it can only be run after all build configurations finish successfully, not when individual builds finish. The PR is quite big, so I will do a similar follow up for conda build after this using the same `_binary_upload` reusable workflow.
- Loading branch information
Showing
6 changed files
with
226 additions
and
109 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,9 +7,9 @@ inputs: | |
| description: If set to any value, don't use sudo to clean the workspace | ||
| required: false | ||
| type: string | ||
| default: "" | ||
| default: '' | ||
| ref: | ||
| description: Works as stated in actions/checkout, but the default value is recursive | ||
| description: Works as stated in actions/checkout | ||
| required: false | ||
| type: string | ||
| default: nightly | ||
|
|
@@ -19,15 +19,27 @@ inputs: | |
| type: string | ||
| default: recursive | ||
| setup-miniconda: | ||
| description: Works as stated in actions/checkout, but the default value is recursive | ||
| description: Set to true if setup-miniconda is needed | ||
| required: false | ||
| type: boolean | ||
| default: false | ||
| python-version: | ||
| description: Works as stated in actions/checkout, but the default value is recursive | ||
| description: The target Python version | ||
| required: true | ||
| type: string | ||
| cuda-version: | ||
| description: The target CUDA version | ||
| required: true | ||
| type: string | ||
| arch: | ||
| description: The target ARCH | ||
| required: true | ||
| type: string | ||
| upload-to-base-bucket: | ||
| description: One of the parameter used by pkg-helpers | ||
| required: false | ||
| type: boolean | ||
| default: false | ||
| default: no | ||
|
|
||
| runs: | ||
| using: composite | ||
|
|
@@ -62,11 +74,13 @@ runs: | |
| shell: bash | ||
| env: | ||
| PYTHON_VERSION: ${{ inputs.python-version }} | ||
| CU_VERSION: ${{ inputs.cuda-version }} | ||
| ARCH: ${{ inputs.arch }} | ||
| run: | | ||
| set -euxo pipefail | ||
| # Set artifact name here since github actions doesn't have string manipulation tools | ||
| # and "/" is not allowed in artifact names | ||
| echo "ARTIFACT_NAME=${REPOSITORY/\//_}_${REF}_${PYTHON_VERSION}" >> "${GITHUB_ENV}" | ||
| echo "ARTIFACT_NAME=${REPOSITORY/\//_}_${REF}_${PYTHON_VERSION}_${CU_VERSION}_${ARCH}" >> "${GITHUB_ENV}" | ||
| - name: Setup miniconda (for pytorch_pkg_helpers) | ||
| if: ${{ inputs.setup-miniconda == 'true' }} | ||
| uses: conda-incubator/[email protected] | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| name: Set up binary upload jobs | ||
|
|
||
| description: Setup a GitHub ephemeral runner to upload binary wheel and conda artifacts | ||
|
|
||
| inputs: | ||
| repository: | ||
| description: The repository name, i.e. pytorch/vision | ||
| required: true | ||
| type: string | ||
| ref: | ||
| description: Part of the artifact name | ||
| required: false | ||
| type: string | ||
| default: '' | ||
| python-version: | ||
| description: Part of the artifact name | ||
| required: true | ||
| type: string | ||
| cuda-version: | ||
| description: Part of the artifact name | ||
| required: true | ||
| type: string | ||
| arch: | ||
| description: Part of the artifact name | ||
| required: true | ||
| type: string | ||
| upload-to-base-bucket: | ||
| description: One of the parameter used by pkg-helpers | ||
| required: false | ||
| type: boolean | ||
| default: no | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: '3.11' | ||
| cache: pip | ||
|
|
||
| - name: Set the artifact name | ||
| shell: bash | ||
| env: | ||
| REPOSITORY: ${{ inputs.repository }} | ||
| REF: ${{ inputs.ref }} | ||
| PYTHON_VERSION: ${{ inputs.python-version }} | ||
| CU_VERSION: ${{ inputs.cuda-version }} | ||
| ARCH: ${{ inputs.arch }} | ||
| run: | | ||
| set -ex | ||
| # Set artifact name here since github actions doesn't have string manipulation tools | ||
| # and "/" is not allowed in artifact names | ||
| echo "ARTIFACT_NAME=${REPOSITORY/\//_}_${REF}_${PYTHON_VERSION}_${CU_VERSION}_${ARCH}" >> "${GITHUB_ENV}" | ||
| - name: Generate env variables from pytorch_pkg_helpers | ||
| shell: bash | ||
| env: | ||
| REPOSITORY: ${{ inputs.repository }} | ||
| REF: ${{ inputs.ref }} | ||
| PYTHON_VERSION: ${{ inputs.python-version }} | ||
| CU_VERSION: ${{ inputs.cuda-version }} | ||
| ARCH: ${{ inputs.arch }} | ||
| run: | | ||
| set -ex | ||
| python -m pip install tools/pkg-helpers | ||
| BUILD_ENV_FILE="${RUNNER_TEMP}/build_env_${GITHUB_RUN_ID}" | ||
| python -m pytorch_pkg_helpers > "${BUILD_ENV_FILE}" | ||
| cat "${BUILD_ENV_FILE}" | ||
| echo "BUILD_ENV_FILE=${BUILD_ENV_FILE}" >> "${GITHUB_ENV}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| name: upload | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| repository: | ||
| description: 'Repository to checkout, defaults to ""' | ||
| default: '' | ||
| type: string | ||
| ref: | ||
| description: 'Reference to checkout, defaults to "nightly"' | ||
| default: 'nightly' | ||
| type: string | ||
| build-matrix: | ||
| description: "Build matrix to utilize" | ||
| default: '' | ||
| type: string | ||
| architecture: | ||
| description: Architecture to build for x86_64 for default Linux, or aarch64 for Linux aarch64 builds | ||
| required: false | ||
| type: string | ||
| default: '' | ||
| trigger-event: | ||
| description: "Trigger Event in caller that determines whether or not to upload" | ||
| type: string | ||
| default: '' | ||
|
|
||
| jobs: | ||
| upload: | ||
| runs-on: ubuntu-22.04 | ||
| environment: ${{(inputs.trigger-event == 'push' && (startsWith(github.event.ref, 'refs/heads/nightly') || startsWith(github.event.ref, 'refs/tags/v'))) && 'pytorchbot-env' || ''}} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{ fromJSON(inputs.build-matrix) }} | ||
| timeout-minutes: 30 | ||
| name: ${{ matrix.build_name }} | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
|
|
||
| # For pytorch_pkg_helpers which we need to run to generate the artifact name and target S3 buckets | ||
| - uses: ./.github/actions/setup-binary-upload | ||
| with: | ||
| repository: ${{ inputs.repository }} | ||
| ref: ${{ inputs.ref }} | ||
| python-version: ${{ matrix.python_version }} | ||
| cuda-version: ${{ matrix.desired_cuda }} | ||
| arch: ${{ inputs.architecture }} | ||
| upload-to-base-bucket: ${{ matrix.upload_to_base_bucket }} | ||
|
|
||
| - uses: ./.github/actions/set-channel | ||
|
|
||
| - name: Download the artifact | ||
| uses: actions/download-artifact@v3 | ||
| with: | ||
| name: ${{ env.ARTIFACT_NAME }} | ||
| path: ${{ inputs.repository }}/dist/ | ||
|
|
||
| - name: Configure aws credentials (pytorch account) | ||
| if: ${{ inputs.trigger-event == 'push' && startsWith(github.event.ref, 'refs/heads/nightly') }} | ||
| uses: aws-actions/configure-aws-credentials@v3 | ||
| with: | ||
| role-to-assume: arn:aws:iam::749337293305:role/gha_workflow_nightly_build_wheels | ||
| aws-region: us-east-1 | ||
|
|
||
| - name: Configure aws credentials (pytorch account) | ||
| if: ${{ env.CHANNEL == 'test' && startsWith(github.event.ref, 'refs/tags/v') }} | ||
| uses: aws-actions/configure-aws-credentials@v3 | ||
| with: | ||
| role-to-assume: arn:aws:iam::749337293305:role/gha_workflow_test_build_wheels | ||
| aws-region: us-east-1 | ||
|
|
||
| - name: Nightly or release RC | ||
| if: ${{ (inputs.trigger-event == 'push' && startsWith(github.event.ref, 'refs/heads/nightly')) || (env.CHANNEL == 'test' && startsWith(github.event.ref, 'refs/tags/')) }} | ||
| shell: bash | ||
| run: | | ||
| set -ex | ||
| echo "NIGHTLY_OR_TEST=1" >> "${GITHUB_ENV}" | ||
| - name: Upload package to pytorch.org | ||
| shell: bash | ||
| working-directory: ${{ inputs.repository }} | ||
| run: | | ||
| set -ex | ||
| # shellcheck disable=SC1090 | ||
| source "${BUILD_ENV_FILE}" | ||
| pip install awscli==1.32.18 | ||
| AWS_CMD="aws s3 cp --dryrun" | ||
| if [[ "${NIGHTLY_OR_TEST:-0}" == "1" ]]; then | ||
| AWS_CMD="aws s3 cp" | ||
| fi | ||
| for pkg in dist/*; do | ||
| ${AWS_CMD} "$pkg" "${PYTORCH_S3_BUCKET_PATH}" --acl public-read | ||
| done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.