[EXPERIMENT] Using Github Artifact Store to store build artifacts #1810

VenkateshJaya · 2025-02-03T19:05:05Z

Description

Exploring the usage of Github Artifact store in place of downloads.rapids.ai, see rapidsai/ops#2982

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2025-02-03T19:05:10Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: VenkateshJaya <[email protected]>

copy-pr-bot · 2025-02-03T22:05:38Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

VenkateshJaya · 2025-02-03T22:17:03Z

/ok to test

VenkateshJaya · 2025-02-03T22:27:19Z

/ok to test

VenkateshJaya · 2025-02-06T00:44:42Z

/ok to test

VenkateshJaya · 2025-02-11T00:14:30Z

/ok to test

VenkateshJaya · 2025-02-11T00:23:17Z

/ok to test

VenkateshJaya · 2025-02-11T00:33:16Z

/ok to test

xref rapidsai/build-planning#147 Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#1811

Fixes `build_type` input not being used in `test` workflows. See rapidsai#1811 (comment).

This completes the migration to NVKS runners now that all libraries have been tested and rapidsai/shared-workflows#273 has been merged. xref: rapidsai/build-infra#184 Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) URL: rapidsai#1816

This change helps completely insulate rmm (and transitively) the rest of RAPIDS from fmt and spdlog as dependencies, thereby solving a large number of issues around ABI stability, symbol visibility, package clobbering, and more. See rapidsai/build-planning#104 for more information. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Matthew Murray (https://github.com/Matt711) - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) URL: rapidsai#1808

Addresses rapidsai#1808 (comment) Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) URL: rapidsai#1820

…#1809) A pair of doxygen comments in `host_memory_resource` referenced `device_memory_resource` when it didn't mean to, very likely a simple copy/paste issue. rapidsai#1794 Authors: - Nicholas Sielicki (https://github.com/aws-nslick) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#1809

This is a cleanup PR. I found that we were extraneously including `<thrust/optional.h>` in the pool memory resource (also `thrust::optional` is deprecated in favor of `cuda::std::optional` in the upcoming major release of CCCL). I did a pass with IWYU to see what else could be fixed. IWYU could only really analyze our tests, since RMM is header-only. There are a lot of false positives/negatives, so I don't think it is appropriate to automate IWYU in our CI. However, this felt valuable enough to open a refactoring PR. I also updated some deprecated GTest code which was using `TYPED_TEST_CASE` instead of `TYPED_TEST_SUITE` and replaced some uses of `::value` with the corresponding `_v` STL features. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: rapidsai#1821

This change helps completely insulate rmm (and transitively) the rest of RAPIDS from fmt and spdlog as dependencies, thereby solving a large number of issues around ABI stability, symbol visibility, package clobbering, and more. See rapidsai/build-planning#104 for more information. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Matthew Murray (https://github.com/Matt711) - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) URL: rapidsai#1808

Addresses rapidsai#1808 (comment) Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) URL: rapidsai#1820

VenkateshJaya · 2025-02-12T18:08:28Z

/ok to test

Issue: rapidsai/build-planning#22 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#1824

Summary: - We use the [multi-output cache](https://rattler.build/latest/multiple_output_cache/) to avoid double-compiling. The `build` environment compiles things, the individual outputs call `cmake --install` - We make use of the built-in `git` functions for grabbing the short-SHA (https://rattler.build/latest/experimental_features/#git-functions) - We use `load_from_file` to pull in metadata from the corresponding `pyproject.toml` (https://rattler.build/latest/experimental_features/#load_from_filefile_path) - Relatively "simple" `*_build.sh` scripts are inlined into `recipe.yaml` instead of existing as separate files - We use `--no-build-id` to allow `sccache` to look in a predictable place, see: https://rattler.build/latest/tips_and_tricks/#using-sccache-or-ccache-with-rattler-build - Depending on whether `rapids-is-release-build`, we include either `rapidsai` (release) or `rapidsai-nightly` (non-release) in the channel listing - Channels must be specified at the command-line - This uses https://github.com/rapidsai/gha-tools/blob/main/tools/rapids-rattler-channel-string to generate an array of channels - We remove the `build_cache` directory after building so it doesn't get packaged up with the other artifacts and uploaded to S3 xref: rapidsai/build-planning#47 Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#1796

Issue: rapidsai/build-planning#22 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#1828

bdice · 2025-02-19T21:42:47Z

ci/build_wheel_cpp.sh

@@ -4,24 +4,21 @@
 set -euo pipefail

 package_dir="python/librmm"
+wheel_dir=${RAPIDS_WHEEL_DIR:-"dist"}


Maybe set RAPIDS_WHEEL_DIR to some fixed path like /tmp/wheelhouse so we have a known name.

We can define RAPIDS_WHEEL_OUTPUT_DIR in the CI images. Match the naming we use for conda.

bdice · 2025-02-19T21:45:40Z

ci/build_wheel_python.sh

@@ -14,7 +15,7 @@ rapids-generate-version > ./VERSION
 pushd "${package_dir}"

 RAPIDS_PY_CUDA_SUFFIX=$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")
-CPP_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="rmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp /tmp/librmm_dist)
+CPP_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="rmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github cpp)


Please verify if GitHub artifacts can be downloaded from the CLI without a GitHub token.

It looks like no, this is not possible.

There is discussion here: actions/upload-artifact#51

And a possible solution: https://nightly.link/

Let's figure out how rapids-download-wheels-from-github behaves for users with no token / in a non-interactive shell. We need to be sure that it prompts for auth or errors cleanly.

VenkateshJaya closed this Feb 3, 2025

VenkateshJaya deleted the test-github-artifacts branch February 3, 2025 19:44

VenkateshJaya reopened this Feb 3, 2025

Added Upload Artifact in CPP build

bf8755b

Signed-off-by: VenkateshJaya <[email protected]>

VenkateshJaya force-pushed the test-github-artifacts branch from 1fb0985 to bf8755b Compare February 3, 2025 22:05

commented out unchanged steps in CI

ce0ac2b

VenkateshJaya added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change DO NOT MERGE Hold off on merging; see PR for details and removed DO NOT MERGE Hold off on merging; see PR for details labels Feb 3, 2025

VenkateshJaya added 2 commits February 3, 2025 15:22

commented out test steps from pr-builder

199df33

added telemetry setup to pr-builder

6e71cba

added github download for cpp tests

7cc8e2c

github-actions bot added the ci label Feb 5, 2025

VenkateshJaya added 2 commits February 5, 2025 17:27

added downloads from github

51ab1ad

fixed error in pr workflow

c38201f

made changes to wheel build and test steps to use github artifacts

86e101d

added newline at the end

11cd801

VenkateshJaya added 2 commits February 10, 2025 17:30

fixed unused import in shell

42b248f

temporarily removing devcontainer steps

dd6d22f

gforsyth and others added 2 commits February 11, 2025 12:44

Add build_type to workflow inputs (rapidsai#1811)

85dc3b0

xref rapidsai/build-planning#147 Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#1811

Use build_type input (rapidsai#1812)

2c58346

Fixes `build_type` input not being used in `test` workflows. See rapidsai#1811 (comment).

bdice and others added 6 commits February 11, 2025 14:35

Remove unnecessary index (rapidsai#1820)

449a7dc

Addresses rapidsai#1808 (comment) Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) URL: rapidsai#1820

made path changes to wheel builds

76d71fa

github-actions bot added CMake Python Related to RMM Python API conda cpp Pertains to C++ code labels Feb 11, 2025

vyasr and others added 3 commits February 11, 2025 15:28

Remove unnecessary index (rapidsai#1820)

576a774

Addresses rapidsai#1808 (comment) Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) URL: rapidsai#1820

Merge branch 'branch-25.04' into test-github-artifacts

b36d5b5

github-actions bot removed CMake Python Related to RMM Python API conda cpp Pertains to C++ code labels Feb 11, 2025

VenkateshJaya and others added 5 commits February 18, 2025 10:04

removed comments

336bf21

Create Conda CI test env in one step (rapidsai#1824)

8d96296

Issue: rapidsai/build-planning#22 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#1824

Update Changelog [skip ci]

757d239

Consolidate more Conda solves in CI (rapidsai#1828)

1327a21

Issue: rapidsai/build-planning#22 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#1828

github-actions bot added Python Related to RMM Python API conda labels Feb 19, 2025

Merge branch 'branch-25.04' into test-github-artifacts

6f3ea56

github-actions bot removed Python Related to RMM Python API conda labels Feb 19, 2025

bdice reviewed Feb 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EXPERIMENT] Using Github Artifact Store to store build artifacts #1810

[EXPERIMENT] Using Github Artifact Store to store build artifacts #1810

VenkateshJaya commented Feb 3, 2025 •

edited

Loading

copy-pr-bot bot commented Feb 3, 2025

copy-pr-bot bot commented Feb 3, 2025

VenkateshJaya commented Feb 3, 2025

VenkateshJaya commented Feb 3, 2025

VenkateshJaya commented Feb 6, 2025

VenkateshJaya commented Feb 11, 2025

VenkateshJaya commented Feb 11, 2025

VenkateshJaya commented Feb 11, 2025

VenkateshJaya commented Feb 12, 2025

bdice Feb 19, 2025 •

edited

Loading

bdice Feb 19, 2025

bdice Feb 19, 2025

bdice Feb 19, 2025

bdice Feb 19, 2025 •

edited

Loading

[EXPERIMENT] Using Github Artifact Store to store build artifacts #1810

Are you sure you want to change the base?

[EXPERIMENT] Using Github Artifact Store to store build artifacts #1810

Conversation

VenkateshJaya commented Feb 3, 2025 • edited Loading

Description

Checklist

copy-pr-bot bot commented Feb 3, 2025

copy-pr-bot bot commented Feb 3, 2025

VenkateshJaya commented Feb 3, 2025

VenkateshJaya commented Feb 3, 2025

VenkateshJaya commented Feb 6, 2025

VenkateshJaya commented Feb 11, 2025

VenkateshJaya commented Feb 11, 2025

VenkateshJaya commented Feb 11, 2025

VenkateshJaya commented Feb 12, 2025

bdice Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

bdice Feb 19, 2025

Choose a reason for hiding this comment

bdice Feb 19, 2025

Choose a reason for hiding this comment

bdice Feb 19, 2025

Choose a reason for hiding this comment

bdice Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

VenkateshJaya commented Feb 3, 2025 •

edited

Loading

bdice Feb 19, 2025 •

edited

Loading

bdice Feb 19, 2025 •

edited

Loading