From 5836d08c8e45212e35cfa11222b51c9802266d63 Mon Sep 17 00:00:00 2001 From: Ray Douglass Date: Wed, 11 Dec 2024 13:10:31 -0500 Subject: [PATCH 001/129] Update Changelog [skip ci] --- CHANGELOG.md | 329 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 329 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a75b2a95a4..97f7afb33a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,332 @@ +# cudf 24.12.00 (11 Dec 2024) + +## 🚨 Breaking Changes + +- Fix reading Parquet string cols when `nrows` and `input_pass_limit` > 0 ([#17321](https://github.com/rapidsai/cudf/pull/17321)) [@mhaseeb123](https://github.com/mhaseeb123) +- prefer wheel-provided libcudf.so in load_library(), use RTLD_LOCAL ([#17316](https://github.com/rapidsai/cudf/pull/17316)) [@jameslamb](https://github.com/jameslamb) +- Deprecate single component extraction methods in libcudf ([#17221](https://github.com/rapidsai/cudf/pull/17221)) [@Matt711](https://github.com/Matt711) +- Move detail header floating_conversion.hpp to detail subdirectory ([#17209](https://github.com/rapidsai/cudf/pull/17209)) [@davidwendt](https://github.com/davidwendt) +- Refactor Dask cuDF legacy code ([#17205](https://github.com/rapidsai/cudf/pull/17205)) [@rjzamora](https://github.com/rjzamora) +- Make HostMemoryBuffer call into the DefaultHostMemoryAllocator ([#17204](https://github.com/rapidsai/cudf/pull/17204)) [@revans2](https://github.com/revans2) +- Remove java reservation ([#17189](https://github.com/rapidsai/cudf/pull/17189)) [@revans2](https://github.com/revans2) +- Separate evaluation logic from `IR` objects in cudf-polars ([#17175](https://github.com/rapidsai/cudf/pull/17175)) [@rjzamora](https://github.com/rjzamora) +- Upgrade to polars 1.11 in cudf-polars ([#17154](https://github.com/rapidsai/cudf/pull/17154)) [@wence-](https://github.com/wence-) +- Remove the additional host register calls initially intended for performance improvement on Grace Hopper ([#17092](https://github.com/rapidsai/cudf/pull/17092)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Correctly set `is_device_accesible` when creating `host_span`s from other container/span types ([#17079](https://github.com/rapidsai/cudf/pull/17079)) [@vuule](https://github.com/vuule) +- Unify treatment of `Expr` and `IR` nodes in cudf-polars DSL ([#17016](https://github.com/rapidsai/cudf/pull/17016)) [@wence-](https://github.com/wence-) +- Deprecate support for directly accessing logger ([#16964](https://github.com/rapidsai/cudf/pull/16964)) [@vyasr](https://github.com/vyasr) +- Made cudftestutil header-only and removed GTest dependency ([#16839](https://github.com/rapidsai/cudf/pull/16839)) [@lamarrr](https://github.com/lamarrr) + +## 🐛 Bug Fixes + +- Turn off cudf.pandas 3rd party integrations tests for 24.12 ([#17500](https://github.com/rapidsai/cudf/pull/17500)) [@Matt711](https://github.com/Matt711) +- Ignore errors when testing glibc versions ([#17389](https://github.com/rapidsai/cudf/pull/17389)) [@vyasr](https://github.com/vyasr) +- Adapt to KvikIO API change in the compatibility mode ([#17377](https://github.com/rapidsai/cudf/pull/17377)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Support pivot with index or column arguments as lists ([#17373](https://github.com/rapidsai/cudf/pull/17373)) [@mroeschke](https://github.com/mroeschke) +- Deselect failing polars tests ([#17362](https://github.com/rapidsai/cudf/pull/17362)) [@pentschev](https://github.com/pentschev) +- Fix integer overflow in compiled binaryop ([#17354](https://github.com/rapidsai/cudf/pull/17354)) [@wence-](https://github.com/wence-) +- Update cmake to 3.28.6 in JNI Dockerfile ([#17342](https://github.com/rapidsai/cudf/pull/17342)) [@jlowe](https://github.com/jlowe) +- fix library-loading issues in editable installs ([#17338](https://github.com/rapidsai/cudf/pull/17338)) [@jameslamb](https://github.com/jameslamb) +- Bug fix: restrict lines=True to JSON format in Kafka read_gdf method ([#17333](https://github.com/rapidsai/cudf/pull/17333)) [@a-hirota](https://github.com/a-hirota) +- Fix various issues with `replace` API and add support in `datetime` and `timedelta` columns ([#17331](https://github.com/rapidsai/cudf/pull/17331)) [@galipremsagar](https://github.com/galipremsagar) +- Do not exclude nanoarrow and flatbuffers from installation if statically linked ([#17322](https://github.com/rapidsai/cudf/pull/17322)) [@hyperbolic2346](https://github.com/hyperbolic2346) +- Fix reading Parquet string cols when `nrows` and `input_pass_limit` > 0 ([#17321](https://github.com/rapidsai/cudf/pull/17321)) [@mhaseeb123](https://github.com/mhaseeb123) +- Remove another reference to `FindcuFile` ([#17315](https://github.com/rapidsai/cudf/pull/17315)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA) +- Fix reading of single-row unterminated CSV files ([#17305](https://github.com/rapidsai/cudf/pull/17305)) [@vuule](https://github.com/vuule) +- Fixed lifetime issue in ast transform tests ([#17292](https://github.com/rapidsai/cudf/pull/17292)) [@lamarrr](https://github.com/lamarrr) +- Switch to using `TaskSpec` ([#17285](https://github.com/rapidsai/cudf/pull/17285)) [@galipremsagar](https://github.com/galipremsagar) +- Fix data_type ctor call in JSON_TEST ([#17273](https://github.com/rapidsai/cudf/pull/17273)) [@davidwendt](https://github.com/davidwendt) +- Expose delimiter character in JSON reader options to JSON reader APIs ([#17266](https://github.com/rapidsai/cudf/pull/17266)) [@shrshi](https://github.com/shrshi) +- Fix extract-datetime deprecation warning in ndsh benchmark ([#17254](https://github.com/rapidsai/cudf/pull/17254)) [@davidwendt](https://github.com/davidwendt) +- Disallow cuda-python 12.6.1 and 11.8.4 ([#17253](https://github.com/rapidsai/cudf/pull/17253)) [@bdice](https://github.com/bdice) +- Wrap custom iterator result ([#17251](https://github.com/rapidsai/cudf/pull/17251)) [@galipremsagar](https://github.com/galipremsagar) +- Fix binop with LHS numpy datetimelike scalar ([#17226](https://github.com/rapidsai/cudf/pull/17226)) [@mroeschke](https://github.com/mroeschke) +- Fix `Dataframe.__setitem__` slow-downs ([#17222](https://github.com/rapidsai/cudf/pull/17222)) [@galipremsagar](https://github.com/galipremsagar) +- Fix groupby.get_group with length-1 tuple with list-like grouper ([#17216](https://github.com/rapidsai/cudf/pull/17216)) [@mroeschke](https://github.com/mroeschke) +- Fix discoverability of submodules inside `pd.util` ([#17215](https://github.com/rapidsai/cudf/pull/17215)) [@galipremsagar](https://github.com/galipremsagar) +- Fix `Schema.Builder` does not propagate precision value to `Builder` instance ([#17214](https://github.com/rapidsai/cudf/pull/17214)) [@ttnghia](https://github.com/ttnghia) +- Mark column chunks in a PQ reader `pass` as large strings when the cumulative `offsets` exceeds the large strings threshold. ([#17207](https://github.com/rapidsai/cudf/pull/17207)) [@mhaseeb123](https://github.com/mhaseeb123) +- [BUG] Replace `repo_token` with `github_token` in Auto Assign PR GHA ([#17203](https://github.com/rapidsai/cudf/pull/17203)) [@Matt711](https://github.com/Matt711) +- Remove unsanitized nulls from input strings columns in reduction gtests ([#17202](https://github.com/rapidsai/cudf/pull/17202)) [@davidwendt](https://github.com/davidwendt) +- Fix ``to_parquet`` append behavior with global metadata file ([#17198](https://github.com/rapidsai/cudf/pull/17198)) [@rjzamora](https://github.com/rjzamora) +- Check `num_children() == 0` in `Column.from_column_view` ([#17193](https://github.com/rapidsai/cudf/pull/17193)) [@cwharris](https://github.com/cwharris) +- Fix host-to-device copy missing sync in strings/duration convert ([#17149](https://github.com/rapidsai/cudf/pull/17149)) [@davidwendt](https://github.com/davidwendt) +- Add JNI Support for Multi-line Delimiters and Include Test ([#17139](https://github.com/rapidsai/cudf/pull/17139)) [@SurajAralihalli](https://github.com/SurajAralihalli) +- Ignore loud dask warnings about legacy dataframe implementation ([#17137](https://github.com/rapidsai/cudf/pull/17137)) [@galipremsagar](https://github.com/galipremsagar) +- Fix the GDS read/write segfault/bus error when the cuFile policy is set to GDS or ALWAYS ([#17122](https://github.com/rapidsai/cudf/pull/17122)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Fix `DataFrame._from_arrays` and introduce validations ([#17112](https://github.com/rapidsai/cudf/pull/17112)) [@galipremsagar](https://github.com/galipremsagar) +- [Bug] Fix Arrow-FS parquet reader for larger files ([#17099](https://github.com/rapidsai/cudf/pull/17099)) [@rjzamora](https://github.com/rjzamora) +- Fix bug in recovering invalid lines in JSONL inputs ([#17098](https://github.com/rapidsai/cudf/pull/17098)) [@shrshi](https://github.com/shrshi) +- Reenable huge pages for arrow host copying ([#17097](https://github.com/rapidsai/cudf/pull/17097)) [@vyasr](https://github.com/vyasr) +- Correctly set `is_device_accesible` when creating `host_span`s from other container/span types ([#17079](https://github.com/rapidsai/cudf/pull/17079)) [@vuule](https://github.com/vuule) +- Fix ORC reader when using `device_read_async` while the destination device buffers are not ready ([#17074](https://github.com/rapidsai/cudf/pull/17074)) [@ttnghia](https://github.com/ttnghia) +- Fix regex handling of fixed quantifier with 0 range ([#17067](https://github.com/rapidsai/cudf/pull/17067)) [@davidwendt](https://github.com/davidwendt) +- Limit the number of keys to calculate column sizes and page starts in PQ reader to 1B ([#17059](https://github.com/rapidsai/cudf/pull/17059)) [@mhaseeb123](https://github.com/mhaseeb123) +- Adding assertion to check for regular JSON inputs of size greater than `INT_MAX` bytes ([#17057](https://github.com/rapidsai/cudf/pull/17057)) [@shrshi](https://github.com/shrshi) +- bug fix: use `self.ck_consumer` in `poll` method of kafka.py to align with `__init__` ([#17044](https://github.com/rapidsai/cudf/pull/17044)) [@a-hirota](https://github.com/a-hirota) +- Disable kvikio remote I/O to avoid openssl dependencies in JNI build ([#17026](https://github.com/rapidsai/cudf/pull/17026)) [@pxLi](https://github.com/pxLi) +- Fix `host_span` constructor to correctly copy `is_device_accessible` ([#17020](https://github.com/rapidsai/cudf/pull/17020)) [@vuule](https://github.com/vuule) +- Add pinning for pyarrow in wheels ([#17018](https://github.com/rapidsai/cudf/pull/17018)) [@vyasr](https://github.com/vyasr) +- Use std::optional for host types ([#17015](https://github.com/rapidsai/cudf/pull/17015)) [@robertmaynard](https://github.com/robertmaynard) +- Fix write_json to handle empty string column ([#16995](https://github.com/rapidsai/cudf/pull/16995)) [@karthikeyann](https://github.com/karthikeyann) +- Restore export of nvcomp outside of wheel builds ([#16988](https://github.com/rapidsai/cudf/pull/16988)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA) +- Allow melt(var_name=) to be a falsy label ([#16981](https://github.com/rapidsai/cudf/pull/16981)) [@mroeschke](https://github.com/mroeschke) +- Fix astype from tz-aware type to tz-aware type ([#16980](https://github.com/rapidsai/cudf/pull/16980)) [@mroeschke](https://github.com/mroeschke) +- Use `libcudf` wheel from PR rather than nightly for `polars-polars` CI test job ([#16975](https://github.com/rapidsai/cudf/pull/16975)) [@brandon-b-miller](https://github.com/brandon-b-miller) +- Fix order-preservation in pandas-compat unsorted groupby ([#16942](https://github.com/rapidsai/cudf/pull/16942)) [@wence-](https://github.com/wence-) +- Fix cudf::strings::findall error with empty input ([#16928](https://github.com/rapidsai/cudf/pull/16928)) [@davidwendt](https://github.com/davidwendt) +- Fix JsonLargeReaderTest.MultiBatch use of LIBCUDF_JSON_BATCH_SIZE env var ([#16927](https://github.com/rapidsai/cudf/pull/16927)) [@davidwendt](https://github.com/davidwendt) +- Parse newline as whitespace character while tokenizing JSONL inputs with non-newline delimiter ([#16923](https://github.com/rapidsai/cudf/pull/16923)) [@shrshi](https://github.com/shrshi) +- Respect groupby.nunique(dropna=False) ([#16921](https://github.com/rapidsai/cudf/pull/16921)) [@mroeschke](https://github.com/mroeschke) +- Update all rmm imports to use pylibrmm/librmm ([#16913](https://github.com/rapidsai/cudf/pull/16913)) [@Matt711](https://github.com/Matt711) +- Fix order-preservation in cudf-polars groupby ([#16907](https://github.com/rapidsai/cudf/pull/16907)) [@wence-](https://github.com/wence-) +- Add a shortcut for when the input clusters are all empty for the tdigest merge ([#16897](https://github.com/rapidsai/cudf/pull/16897)) [@jihoonson](https://github.com/jihoonson) +- Properly handle the mapped and registered regions in `memory_mapped_source` ([#16865](https://github.com/rapidsai/cudf/pull/16865)) [@vuule](https://github.com/vuule) +- Fix performance regression for generate_character_ngrams ([#16849](https://github.com/rapidsai/cudf/pull/16849)) [@davidwendt](https://github.com/davidwendt) +- Fix regex parsing logic handling of nested quantifiers ([#16798](https://github.com/rapidsai/cudf/pull/16798)) [@davidwendt](https://github.com/davidwendt) +- Compute whole column variance using numerically stable approach ([#16448](https://github.com/rapidsai/cudf/pull/16448)) [@wence-](https://github.com/wence-) + +## 📖 Documentation + +- Add documentation for low memory readers ([#17314](https://github.com/rapidsai/cudf/pull/17314)) [@btepera](https://github.com/btepera) +- Fix the example in documentation for `get_dremel_data()` ([#17242](https://github.com/rapidsai/cudf/pull/17242)) [@mhaseeb123](https://github.com/mhaseeb123) +- Fix some documentation rendering for pylibcudf ([#17217](https://github.com/rapidsai/cudf/pull/17217)) [@mroeschke](https://github.com/mroeschke) +- Move detail header floating_conversion.hpp to detail subdirectory ([#17209](https://github.com/rapidsai/cudf/pull/17209)) [@davidwendt](https://github.com/davidwendt) +- Add TokenizeVocabulary to api docs ([#17208](https://github.com/rapidsai/cudf/pull/17208)) [@davidwendt](https://github.com/davidwendt) +- Add jaccard_index to generated cuDF docs ([#17199](https://github.com/rapidsai/cudf/pull/17199)) [@davidwendt](https://github.com/davidwendt) +- [no ci] Add empty-columns section to the libcudf developer guide ([#17183](https://github.com/rapidsai/cudf/pull/17183)) [@davidwendt](https://github.com/davidwendt) +- Add 2-cpp approvers text to contributing guide [no ci] ([#17182](https://github.com/rapidsai/cudf/pull/17182)) [@davidwendt](https://github.com/davidwendt) +- Changing developer guide int_64_t to int64_t ([#17130](https://github.com/rapidsai/cudf/pull/17130)) [@hyperbolic2346](https://github.com/hyperbolic2346) +- docs: change 'CSV' to 'csv' in python/custreamz/README.md to match kafka.py ([#17041](https://github.com/rapidsai/cudf/pull/17041)) [@a-hirota](https://github.com/a-hirota) +- [DOC] Document limitation using `cudf.pandas` proxy arrays ([#16955](https://github.com/rapidsai/cudf/pull/16955)) [@Matt711](https://github.com/Matt711) +- [DOC] Document environment variable for failing on fallback in `cudf.pandas` ([#16932](https://github.com/rapidsai/cudf/pull/16932)) [@Matt711](https://github.com/Matt711) + +## 🚀 New Features + +- Add version config ([#17312](https://github.com/rapidsai/cudf/pull/17312)) [@vyasr](https://github.com/vyasr) +- Java JNI for Multiple contains ([#17281](https://github.com/rapidsai/cudf/pull/17281)) [@res-life](https://github.com/res-life) +- Add `cudf::calendrical_month_sequence` to pylibcudf ([#17277](https://github.com/rapidsai/cudf/pull/17277)) [@Matt711](https://github.com/Matt711) +- Raise errors on specific types of fallback in `cudf.pandas` ([#17268](https://github.com/rapidsai/cudf/pull/17268)) [@Matt711](https://github.com/Matt711) +- Add `catboost` to the third-party integration tests ([#17267](https://github.com/rapidsai/cudf/pull/17267)) [@Matt711](https://github.com/Matt711) +- Add type stubs for pylibcudf ([#17258](https://github.com/rapidsai/cudf/pull/17258)) [@wence-](https://github.com/wence-) +- Use pylibcudf contiguous split APIs in cudf python ([#17246](https://github.com/rapidsai/cudf/pull/17246)) [@Matt711](https://github.com/Matt711) +- Upgrade nvcomp to 4.1.0.6 ([#17201](https://github.com/rapidsai/cudf/pull/17201)) [@bdice](https://github.com/bdice) +- Added Arrow Interop Benchmarks ([#17194](https://github.com/rapidsai/cudf/pull/17194)) [@lamarrr](https://github.com/lamarrr) +- Rewrite Java API `Table.readJSON` to return the output from libcudf `read_json` directly ([#17180](https://github.com/rapidsai/cudf/pull/17180)) [@ttnghia](https://github.com/ttnghia) +- Support storing `precision` of decimal types in `Schema` class ([#17176](https://github.com/rapidsai/cudf/pull/17176)) [@ttnghia](https://github.com/ttnghia) +- Migrate CSV writer to pylibcudf ([#17163](https://github.com/rapidsai/cudf/pull/17163)) [@Matt711](https://github.com/Matt711) +- Add compute_shared_memory_aggs used by shared memory groupby ([#17162](https://github.com/rapidsai/cudf/pull/17162)) [@PointKernel](https://github.com/PointKernel) +- Added ast tree to simplify expression lifetime management ([#17156](https://github.com/rapidsai/cudf/pull/17156)) [@lamarrr](https://github.com/lamarrr) +- Add compute_mapping_indices used by shared memory groupby ([#17147](https://github.com/rapidsai/cudf/pull/17147)) [@PointKernel](https://github.com/PointKernel) +- Add remaining datetime APIs to pylibcudf ([#17143](https://github.com/rapidsai/cudf/pull/17143)) [@Matt711](https://github.com/Matt711) +- Added strings AST vs BINARY_OP benchmarks ([#17128](https://github.com/rapidsai/cudf/pull/17128)) [@lamarrr](https://github.com/lamarrr) +- Use `libcudf_exception_handler` throughout `pylibcudf.libcudf` ([#17109](https://github.com/rapidsai/cudf/pull/17109)) [@brandon-b-miller](https://github.com/brandon-b-miller) +- Include timezone file path in error message ([#17102](https://github.com/rapidsai/cudf/pull/17102)) [@bdice](https://github.com/bdice) +- Migrate NVText Byte Pair Encoding APIs to pylibcudf ([#17101](https://github.com/rapidsai/cudf/pull/17101)) [@Matt711](https://github.com/Matt711) +- Migrate NVText Tokenizing APIs to pylibcudf ([#17100](https://github.com/rapidsai/cudf/pull/17100)) [@Matt711](https://github.com/Matt711) +- Migrate NVtext subword tokenizing APIs to pylibcudf ([#17096](https://github.com/rapidsai/cudf/pull/17096)) [@Matt711](https://github.com/Matt711) +- Migrate NVText Stemming APIs to pylibcudf ([#17085](https://github.com/rapidsai/cudf/pull/17085)) [@Matt711](https://github.com/Matt711) +- Migrate NVText Replacing APIs to pylibcudf ([#17084](https://github.com/rapidsai/cudf/pull/17084)) [@Matt711](https://github.com/Matt711) +- Add IWYU to CI ([#17078](https://github.com/rapidsai/cudf/pull/17078)) [@vyasr](https://github.com/vyasr) +- `cudf-polars` string/numeric casting ([#17076](https://github.com/rapidsai/cudf/pull/17076)) [@brandon-b-miller](https://github.com/brandon-b-miller) +- Migrate NVText Normalizing APIs to Pylibcudf ([#17072](https://github.com/rapidsai/cudf/pull/17072)) [@Matt711](https://github.com/Matt711) +- Migrate remaining nvtext NGrams APIs to pylibcudf ([#17070](https://github.com/rapidsai/cudf/pull/17070)) [@Matt711](https://github.com/Matt711) +- Add profilers to CUDA 12 conda devcontainers ([#17066](https://github.com/rapidsai/cudf/pull/17066)) [@vyasr](https://github.com/vyasr) +- Add conda recipe for cudf-polars ([#17037](https://github.com/rapidsai/cudf/pull/17037)) [@bdice](https://github.com/bdice) +- Implement batch construction for strings columns ([#17035](https://github.com/rapidsai/cudf/pull/17035)) [@ttnghia](https://github.com/ttnghia) +- Add device aggregators used by shared memory groupby ([#17031](https://github.com/rapidsai/cudf/pull/17031)) [@PointKernel](https://github.com/PointKernel) +- Add optional column_order in JSON reader ([#17029](https://github.com/rapidsai/cudf/pull/17029)) [@karthikeyann](https://github.com/karthikeyann) +- Migrate Min Hashing APIs to pylibcudf ([#17021](https://github.com/rapidsai/cudf/pull/17021)) [@Matt711](https://github.com/Matt711) +- Reorganize `cudf_polars` expression code ([#17014](https://github.com/rapidsai/cudf/pull/17014)) [@brandon-b-miller](https://github.com/brandon-b-miller) +- Migrate nvtext jaccard API to pylibcudf ([#17007](https://github.com/rapidsai/cudf/pull/17007)) [@Matt711](https://github.com/Matt711) +- Migrate nvtext generate_ngrams APIs to pylibcudf ([#17006](https://github.com/rapidsai/cudf/pull/17006)) [@Matt711](https://github.com/Matt711) +- Control whether a file data source memory-maps the file with an environment variable ([#17004](https://github.com/rapidsai/cudf/pull/17004)) [@vuule](https://github.com/vuule) +- Switched BINARY_OP Benchmarks from GoogleBench to NVBench ([#16963](https://github.com/rapidsai/cudf/pull/16963)) [@lamarrr](https://github.com/lamarrr) +- [FEA] Report all unsupported operations for a query in cudf.polars ([#16960](https://github.com/rapidsai/cudf/pull/16960)) [@Matt711](https://github.com/Matt711) +- [FEA] Migrate nvtext/edit_distance APIs to pylibcudf ([#16957](https://github.com/rapidsai/cudf/pull/16957)) [@Matt711](https://github.com/Matt711) +- Switched AST benchmarks from GoogleBench to NVBench ([#16952](https://github.com/rapidsai/cudf/pull/16952)) [@lamarrr](https://github.com/lamarrr) +- Extend `device_scalar` to optionally use pinned bounce buffer ([#16947](https://github.com/rapidsai/cudf/pull/16947)) [@vuule](https://github.com/vuule) +- Implement `cudf-polars` chunked parquet reading ([#16944](https://github.com/rapidsai/cudf/pull/16944)) [@brandon-b-miller](https://github.com/brandon-b-miller) +- Expose streams in public round APIs ([#16925](https://github.com/rapidsai/cudf/pull/16925)) [@Matt711](https://github.com/Matt711) +- add telemetry setup to test ([#16924](https://github.com/rapidsai/cudf/pull/16924)) [@msarahan](https://github.com/msarahan) +- Add cudf::strings::contains_multiple ([#16900](https://github.com/rapidsai/cudf/pull/16900)) [@davidwendt](https://github.com/davidwendt) +- Made cudftestutil header-only and removed GTest dependency ([#16839](https://github.com/rapidsai/cudf/pull/16839)) [@lamarrr](https://github.com/lamarrr) +- Add an example to demonstrate multithreaded `read_parquet` pipelines ([#16828](https://github.com/rapidsai/cudf/pull/16828)) [@mhaseeb123](https://github.com/mhaseeb123) +- Implement `extract_datetime_component` in `libcudf`/`pylibcudf` ([#16776](https://github.com/rapidsai/cudf/pull/16776)) [@brandon-b-miller](https://github.com/brandon-b-miller) +- Add cudf::strings::find_re API ([#16742](https://github.com/rapidsai/cudf/pull/16742)) [@davidwendt](https://github.com/davidwendt) +- Migrate hashing operations to `pylibcudf` ([#15418](https://github.com/rapidsai/cudf/pull/15418)) [@brandon-b-miller](https://github.com/brandon-b-miller) + +## 🛠️ Improvements + +- Simplify serialization protocols ([#17552](https://github.com/rapidsai/cudf/pull/17552)) [@vyasr](https://github.com/vyasr) +- Add `pynvml` as a dependency for `dask-cudf` ([#17386](https://github.com/rapidsai/cudf/pull/17386)) [@pentschev](https://github.com/pentschev) +- Enable unified memory by default in `cudf_polars` ([#17375](https://github.com/rapidsai/cudf/pull/17375)) [@galipremsagar](https://github.com/galipremsagar) +- Support polars 1.14 ([#17355](https://github.com/rapidsai/cudf/pull/17355)) [@wence-](https://github.com/wence-) +- Remove cudf._lib.quantiles in favor of inlining pylibcudf ([#17347](https://github.com/rapidsai/cudf/pull/17347)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.labeling in favor of inlining pylibcudf ([#17346](https://github.com/rapidsai/cudf/pull/17346)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.hash in favor of inlining pylibcudf ([#17345](https://github.com/rapidsai/cudf/pull/17345)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.concat in favor of inlining pylibcudf ([#17344](https://github.com/rapidsai/cudf/pull/17344)) [@mroeschke](https://github.com/mroeschke) +- Extract ``GPUEngine`` config options at translation time ([#17339](https://github.com/rapidsai/cudf/pull/17339)) [@rjzamora](https://github.com/rjzamora) +- Update java datetime APIs to match CUDF. ([#17329](https://github.com/rapidsai/cudf/pull/17329)) [@revans2](https://github.com/revans2) +- Move strings url_decode benchmarks to nvbench ([#17328](https://github.com/rapidsai/cudf/pull/17328)) [@davidwendt](https://github.com/davidwendt) +- Move strings translate benchmarks to nvbench ([#17325](https://github.com/rapidsai/cudf/pull/17325)) [@davidwendt](https://github.com/davidwendt) +- Writing compressed output using JSON writer ([#17323](https://github.com/rapidsai/cudf/pull/17323)) [@shrshi](https://github.com/shrshi) +- Test the full matrix for polars and dask wheels on nightlies ([#17320](https://github.com/rapidsai/cudf/pull/17320)) [@vyasr](https://github.com/vyasr) +- Remove cudf._lib.avro in favor of inlining pylicudf ([#17319](https://github.com/rapidsai/cudf/pull/17319)) [@mroeschke](https://github.com/mroeschke) +- Move cudf._lib.unary to cudf.core._internals ([#17318](https://github.com/rapidsai/cudf/pull/17318)) [@mroeschke](https://github.com/mroeschke) +- prefer wheel-provided libcudf.so in load_library(), use RTLD_LOCAL ([#17316](https://github.com/rapidsai/cudf/pull/17316)) [@jameslamb](https://github.com/jameslamb) +- Clean up misc, unneeded pylibcudf.libcudf in cudf._lib ([#17309](https://github.com/rapidsai/cudf/pull/17309)) [@mroeschke](https://github.com/mroeschke) +- Exclude nanoarrow and flatbuffers from installation ([#17308](https://github.com/rapidsai/cudf/pull/17308)) [@vyasr](https://github.com/vyasr) +- Update CI jobs to include Polars in nightlies and improve IWYU ([#17306](https://github.com/rapidsai/cudf/pull/17306)) [@vyasr](https://github.com/vyasr) +- Move strings repeat benchmarks to nvbench ([#17304](https://github.com/rapidsai/cudf/pull/17304)) [@davidwendt](https://github.com/davidwendt) +- Fix synchronization bug in bool parquet mukernels ([#17302](https://github.com/rapidsai/cudf/pull/17302)) [@pmattione-nvidia](https://github.com/pmattione-nvidia) +- Move strings replace benchmarks to nvbench ([#17301](https://github.com/rapidsai/cudf/pull/17301)) [@davidwendt](https://github.com/davidwendt) +- Support polars 1.13 ([#17299](https://github.com/rapidsai/cudf/pull/17299)) [@wence-](https://github.com/wence-) +- Replace FindcuFile with upstream FindCUDAToolkit support ([#17298](https://github.com/rapidsai/cudf/pull/17298)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA) +- Expose stream-ordering in public transpose API ([#17294](https://github.com/rapidsai/cudf/pull/17294)) [@shrshi](https://github.com/shrshi) +- Replace workaround of JNI build with CUDF_KVIKIO_REMOTE_IO=OFF ([#17293](https://github.com/rapidsai/cudf/pull/17293)) [@pxLi](https://github.com/pxLi) +- cmake option: `CUDF_KVIKIO_REMOTE_IO` ([#17291](https://github.com/rapidsai/cudf/pull/17291)) [@madsbk](https://github.com/madsbk) +- Use more pylibcudf Python enums in cudf._lib ([#17288](https://github.com/rapidsai/cudf/pull/17288)) [@mroeschke](https://github.com/mroeschke) +- Use pylibcudf enums in cudf Python quantile ([#17287](https://github.com/rapidsai/cudf/pull/17287)) [@mroeschke](https://github.com/mroeschke) +- enforce wheel size limits, README formatting in CI ([#17284](https://github.com/rapidsai/cudf/pull/17284)) [@jameslamb](https://github.com/jameslamb) +- Use numba-cuda<0.0.18 ([#17280](https://github.com/rapidsai/cudf/pull/17280)) [@gmarkall](https://github.com/gmarkall) +- Add compute_column_expression to pylibcudf for transform.compute_column ([#17279](https://github.com/rapidsai/cudf/pull/17279)) [@mroeschke](https://github.com/mroeschke) +- Optimize distinct inner join to use set `find` instead of `retrieve` ([#17278](https://github.com/rapidsai/cudf/pull/17278)) [@PointKernel](https://github.com/PointKernel) +- remove WheelHelpers.cmake ([#17276](https://github.com/rapidsai/cudf/pull/17276)) [@jameslamb](https://github.com/jameslamb) +- Plumb pylibcudf datetime APIs through cudf python ([#17275](https://github.com/rapidsai/cudf/pull/17275)) [@Matt711](https://github.com/Matt711) +- Follow up making Python tests more deterministic ([#17272](https://github.com/rapidsai/cudf/pull/17272)) [@mroeschke](https://github.com/mroeschke) +- Use pylibcudf.search APIs in cudf python ([#17271](https://github.com/rapidsai/cudf/pull/17271)) [@Matt711](https://github.com/Matt711) +- Use `pylibcudf.strings.convert.convert_integers.is_integer` in cudf python ([#17270](https://github.com/rapidsai/cudf/pull/17270)) [@Matt711](https://github.com/Matt711) +- Move strings filter benchmarks to nvbench ([#17269](https://github.com/rapidsai/cudf/pull/17269)) [@davidwendt](https://github.com/davidwendt) +- Make constructor of DeviceMemoryBufferView public ([#17265](https://github.com/rapidsai/cudf/pull/17265)) [@liurenjie1024](https://github.com/liurenjie1024) +- Put a ceiling on cuda-python ([#17264](https://github.com/rapidsai/cudf/pull/17264)) [@jameslamb](https://github.com/jameslamb) +- Always prefer `device_read`s and `device_write`s when kvikIO is enabled ([#17260](https://github.com/rapidsai/cudf/pull/17260)) [@vuule](https://github.com/vuule) +- Expose streams in public quantile APIs ([#17257](https://github.com/rapidsai/cudf/pull/17257)) [@shrshi](https://github.com/shrshi) +- Add support for `pyarrow-18` ([#17256](https://github.com/rapidsai/cudf/pull/17256)) [@galipremsagar](https://github.com/galipremsagar) +- Move strings/numeric convert benchmarks to nvbench ([#17255](https://github.com/rapidsai/cudf/pull/17255)) [@davidwendt](https://github.com/davidwendt) +- Add new ``dask_cudf.read_parquet`` API ([#17250](https://github.com/rapidsai/cudf/pull/17250)) [@rjzamora](https://github.com/rjzamora) +- Add read_parquet_metadata to pylibcudf ([#17245](https://github.com/rapidsai/cudf/pull/17245)) [@mroeschke](https://github.com/mroeschke) +- Search for kvikio with lowercase ([#17243](https://github.com/rapidsai/cudf/pull/17243)) [@vyasr](https://github.com/vyasr) +- KvikIO shared library ([#17239](https://github.com/rapidsai/cudf/pull/17239)) [@madsbk](https://github.com/madsbk) +- Use more pylibcudf.io.types enums in cudf._libs ([#17237](https://github.com/rapidsai/cudf/pull/17237)) [@mroeschke](https://github.com/mroeschke) +- Expose mixed and conditional joins in pylibcudf ([#17235](https://github.com/rapidsai/cudf/pull/17235)) [@wence-](https://github.com/wence-) +- Add io.text APIs to pylibcudf ([#17232](https://github.com/rapidsai/cudf/pull/17232)) [@mroeschke](https://github.com/mroeschke) +- Add `num_iterations` axis to the multi-threaded Parquet benchmarks ([#17231](https://github.com/rapidsai/cudf/pull/17231)) [@vuule](https://github.com/vuule) +- Move strings to date/time types benchmarks to nvbench ([#17229](https://github.com/rapidsai/cudf/pull/17229)) [@davidwendt](https://github.com/davidwendt) +- Support for polars 1.12 in cudf-polars ([#17227](https://github.com/rapidsai/cudf/pull/17227)) [@wence-](https://github.com/wence-) +- Allow generating large strings in benchmarks ([#17224](https://github.com/rapidsai/cudf/pull/17224)) [@davidwendt](https://github.com/davidwendt) +- Refactor gather/scatter benchmarks for strings ([#17223](https://github.com/rapidsai/cudf/pull/17223)) [@davidwendt](https://github.com/davidwendt) +- Deprecate single component extraction methods in libcudf ([#17221](https://github.com/rapidsai/cudf/pull/17221)) [@Matt711](https://github.com/Matt711) +- Remove `nvtext::load_vocabulary` from pylibcudf ([#17220](https://github.com/rapidsai/cudf/pull/17220)) [@Matt711](https://github.com/Matt711) +- Benchmarking JSON reader for compressed inputs ([#17219](https://github.com/rapidsai/cudf/pull/17219)) [@shrshi](https://github.com/shrshi) +- Expose stream-ordering in partitioning API ([#17213](https://github.com/rapidsai/cudf/pull/17213)) [@shrshi](https://github.com/shrshi) +- Move strings::concatenate benchmark to nvbench ([#17211](https://github.com/rapidsai/cudf/pull/17211)) [@davidwendt](https://github.com/davidwendt) +- Expose stream-ordering in subword tokenizer API ([#17206](https://github.com/rapidsai/cudf/pull/17206)) [@shrshi](https://github.com/shrshi) +- Refactor Dask cuDF legacy code ([#17205](https://github.com/rapidsai/cudf/pull/17205)) [@rjzamora](https://github.com/rjzamora) +- Make HostMemoryBuffer call into the DefaultHostMemoryAllocator ([#17204](https://github.com/rapidsai/cudf/pull/17204)) [@revans2](https://github.com/revans2) +- Unified binary_ops and ast benchmarks parameter names ([#17200](https://github.com/rapidsai/cudf/pull/17200)) [@lamarrr](https://github.com/lamarrr) +- Add in new java API for raw host memory allocation ([#17197](https://github.com/rapidsai/cudf/pull/17197)) [@revans2](https://github.com/revans2) +- Remove java reservation ([#17189](https://github.com/rapidsai/cudf/pull/17189)) [@revans2](https://github.com/revans2) +- Fixed unused attribute compilation error for GCC 13 ([#17188](https://github.com/rapidsai/cudf/pull/17188)) [@lamarrr](https://github.com/lamarrr) +- Change default KvikIO parameters in cuDF: set the thread pool size to 4, and compatibility mode to ON ([#17185](https://github.com/rapidsai/cudf/pull/17185)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Use make_device_uvector instead of cudaMemcpyAsync in inplace_bitmask_binop ([#17181](https://github.com/rapidsai/cudf/pull/17181)) [@davidwendt](https://github.com/davidwendt) +- Make ai.rapids.cudf.HostMemoryBuffer#copyFromStream public. ([#17179](https://github.com/rapidsai/cudf/pull/17179)) [@liurenjie1024](https://github.com/liurenjie1024) +- Separate evaluation logic from `IR` objects in cudf-polars ([#17175](https://github.com/rapidsai/cudf/pull/17175)) [@rjzamora](https://github.com/rjzamora) +- Move nvtext ngrams benchmarks to nvbench ([#17173](https://github.com/rapidsai/cudf/pull/17173)) [@davidwendt](https://github.com/davidwendt) +- Remove includes suggested by include-what-you-use ([#17170](https://github.com/rapidsai/cudf/pull/17170)) [@vyasr](https://github.com/vyasr) +- Reading multi-source compressed JSONL files ([#17161](https://github.com/rapidsai/cudf/pull/17161)) [@shrshi](https://github.com/shrshi) +- Process parquet bools with microkernels ([#17157](https://github.com/rapidsai/cudf/pull/17157)) [@pmattione-nvidia](https://github.com/pmattione-nvidia) +- Upgrade to polars 1.11 in cudf-polars ([#17154](https://github.com/rapidsai/cudf/pull/17154)) [@wence-](https://github.com/wence-) +- Deprecate current libcudf nvtext minhash functions ([#17152](https://github.com/rapidsai/cudf/pull/17152)) [@davidwendt](https://github.com/davidwendt) +- Remove unused variable in internal merge_tdigests utility ([#17151](https://github.com/rapidsai/cudf/pull/17151)) [@davidwendt](https://github.com/davidwendt) +- Use the full ref name of `rmm.DeviceBuffer` in the sphinx config file ([#17150](https://github.com/rapidsai/cudf/pull/17150)) [@Matt711](https://github.com/Matt711) +- Move `segmented_gather` function from the copying module to the lists module ([#17148](https://github.com/rapidsai/cudf/pull/17148)) [@Matt711](https://github.com/Matt711) +- Use async execution policy for true_if ([#17146](https://github.com/rapidsai/cudf/pull/17146)) [@PointKernel](https://github.com/PointKernel) +- Add conversion from cudf-polars expressions to libcudf ast for parquet filters ([#17141](https://github.com/rapidsai/cudf/pull/17141)) [@wence-](https://github.com/wence-) +- devcontainer: replace `VAULT_HOST` with `AWS_ROLE_ARN` ([#17134](https://github.com/rapidsai/cudf/pull/17134)) [@jjacobelli](https://github.com/jjacobelli) +- Replace direct `cudaMemcpyAsync` calls with utility functions (limited to `cudf::io`) ([#17132](https://github.com/rapidsai/cudf/pull/17132)) [@vuule](https://github.com/vuule) +- use rapids-generate-pip-constraints to pin to oldest dependencies in CI ([#17131](https://github.com/rapidsai/cudf/pull/17131)) [@jameslamb](https://github.com/jameslamb) +- Set the default number of threads in KvikIO thread pool to 8 ([#17126](https://github.com/rapidsai/cudf/pull/17126)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Fix clang-tidy violations for span.hpp and hostdevice_vector.hpp ([#17124](https://github.com/rapidsai/cudf/pull/17124)) [@davidwendt](https://github.com/davidwendt) +- Disable the Parquet reader's wide lists tables GTest by default ([#17120](https://github.com/rapidsai/cudf/pull/17120)) [@mhaseeb123](https://github.com/mhaseeb123) +- Add compile time check to ensure the `counting_iterator` type in `counting_transform_iterator` fits in `size_type` ([#17118](https://github.com/rapidsai/cudf/pull/17118)) [@mhaseeb123](https://github.com/mhaseeb123) +- Minor I/O code quality improvements ([#17105](https://github.com/rapidsai/cudf/pull/17105)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Remove the additional host register calls initially intended for performance improvement on Grace Hopper ([#17092](https://github.com/rapidsai/cudf/pull/17092)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Split hash-based groupby into multiple smaller files to reduce build time ([#17089](https://github.com/rapidsai/cudf/pull/17089)) [@PointKernel](https://github.com/PointKernel) +- build wheels without build isolation ([#17088](https://github.com/rapidsai/cudf/pull/17088)) [@jameslamb](https://github.com/jameslamb) +- Polars: DataFrame Serialization ([#17062](https://github.com/rapidsai/cudf/pull/17062)) [@madsbk](https://github.com/madsbk) +- Remove unused hash helper functions ([#17056](https://github.com/rapidsai/cudf/pull/17056)) [@PointKernel](https://github.com/PointKernel) +- Add to_dlpack/from_dlpack APIs to pylibcudf ([#17055](https://github.com/rapidsai/cudf/pull/17055)) [@mroeschke](https://github.com/mroeschke) +- Move `flatten_single_pass_aggs` to its own TU ([#17053](https://github.com/rapidsai/cudf/pull/17053)) [@PointKernel](https://github.com/PointKernel) +- Replace deprecated cuco APIs with updated versions ([#17052](https://github.com/rapidsai/cudf/pull/17052)) [@PointKernel](https://github.com/PointKernel) +- Refactor ORC dictionary encoding to migrate to the new `cuco::static_map` ([#17049](https://github.com/rapidsai/cudf/pull/17049)) [@mhaseeb123](https://github.com/mhaseeb123) +- Move pylibcudf/libcudf/wrappers/decimals to pylibcudf/libcudf/fixed_point ([#17048](https://github.com/rapidsai/cudf/pull/17048)) [@mroeschke](https://github.com/mroeschke) +- make conda installs in CI stricter (part 2) ([#17042](https://github.com/rapidsai/cudf/pull/17042)) [@jameslamb](https://github.com/jameslamb) +- Use managed memory for NDSH benchmarks ([#17039](https://github.com/rapidsai/cudf/pull/17039)) [@karthikeyann](https://github.com/karthikeyann) +- Clean up hash-groupby `var_hash_functor` ([#17034](https://github.com/rapidsai/cudf/pull/17034)) [@PointKernel](https://github.com/PointKernel) +- Add json APIs to pylibcudf ([#17025](https://github.com/rapidsai/cudf/pull/17025)) [@mroeschke](https://github.com/mroeschke) +- Add string.replace_re APIs to pylibcudf ([#17023](https://github.com/rapidsai/cudf/pull/17023)) [@mroeschke](https://github.com/mroeschke) +- Replace old host tree algorithm with new algorithm in JSON reader ([#17019](https://github.com/rapidsai/cudf/pull/17019)) [@karthikeyann](https://github.com/karthikeyann) +- Unify treatment of `Expr` and `IR` nodes in cudf-polars DSL ([#17016](https://github.com/rapidsai/cudf/pull/17016)) [@wence-](https://github.com/wence-) +- make conda installs in CI stricter ([#17013](https://github.com/rapidsai/cudf/pull/17013)) [@jameslamb](https://github.com/jameslamb) +- Pylibcudf: pack and unpack ([#17012](https://github.com/rapidsai/cudf/pull/17012)) [@madsbk](https://github.com/madsbk) +- Remove unneeded pylibcudf.libcudf.wrappers.duration usage in cudf ([#17010](https://github.com/rapidsai/cudf/pull/17010)) [@mroeschke](https://github.com/mroeschke) +- Add custom "fused" groupby aggregation to Dask cuDF ([#17009](https://github.com/rapidsai/cudf/pull/17009)) [@rjzamora](https://github.com/rjzamora) +- Make tests more deterministic ([#17008](https://github.com/rapidsai/cudf/pull/17008)) [@galipremsagar](https://github.com/galipremsagar) +- Remove unused import ([#17005](https://github.com/rapidsai/cudf/pull/17005)) [@Matt711](https://github.com/Matt711) +- Add string.convert.convert_urls APIs to pylibcudf ([#17003](https://github.com/rapidsai/cudf/pull/17003)) [@mroeschke](https://github.com/mroeschke) +- Add release tracking to project automation scripts ([#17001](https://github.com/rapidsai/cudf/pull/17001)) [@jarmak-nv](https://github.com/jarmak-nv) +- Implement inequality joins by translation to conditional joins ([#17000](https://github.com/rapidsai/cudf/pull/17000)) [@wence-](https://github.com/wence-) +- Add string.convert.convert_lists APIs to pylibcudf ([#16997](https://github.com/rapidsai/cudf/pull/16997)) [@mroeschke](https://github.com/mroeschke) +- Performance optimization of JSON validation ([#16996](https://github.com/rapidsai/cudf/pull/16996)) [@karthikeyann](https://github.com/karthikeyann) +- Add string.convert.convert_ipv4 APIs to pylibcudf ([#16994](https://github.com/rapidsai/cudf/pull/16994)) [@mroeschke](https://github.com/mroeschke) +- Add string.convert.convert_integers APIs to pylibcudf ([#16991](https://github.com/rapidsai/cudf/pull/16991)) [@mroeschke](https://github.com/mroeschke) +- Add string.convert_floats APIs to pylibcudf ([#16990](https://github.com/rapidsai/cudf/pull/16990)) [@mroeschke](https://github.com/mroeschke) +- Add string.convert.convert_fixed_type APIs to pylibcudf ([#16984](https://github.com/rapidsai/cudf/pull/16984)) [@mroeschke](https://github.com/mroeschke) +- Remove unnecessary `std::move`'s in pylibcudf ([#16983](https://github.com/rapidsai/cudf/pull/16983)) [@Matt711](https://github.com/Matt711) +- Add docstrings and test for strings.convert_durations APIs for pylibcudf ([#16982](https://github.com/rapidsai/cudf/pull/16982)) [@mroeschke](https://github.com/mroeschke) +- JSON tokenizer memory optimizations ([#16978](https://github.com/rapidsai/cudf/pull/16978)) [@shrshi](https://github.com/shrshi) +- Turn on `xfail_strict = true` for all python packages ([#16977](https://github.com/rapidsai/cudf/pull/16977)) [@wence-](https://github.com/wence-) +- Add string.convert.convert_datetime/convert_booleans APIs to pylibcudf ([#16971](https://github.com/rapidsai/cudf/pull/16971)) [@mroeschke](https://github.com/mroeschke) +- Auto assign PR to author ([#16969](https://github.com/rapidsai/cudf/pull/16969)) [@Matt711](https://github.com/Matt711) +- Deprecate support for directly accessing logger ([#16964](https://github.com/rapidsai/cudf/pull/16964)) [@vyasr](https://github.com/vyasr) +- Expunge NamedColumn ([#16962](https://github.com/rapidsai/cudf/pull/16962)) [@wence-](https://github.com/wence-) +- Add clang-tidy to CI ([#16958](https://github.com/rapidsai/cudf/pull/16958)) [@vyasr](https://github.com/vyasr) +- Address all remaining clang-tidy errors ([#16956](https://github.com/rapidsai/cudf/pull/16956)) [@vyasr](https://github.com/vyasr) +- Apply clang-tidy autofixes ([#16949](https://github.com/rapidsai/cudf/pull/16949)) [@vyasr](https://github.com/vyasr) +- Use nvcomp wheel instead of bundling nvcomp ([#16946](https://github.com/rapidsai/cudf/pull/16946)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA) +- Refactor the `cuda_memcpy` functions to make them more usable ([#16945](https://github.com/rapidsai/cudf/pull/16945)) [@vuule](https://github.com/vuule) +- Add string.split APIs to pylibcudf ([#16940](https://github.com/rapidsai/cudf/pull/16940)) [@mroeschke](https://github.com/mroeschke) +- clang-tidy fixes part 3 ([#16939](https://github.com/rapidsai/cudf/pull/16939)) [@vyasr](https://github.com/vyasr) +- clang-tidy fixes part 2 ([#16938](https://github.com/rapidsai/cudf/pull/16938)) [@vyasr](https://github.com/vyasr) +- clang-tidy fixes part 1 ([#16937](https://github.com/rapidsai/cudf/pull/16937)) [@vyasr](https://github.com/vyasr) +- Add string.wrap APIs to pylibcudf ([#16935](https://github.com/rapidsai/cudf/pull/16935)) [@mroeschke](https://github.com/mroeschke) +- Add string.translate APIs to pylibcudf ([#16934](https://github.com/rapidsai/cudf/pull/16934)) [@mroeschke](https://github.com/mroeschke) +- Add string.find_multiple APIs to pylibcudf ([#16920](https://github.com/rapidsai/cudf/pull/16920)) [@mroeschke](https://github.com/mroeschke) +- Batch memcpy the last offsets for output buffers of str and list cols in PQ reader ([#16905](https://github.com/rapidsai/cudf/pull/16905)) [@mhaseeb123](https://github.com/mhaseeb123) +- reduce wheel build verbosity, narrow deprecation warning filter ([#16896](https://github.com/rapidsai/cudf/pull/16896)) [@jameslamb](https://github.com/jameslamb) +- Improve aggregation device functors ([#16884](https://github.com/rapidsai/cudf/pull/16884)) [@PointKernel](https://github.com/PointKernel) +- Upgrade pandas pinnings to support `2.2.3` ([#16882](https://github.com/rapidsai/cudf/pull/16882)) [@galipremsagar](https://github.com/galipremsagar) +- Fix 24.10 to 24.12 forward merge ([#16876](https://github.com/rapidsai/cudf/pull/16876)) [@bdice](https://github.com/bdice) +- Manually resolve conflicts in between branch-24.12 and branch-24.10 ([#16871](https://github.com/rapidsai/cudf/pull/16871)) [@galipremsagar](https://github.com/galipremsagar) +- Add in support for setting delim when parsing JSON through java ([#16867](https://github.com/rapidsai/cudf/pull/16867)) [@revans2](https://github.com/revans2) +- Reapply `mixed_semi_join` refactoring and bug fixes ([#16859](https://github.com/rapidsai/cudf/pull/16859)) [@mhaseeb123](https://github.com/mhaseeb123) +- Add string padding and side_type APIs to pylibcudf ([#16833](https://github.com/rapidsai/cudf/pull/16833)) [@mroeschke](https://github.com/mroeschke) +- Organize parquet reader mukernel non-nullable code, introduce manual block scans ([#16830](https://github.com/rapidsai/cudf/pull/16830)) [@pmattione-nvidia](https://github.com/pmattione-nvidia) +- Remove superfluous use of std::vector for std::future ([#16829](https://github.com/rapidsai/cudf/pull/16829)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Rework `read_csv` IO to avoid reading whole input with a single `host_read` ([#16826](https://github.com/rapidsai/cudf/pull/16826)) [@vuule](https://github.com/vuule) +- Add strings.combine APIs to pylibcudf ([#16790](https://github.com/rapidsai/cudf/pull/16790)) [@mroeschke](https://github.com/mroeschke) +- Add remaining string.char_types APIs to pylibcudf ([#16788](https://github.com/rapidsai/cudf/pull/16788)) [@mroeschke](https://github.com/mroeschke) +- Add new nvtext minhash_permuted API ([#16756](https://github.com/rapidsai/cudf/pull/16756)) [@davidwendt](https://github.com/davidwendt) +- Avoid public constructors when called with columns to avoid unnecessary validation ([#16747](https://github.com/rapidsai/cudf/pull/16747)) [@mroeschke](https://github.com/mroeschke) +- Use `changed-files` shared workflow ([#16713](https://github.com/rapidsai/cudf/pull/16713)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA) +- lint: replace `isort` with Ruff's rule I ([#16685](https://github.com/rapidsai/cudf/pull/16685)) [@Borda](https://github.com/Borda) +- Improve the performance of low cardinality groupby ([#16619](https://github.com/rapidsai/cudf/pull/16619)) [@PointKernel](https://github.com/PointKernel) +- Parquet reader list microkernel ([#16538](https://github.com/rapidsai/cudf/pull/16538)) [@pmattione-nvidia](https://github.com/pmattione-nvidia) +- AWS S3 IO through KvikIO ([#16499](https://github.com/rapidsai/cudf/pull/16499)) [@madsbk](https://github.com/madsbk) +- Refactor `histogram` reduction using `cuco::static_set::insert_and_find` ([#16485](https://github.com/rapidsai/cudf/pull/16485)) [@srinivasyadav18](https://github.com/srinivasyadav18) +- Use numba-cuda>=0.0.13 ([#16474](https://github.com/rapidsai/cudf/pull/16474)) [@gmarkall](https://github.com/gmarkall) + # cudf 24.10.00 (9 Oct 2024) ## 🚨 Breaking Changes From 2da273c072f861ef31a36f97d7bbc7f690d3b9a7 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Wed, 5 Feb 2025 22:25:03 -0500 Subject: [PATCH 002/129] Fix torch integration test (#17923) Apart of #17490. Creating a torch tensor from a cudf.pandas proxy `Series` or `DataFrame` creates a device tensor if the underlying object is cudf. So this PR updates the `assert_eq` function in the module to convert the device tensor to a host tensor before comparing equality. Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17923 --- .../tests/test_pytorch.py | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_pytorch.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_pytorch.py index 7cea635afc4..f728c79778b 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_pytorch.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_pytorch.py @@ -1,11 +1,20 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. import numpy as np import pandas as pd import pytest import torch -pytestmark = pytest.mark.assert_eq(fn=torch.testing.assert_close) +pytestmark = pytest.mark.assert_eq( + fn=lambda expect, got, **kwargs: torch.testing.assert_close( + got, expect, **kwargs + ) +) + + +def torch_ctor_assert_eq(expect, got, **kwargs): + assert got.is_cuda, "torch.Tensor should be on the device" + torch.testing.assert_close(got.to("cpu"), expect, **kwargs) @pytest.fixture @@ -116,9 +125,7 @@ def test_torch_train(data): return model(test_x1, test_x2) -@pytest.mark.skip( - reason="AssertionError: The values for attribute 'device' do not match: cpu != cuda:0." -) +@pytest.mark.assert_eq(fn=torch_ctor_assert_eq) def test_torch_tensor_ctor(): s = pd.Series(range(5)) return torch.tensor(s.values) From 92cf5605b3a442d349cc834c8037843e5406648a Mon Sep 17 00:00:00 2001 From: Vyas Ramasubramani Date: Wed, 5 Feb 2025 19:52:16 -0800 Subject: [PATCH 003/129] Remove dataframe protocol (#17909) Follow-up to #17736 Authors: - Vyas Ramasubramani (https://github.com/vyasr) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/17909 --- .../user_guide/api_docs/general_functions.rst | 1 - python/cudf/cudf/__init__.py | 4 +- python/cudf/cudf/core/dataframe.py | 45 +- python/cudf/cudf/core/df_protocol.py | 900 ------------------ .../cudf/pandas/scripts/run-pandas-tests.sh | 5 +- python/cudf/cudf/tests/test_df_protocol.py | 286 ------ .../cudf_pandas_tests/test_cudf_pandas.py | 19 - 7 files changed, 6 insertions(+), 1254 deletions(-) delete mode 100644 python/cudf/cudf/core/df_protocol.py delete mode 100644 python/cudf/cudf/tests/test_df_protocol.py diff --git a/docs/cudf/source/user_guide/api_docs/general_functions.rst b/docs/cudf/source/user_guide/api_docs/general_functions.rst index 5c5b5cb3b04..350fc372130 100644 --- a/docs/cudf/source/user_guide/api_docs/general_functions.rst +++ b/docs/cudf/source/user_guide/api_docs/general_functions.rst @@ -26,7 +26,6 @@ Top-level conversions :toctree: api/ to_numeric - from_dataframe from_dlpack from_pandas diff --git a/python/cudf/cudf/__init__.py b/python/cudf/cudf/__init__.py index 843f2670b4d..5f99522a0ae 100644 --- a/python/cudf/cudf/__init__.py +++ b/python/cudf/cudf/__init__.py @@ -1,4 +1,4 @@ -# Copyright (c) 2018-2024, NVIDIA CORPORATION. +# Copyright (c) 2018-2025, NVIDIA CORPORATION. # If libcudf was installed as a wheel, we must request it to load the library symbols. # Otherwise, we assume that the library was installed in a system path that ld can find. @@ -36,7 +36,7 @@ from cudf.api.types import dtype from cudf.core.algorithms import factorize, unique from cudf.core.cut import cut -from cudf.core.dataframe import DataFrame, from_dataframe, from_pandas, merge +from cudf.core.dataframe import DataFrame, from_pandas, merge from cudf.core.dtypes import ( CategoricalDtype, Decimal32Dtype, diff --git a/python/cudf/cudf/core/dataframe.py b/python/cudf/cudf/core/dataframe.py index a71c9f70b00..5041c9be476 100644 --- a/python/cudf/cudf/core/dataframe.py +++ b/python/cudf/cudf/core/dataframe.py @@ -27,7 +27,6 @@ import pandas as pd import pyarrow as pa from nvtx import annotate -from packaging import version from pandas.io.formats import console from pandas.io.formats.printing import pprint_thing from typing_extensions import Self, assert_never @@ -48,7 +47,7 @@ is_scalar, is_string_dtype, ) -from cudf.core import column, df_protocol, indexing_utils, reshape +from cudf.core import column, indexing_utils, reshape from cudf.core._compat import PANDAS_LT_300 from cudf.core.buffer import acquire_spill_lock, as_buffer from cudf.core.column import ( @@ -5564,16 +5563,6 @@ def from_pandas(cls, dataframe, nan_as_null=no_default): # Checks duplicate columns and sets column metadata df.columns = dataframe.columns return df - elif hasattr(dataframe, "__dataframe__"): - # TODO: Probably should be handled in the constructor as - # this isn't pandas specific - assert version.parse(cudf.__version__) < version.parse("25.04.00") - warnings.warn( - "Support for loading dataframes via the `__dataframe__` interchange " - "protocol is deprecated", - FutureWarning, - ) - return from_dataframe(dataframe, allow_copy=True) else: raise TypeError( f"Could not construct DataFrame from {type(dataframe)}" @@ -7730,15 +7719,6 @@ def pct_change( periods=periods, freq=freq, **kwargs ) - def __dataframe__( - self, nan_as_null: bool = False, allow_copy: bool = True - ): - assert version.parse(cudf.__version__) < version.parse("25.04.00") - warnings.warn("Using `__dataframe__` is deprecated", FutureWarning) - return df_protocol.__dataframe__( - self, nan_as_null=nan_as_null, allow_copy=allow_copy - ) - def nunique(self, axis=0, dropna: bool = True) -> Series: """ Count number of distinct elements in specified axis. @@ -8183,29 +8163,6 @@ def from_pylibcudf(cls, table: plc.Table, metadata: dict) -> Self: return df -def from_dataframe(df, allow_copy: bool = False) -> DataFrame: - """ - Build a :class:`DataFrame` from an object supporting the dataframe interchange protocol. - - .. note:: - - If you have a ``pandas.DataFrame``, use :func:`from_pandas` instead. - - Parameters - ---------- - df : DataFrameXchg - Object supporting the interchange protocol, i.e. ``__dataframe__`` method. - allow_copy : bool, default: True - Whether to allow copying the memory to perform the conversion - (if false then zero-copy approach is requested). - - Returns - ------- - :class:`DataFrame` - """ - return df_protocol.from_dataframe(df, allow_copy=allow_copy) - - def make_binop_func(op, postprocess=None): # This function is used to wrap binary operations in Frame with an # appropriate API for DataFrame as required for pandas compatibility. The diff --git a/python/cudf/cudf/core/df_protocol.py b/python/cudf/cudf/core/df_protocol.py deleted file mode 100644 index cc9f39d70ef..00000000000 --- a/python/cudf/cudf/core/df_protocol.py +++ /dev/null @@ -1,900 +0,0 @@ -# Copyright (c) 2021-2025, NVIDIA CORPORATION. -from __future__ import annotations - -import enum -from collections import abc -from typing import TYPE_CHECKING, Any, cast - -import cupy as cp -import numpy as np -from numba.cuda import as_cuda_array - -import rmm - -import cudf -from cudf.core.buffer import Buffer, as_buffer -from cudf.core.column import ( - CategoricalColumn, - NumericalColumn, - as_column, - build_column, -) - -if TYPE_CHECKING: - from collections.abc import Iterable, Mapping, Sequence - -# Implementation of interchange protocol classes -# ---------------------------------------------- - - -class _DtypeKind(enum.IntEnum): - INT = 0 - UINT = 1 - FLOAT = 2 - BOOL = 20 - STRING = 21 # UTF-8 - DATETIME = 22 - CATEGORICAL = 23 - - -class _Device(enum.IntEnum): - CPU = 1 - CUDA = 2 - CPU_PINNED = 3 - OPENCL = 4 - VULKAN = 7 - METAL = 8 - VPI = 9 - ROCM = 10 - - -class _MaskKind(enum.IntEnum): - NON_NULLABLE = 0 - NAN = 1 - SENTINEL = 2 - BITMASK = 3 - BYTEMASK = 4 - - -_SUPPORTED_KINDS = { - _DtypeKind.INT, - _DtypeKind.UINT, - _DtypeKind.FLOAT, - _DtypeKind.CATEGORICAL, - _DtypeKind.BOOL, - _DtypeKind.STRING, -} -ProtoDtype = tuple[_DtypeKind, int, str, str] - - -class _CuDFBuffer: - """ - Data in the buffer is guaranteed to be contiguous in memory. - """ - - def __init__( - self, - buf: Buffer, - dtype: np.dtype, - allow_copy: bool = True, - ) -> None: - """ - Use Buffer object. - """ - # Store the cudf buffer where the data resides as a private - # attribute, so we can use it to retrieve the public attributes - self._buf = buf - self._dtype = dtype - self._allow_copy = allow_copy - - @property - def bufsize(self) -> int: - """ - The Buffer size in bytes. - """ - return self._buf.size - - @property - def ptr(self) -> int: - """ - Pointer to start of the buffer as an integer. - """ - return self._buf.get_ptr(mode="write") - - def __dlpack__(self): - # DLPack not implemented in NumPy yet, so leave it out here. - try: - cuda_array = as_cuda_array(self._buf).view(self._dtype) - return cp.asarray(cuda_array).toDlpack() - except ValueError: - raise TypeError(f"dtype {self._dtype} unsupported by `dlpack`") - - def __dlpack_device__(self) -> tuple[_Device, int]: - """ - _Device type and _Device ID for where the data in the buffer resides. - """ - return (_Device.CUDA, cp.asarray(self._buf).device.id) - - def __repr__(self) -> str: - return f"{self.__class__.__name__}(" + str( - { - "bufsize": self.bufsize, - "ptr": self.ptr, - "device": self.__dlpack_device__()[0].name, - } - ) - +")" - - -class _CuDFColumn: - """ - A column object, with only the methods and properties required by the - interchange protocol defined. - - A column can contain one or more chunks. Each chunk can contain up to three - buffers - a data buffer, a mask buffer (depending on null representation), - and an offsets buffer (if variable-size binary; e.g., variable-length - strings). - - Note: this Column object can only be produced by ``__dataframe__``, so - doesn't need its own version or ``__column__`` protocol. - - """ - - def __init__( - self, - column: cudf.core.column.ColumnBase, - nan_as_null: bool = True, - allow_copy: bool = True, - ) -> None: - """ - Note: doesn't deal with extension arrays yet, just assume a regular - Series/ndarray for now. - """ - if not isinstance(column, cudf.core.column.ColumnBase): - raise TypeError( - "column must be a subtype of df.core.column.ColumnBase," - f"got {type(column)}" - ) - self._col = column - self._nan_as_null = nan_as_null - self._allow_copy = allow_copy - - def size(self) -> int: - """ - Size of the column, in elements. - """ - return self._col.size - - @property - def offset(self) -> int: - """ - Offset of first element. Always zero. - """ - return 0 - - @property - def dtype(self) -> ProtoDtype: - """ - Dtype description as a tuple - ``(kind, bit-width, format string, endianness)`` - - Kind : - - - INT = 0 - - UINT = 1 - - FLOAT = 2 - - BOOL = 20 - - STRING = 21 # UTF-8 - - DATETIME = 22 - - CATEGORICAL = 23 - - Bit-width : the number of bits as an integer - Format string : data type description format string in Apache Arrow C - Data Interface format. - Endianness : current only native endianness (``=``) is supported - - Notes - ----- - - Kind specifiers are aligned with DLPack where possible - (hence the jump to 20, leave enough room for future extension) - - Masks must be specified as boolean with either bit width 1 - (for bit masks) or 8 (for byte masks). - - Dtype width in bits was preferred over bytes - - Endianness isn't too useful, but included now in case - in the future we need to support non-native endianness - - Went with Apache Arrow format strings over NumPy format strings - because they're more complete from a dataframe perspective - - Format strings are mostly useful for datetime specification, - and for categoricals. - - For categoricals, the format string describes the type of the - categorical in the data buffer. In case of a separate encoding - of the categorical (e.g. an integer to string mapping), - this can be derived from ``self.describe_categorical``. - - Data types not included: complex, Arrow-style null, - binary, decimal, and nested (list, struct, map, union) dtypes. - """ - dtype = self._col.dtype - - # For now, assume that, if the column dtype is 'O' (i.e., `object`), - # then we have an array of strings - if not isinstance(dtype, cudf.CategoricalDtype) and dtype.kind == "O": - return (_DtypeKind.STRING, 8, "u", "=") - - return self._dtype_from_cudfdtype(dtype) - - def _dtype_from_cudfdtype(self, dtype) -> ProtoDtype: - """ - See `self.dtype` for details. - """ - # Note: 'c' (complex) not handled yet (not in array spec v1). - # 'b', 'B' (bytes), 'S', 'a', (old-style string) 'V' (void) - # not handled datetime and timedelta both map to datetime - # (is timedelta handled?) - _np_kinds = { - "i": _DtypeKind.INT, - "u": _DtypeKind.UINT, - "f": _DtypeKind.FLOAT, - "b": _DtypeKind.BOOL, - "U": _DtypeKind.STRING, - "M": _DtypeKind.DATETIME, - "m": _DtypeKind.DATETIME, - } - kind = _np_kinds.get(dtype.kind, None) - if kind is None: - # Not a NumPy/CuPy dtype. Check if it's a categorical maybe - if isinstance(dtype, cudf.CategoricalDtype): - kind = _DtypeKind.CATEGORICAL - # Codes and categories' dtypes are different. - # We use codes' dtype as these are stored in the buffer. - codes = cast( - cudf.core.column.CategoricalColumn, self._col - ).codes - dtype = codes.dtype - else: - raise ValueError( - f"Data type {dtype} not supported by exchange protocol" - ) - - if kind not in _SUPPORTED_KINDS: - raise NotImplementedError(f"Data type {dtype} not handled yet") - - bitwidth = dtype.itemsize * 8 - format_str = dtype.str - endianness = dtype.byteorder if kind != _DtypeKind.CATEGORICAL else "=" - return (kind, bitwidth, format_str, endianness) - - @property - def describe_categorical(self) -> tuple[bool, bool, dict[int, Any]]: - """ - If the dtype is categorical, there are two options: - - - There are only values in the data buffer. - - There is a separate dictionary-style encoding for categorical values. - - Raises TypeError if the dtype is not categorical - - Content of returned dict: - - - "is_ordered" : bool, whether the ordering of dictionary - indices is semantically meaningful. - - "is_dictionary" : bool, whether a dictionary-style mapping of - categorical values to other objects exists - - "mapping" : dict, Python-level only (e.g. ``{int: str}``). - None if not a dictionary-style categorical. - """ - if not self.dtype[0] == _DtypeKind.CATEGORICAL: - raise TypeError( - "`describe_categorical only works on " - "a column with categorical dtype!" - ) - categ_col = cast(cudf.core.column.CategoricalColumn, self._col) - ordered = bool(categ_col.dtype.ordered) - is_dictionary = True - # NOTE: this shows the children approach is better, transforming - # `categories` to a "mapping" dict is inefficient - categories = categ_col.categories - mapping = {ix: val for ix, val in enumerate(categories.values_host)} - return ordered, is_dictionary, mapping - - @property - def describe_null(self) -> tuple[int, Any]: - """ - Return the missing value (or "null") representation the column dtype - uses, as a tuple ``(kind, value)``. - - Kind: - - - 0 : non-nullable - - 1 : NaN/NaT - - 2 : sentinel value - - 3 : bit mask - - 4 : byte mask - - Value : if kind is "sentinel value", the actual value. - If kind is a bit mask or a byte mask, the value (0 or 1) - indicating a missing value. - None otherwise. - """ - kind = self.dtype[0] - if self.null_count == 0: - # there is no validity mask so it is non-nullable - return _MaskKind.NON_NULLABLE, None - - elif kind in _SUPPORTED_KINDS: - # currently, we return a bit mask - return _MaskKind.BITMASK, 0 - - else: - raise NotImplementedError( - f"Data type {self.dtype} not yet supported" - ) - - @property - def null_count(self) -> int: - """ - Number of null elements. Should always be known. - """ - return self._col.null_count - - @property - def metadata(self) -> dict[str, Any]: - """ - Store specific metadata of the column. - """ - return {} - - def num_chunks(self) -> int: - """ - Return the number of chunks the column consists of. - """ - return 1 - - def get_chunks( - self, n_chunks: int | None = None - ) -> Iterable["_CuDFColumn"]: - """ - Return an iterable yielding the chunks. - - See `DataFrame.get_chunks` for details on ``n_chunks``. - """ - return (self,) - - def get_buffers( - self, - ) -> Mapping[str, tuple[_CuDFBuffer, ProtoDtype] | None]: - """ - Return a dictionary containing the underlying buffers. - - The returned dictionary has the following contents: - - - "data": a two-element tuple whose first element is a buffer - containing the data and whose second element is the data - buffer's associated dtype. - - "validity": a two-element tuple whose first element is a buffer - containing mask values indicating missing data and - whose second element is the mask value buffer's - associated dtype. None if the null representation is - not a bit or byte mask. - - "offsets": a two-element tuple whose first element is a buffer - containing the offset values for variable-size binary - data (e.g., variable-length strings) and whose second - element is the offsets buffer's associated dtype. None - if the data buffer does not have an associated offsets - buffer. - """ - buffers = {} - try: - buffers["validity"] = self._get_validity_buffer() - except RuntimeError: - buffers["validity"] = None - - try: - buffers["offsets"] = self._get_offsets_buffer() - except RuntimeError: - buffers["offsets"] = None - - buffers["data"] = self._get_data_buffer() - - return buffers - - def _get_validity_buffer( - self, - ) -> tuple[_CuDFBuffer, ProtoDtype] | None: - """ - Return the buffer containing the mask values - indicating missing data and the buffer's associated dtype. - - Raises RuntimeError if null representation is not a bit or byte mask. - """ - null, invalid = self.describe_null - - if null == _MaskKind.BITMASK: - assert self._col.mask is not None - buffer = _CuDFBuffer( - self._col.mask, cp.uint8, allow_copy=self._allow_copy - ) - dtype = (_DtypeKind.UINT, 8, "C", "=") - return buffer, dtype - - elif null == _MaskKind.NAN: - raise RuntimeError( - "This column uses NaN as null so does not have a separate mask" - ) - elif null == _MaskKind.NON_NULLABLE: - raise RuntimeError( - "This column is non-nullable so does not have a mask" - ) - else: - raise NotImplementedError( - f"See {self.__class__.__name__}.describe_null method." - ) - - def _get_offsets_buffer( - self, - ) -> tuple[_CuDFBuffer, ProtoDtype] | None: - """ - Return the buffer containing the offset values for - variable-size binary data (e.g., variable-length strings) - and the buffer's associated dtype. - - Raises RuntimeError if the data buffer does not have an associated - offsets buffer. - """ - if self.dtype[0] == _DtypeKind.STRING: - offsets = self._col.children[0] - assert (offsets is not None) and (offsets.data is not None), " " - "offsets(.data) should not be None for string column" - - buffer = _CuDFBuffer( - offsets.data, offsets.dtype, allow_copy=self._allow_copy - ) - dtype = self._dtype_from_cudfdtype(offsets.dtype) - else: - raise RuntimeError( - "This column has a fixed-length dtype " - "so does not have an offsets buffer" - ) - - return buffer, dtype - - def _get_data_buffer( - self, - ) -> tuple[_CuDFBuffer, ProtoDtype]: - """ - Return the buffer containing the data and - the buffer's associated dtype. - """ - if self.dtype[0] in ( - _DtypeKind.INT, - _DtypeKind.UINT, - _DtypeKind.FLOAT, - _DtypeKind.BOOL, - ): - col_data = self._col - dtype = self.dtype - - elif self.dtype[0] == _DtypeKind.CATEGORICAL: - col_data = cast( - cudf.core.column.CategoricalColumn, self._col - ).codes - dtype = self._dtype_from_cudfdtype(col_data.dtype) - - elif self.dtype[0] == _DtypeKind.STRING: - col_data = build_column( - data=self._col.data, dtype=np.dtype("int8") - ) - dtype = self._dtype_from_cudfdtype(col_data.dtype) - - else: - raise NotImplementedError( - f"Data type {self._col.dtype} not handled yet" - ) - assert (col_data is not None) and (col_data.data is not None), " " - f"col_data(.data) should not be None when dtype = {dtype}" - buffer = _CuDFBuffer( - col_data.data, col_data.dtype, allow_copy=self._allow_copy - ) - - return buffer, dtype - - -class _CuDFDataFrame: - """ - A data frame class, with only the methods required by the interchange - protocol defined. - - Instances of this (private) class are returned from - ``cudf.DataFrame.__dataframe__`` as objects with the methods and - attributes defined on this class. - """ - - def __init__( - self, - df: "cudf.core.dataframe.DataFrame", - nan_as_null: bool = True, - allow_copy: bool = True, - ) -> None: - """ - Constructor - an instance of this (private) class is returned from - `cudf.DataFrame.__dataframe__`. - """ - self._df = df - # ``nan_as_null`` is a keyword intended for the consumer to tell the - # producer to overwrite null values in the data with - # ``NaN`` (or ``NaT``). - # This currently has no effect; once support for nullable extension - # dtypes is added, this value should be propagated to columns. - self._nan_as_null = nan_as_null - self._allow_copy = allow_copy - - def __dataframe__( - self, nan_as_null: bool = False, allow_copy: bool = True - ) -> "_CuDFDataFrame": - """ - See the docstring of the `cudf.DataFrame.__dataframe__` for details - """ - return _CuDFDataFrame( - self._df, nan_as_null=nan_as_null, allow_copy=allow_copy - ) - - @property - def metadata(self): - # `index` isn't a regular column, and the protocol doesn't support row - # labels - so we export it as cuDF-specific metadata here. - return {"cudf.index": self._df.index} - - def num_columns(self) -> int: - return len(self._df._column_names) - - def num_rows(self) -> int: - return len(self._df) - - def num_chunks(self) -> int: - return 1 - - def column_names(self) -> Iterable[str]: - return self._df._column_names - - def get_column(self, i: int) -> _CuDFColumn: - return _CuDFColumn( - as_column(self._df.iloc[:, i]), allow_copy=self._allow_copy - ) - - def get_column_by_name(self, name: str) -> _CuDFColumn: - return _CuDFColumn( - as_column(self._df[name]), allow_copy=self._allow_copy - ) - - def get_columns(self) -> Iterable[_CuDFColumn]: - return [ - _CuDFColumn(as_column(self._df[name]), allow_copy=self._allow_copy) - for name in self._df.columns - ] - - def select_columns(self, indices: Sequence[int]) -> "_CuDFDataFrame": - if not isinstance(indices, abc.Sequence): - raise ValueError("`indices` is not a sequence") - - return _CuDFDataFrame(self._df.iloc[:, indices]) - - def select_columns_by_name(self, names: Sequence[str]) -> "_CuDFDataFrame": - if not isinstance(names, abc.Sequence): - raise ValueError("`names` is not a sequence") - - return _CuDFDataFrame( - self._df.loc[:, names], self._nan_as_null, self._allow_copy - ) - - def get_chunks( - self, n_chunks: int | None = None - ) -> Iterable["_CuDFDataFrame"]: - """ - Return an iterator yielding the chunks. - """ - return (self,) - - -def __dataframe__( - self, nan_as_null: bool = False, allow_copy: bool = True -) -> _CuDFDataFrame: - """ - The public method to attach to cudf.DataFrame. - - ``nan_as_null`` is a keyword intended for the consumer to tell the - producer to overwrite null values in the data with ``NaN`` (or ``NaT``). - This currently has no effect; once support for nullable extension - dtypes is added, this value should be propagated to columns. - - ``allow_copy`` is a keyword that defines whether or not the library is - allowed to make a copy of the data. For example, copying data would be - necessary if a library supports strided buffers, given that this protocol - specifies contiguous buffers. - """ - return _CuDFDataFrame(self, nan_as_null=nan_as_null, allow_copy=allow_copy) - - -""" -Implementation of the dataframe exchange protocol. - -Public API ----------- - -from_dataframe : construct a cudf.DataFrame from an input data frame which - implements the exchange protocol - -Notes ------ - -- Interpreting a raw pointer (as in ``Buffer.ptr``) is annoying and - unsafe to do in pure Python. It's more general but definitely less friendly - than having ``to_arrow`` and ``to_numpy`` methods. So for the buffers which - lack ``__dlpack__`` (e.g., because the column dtype isn't supported by - DLPack), this is worth looking at again. - -""" - - -# A typing protocol could be added later to let Mypy validate code using -# `from_dataframe` better. -DataFrameObject = Any -ColumnObject = Any - - -_INTS = {8: cp.int8, 16: cp.int16, 32: cp.int32, 64: cp.int64} -_UINTS = {8: cp.uint8, 16: cp.uint16, 32: cp.uint32, 64: cp.uint64} -_FLOATS = {32: cp.float32, 64: cp.float64} -_CP_DTYPES = { - 0: _INTS, - 1: _UINTS, - 2: _FLOATS, - 20: {8: bool}, - 21: {8: cp.uint8}, -} - - -def from_dataframe( - df: DataFrameObject, allow_copy: bool = False -) -> cudf.DataFrame: - """ - Construct a ``DataFrame`` from ``df`` if it supports the - dataframe interchange protocol (``__dataframe__``). - - Parameters - ---------- - df : DataFrameObject - Object supporting dataframe interchange protocol - allow_copy : bool - If ``True``, allow copying of the data. If ``False``, a - ``TypeError`` is raised if data copying is required to - construct the ``DataFrame`` (e.g., if ``df`` lives in CPU - memory). - - Returns - ------- - DataFrame - - Examples - -------- - >>> import pandas as pd - >>> pdf = pd.DataFrame({'a': [1, 2, 3], 'b': ['x', 'y', 'z']}) - >>> df = cudf.from_dataframe(pdf, allow_copy=True) - >>> type(df) - cudf.core.dataframe.DataFrame - >>> df - a b - 0 1 x - 1 2 y - 2 3 z - - Notes - ----- - See https://data-apis.org/dataframe-protocol/latest/index.html - for the dataframe interchange protocol spec and API - """ - if isinstance(df, cudf.DataFrame): - return df - - if not hasattr(df, "__dataframe__"): - raise ValueError("`df` does not support __dataframe__") - - df = df.__dataframe__(allow_copy=allow_copy) - - # Check number of chunks, if there's more than one we need to iterate - if df.num_chunks() > 1: - raise NotImplementedError("More than one chunk not handled yet") - - # We need a dict of columns here, with each column being a cudf column. - columns = dict() - _buffers = [] # hold on to buffers, keeps memory alive - for name in df.column_names(): - col = df.get_column_by_name(name) - - if col.dtype[0] in ( - _DtypeKind.INT, - _DtypeKind.UINT, - _DtypeKind.FLOAT, - _DtypeKind.BOOL, - ): - columns[name], _buf = _protocol_to_cudf_column_numeric( - col, allow_copy - ) - - elif col.dtype[0] == _DtypeKind.CATEGORICAL: - columns[name], _buf = _protocol_to_cudf_column_categorical( - col, allow_copy - ) - - elif col.dtype[0] == _DtypeKind.STRING: - columns[name], _buf = _protocol_to_cudf_column_string( - col, allow_copy - ) - - else: - raise NotImplementedError( - f"Data type {col.dtype[0]} not handled yet" - ) - - _buffers.append(_buf) - - df_new = cudf.DataFrame._from_data(columns) - df_new._buffers = _buffers - return df_new - - -def _protocol_to_cudf_column_numeric( - col, allow_copy: bool -) -> tuple[ - cudf.core.column.ColumnBase, - Mapping[str, tuple[_CuDFBuffer, ProtoDtype] | None], -]: - """ - Convert an int, uint, float or bool protocol column - to the corresponding cudf column - """ - if col.offset != 0: - raise NotImplementedError("column.offset > 0 not handled yet") - - buffers = col.get_buffers() - assert buffers["data"] is not None, "data buffer should not be None" - _dbuffer, _ddtype = buffers["data"] - _dbuffer = _ensure_gpu_buffer(_dbuffer, _ddtype, allow_copy) - cudfcol_num = build_column( - _dbuffer._buf, - protocol_dtype_to_cupy_dtype(_ddtype), - ) - return _set_missing_values(col, cudfcol_num, allow_copy), buffers - - -def _ensure_gpu_buffer(buf, data_type, allow_copy: bool) -> _CuDFBuffer: - # if `buf` is a (protocol) buffer that lives on the GPU already, - # return it as is. Otherwise, copy it to the device and return - # the resulting buffer. - if buf.__dlpack_device__()[0] != _Device.CUDA: - if allow_copy: - dbuf = rmm.DeviceBuffer(ptr=buf.ptr, size=buf.bufsize) - return _CuDFBuffer( - as_buffer(dbuf, exposed=True), - protocol_dtype_to_cupy_dtype(data_type), - allow_copy, - ) - else: - raise TypeError( - "This operation must copy data from CPU to GPU. " - "Set `allow_copy=True` to allow it." - ) - return buf - - -def _set_missing_values( - protocol_col, - cudf_col: cudf.core.column.ColumnBase, - allow_copy: bool, -) -> cudf.core.column.ColumnBase: - valid_mask = protocol_col.get_buffers()["validity"] - if valid_mask is not None: - null, invalid = protocol_col.describe_null - if null == _MaskKind.BYTEMASK: - valid_mask = _ensure_gpu_buffer( - valid_mask[0], valid_mask[1], allow_copy - ) - bitmask = as_column(valid_mask._buf, dtype="bool").as_mask() - return cudf_col.set_mask(bitmask) - elif null == _MaskKind.BITMASK: - valid_mask = _ensure_gpu_buffer( - valid_mask[0], valid_mask[1], allow_copy - ) - bitmask = valid_mask._buf - return cudf_col.set_mask(bitmask) - return cudf_col - - -def protocol_dtype_to_cupy_dtype(_dtype: ProtoDtype) -> cp.dtype: - kind = _dtype[0] - bitwidth = _dtype[1] - if _dtype[0] not in _SUPPORTED_KINDS: - raise RuntimeError(f"Data type {_dtype[0]} not handled yet") - - return _CP_DTYPES[kind][bitwidth] - - -def _protocol_to_cudf_column_categorical( - col, allow_copy: bool -) -> tuple[ - cudf.core.column.ColumnBase, - Mapping[str, tuple[_CuDFBuffer, ProtoDtype] | None], -]: - """ - Convert a categorical column to a Series instance - """ - ordered, is_dict, categories = col.describe_categorical - if not is_dict: - raise NotImplementedError( - "Non-dictionary categoricals not supported yet" - ) - buffers = col.get_buffers() - assert buffers["data"] is not None, "data buffer should not be None" - codes_buffer, codes_dtype = buffers["data"] - codes_buffer = _ensure_gpu_buffer(codes_buffer, codes_dtype, allow_copy) - cdtype = np.dtype(protocol_dtype_to_cupy_dtype(codes_dtype)) - codes = NumericalColumn( - data=codes_buffer._buf, - size=None, - dtype=cdtype, - ) - cudfcol = CategoricalColumn( - data=None, - size=codes.size, - dtype=cudf.CategoricalDtype(categories=categories, ordered=ordered), - mask=codes.base_mask, - offset=codes.offset, - children=(codes,), - ) - - return _set_missing_values(col, cudfcol, allow_copy), buffers - - -def _protocol_to_cudf_column_string( - col, allow_copy: bool -) -> tuple[ - cudf.core.column.ColumnBase, - Mapping[str, tuple[_CuDFBuffer, ProtoDtype] | None], -]: - """ - Convert a string ColumnObject to cudf Column object. - """ - # Retrieve the data buffers - buffers = col.get_buffers() - - # Retrieve the data buffer containing the UTF-8 code units - assert buffers["data"] is not None, "data buffer should never be None" - data_buffer, data_dtype = buffers["data"] - data_buffer = _ensure_gpu_buffer(data_buffer, data_dtype, allow_copy) - encoded_string = build_column( - data_buffer._buf, - protocol_dtype_to_cupy_dtype(data_dtype), - ) - - # Retrieve the offsets buffer containing the index offsets demarcating - # the beginning and end of each string - assert buffers["offsets"] is not None, "not possible for string column" - offset_buffer, offset_dtype = buffers["offsets"] - offset_buffer = _ensure_gpu_buffer(offset_buffer, offset_dtype, allow_copy) - offsets = build_column( - offset_buffer._buf, - protocol_dtype_to_cupy_dtype(offset_dtype), - ) - offsets = offsets.astype("int32") - cudfcol_str = build_column( - None, dtype=cp.dtype("O"), children=(offsets, encoded_string) - ) - return _set_missing_values(col, cudfcol_str, allow_copy), buffers - - -def _protocol_buffer_to_cudf_buffer(protocol_buffer): - return as_buffer( - rmm.DeviceBuffer( - ptr=protocol_buffer.ptr, size=protocol_buffer.bufsize - ), - exposed=True, - ) diff --git a/python/cudf/cudf/pandas/scripts/run-pandas-tests.sh b/python/cudf/cudf/pandas/scripts/run-pandas-tests.sh index 9b9ce026571..fe8a0ef24f3 100755 --- a/python/cudf/cudf/pandas/scripts/run-pandas-tests.sh +++ b/python/cudf/cudf/pandas/scripts/run-pandas-tests.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# SPDX-FileCopyrightText: Copyright (c) 2023-2024, NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2023-2025, NVIDIA CORPORATION & AFFILIATES. # All rights reserved. # SPDX-License-Identifier: Apache-2.0 @@ -24,7 +24,8 @@ PANDAS_VERSION=$(python -c "import pandas; print(pandas.__version__)") # tests/io/test_clipboard.py::TestClipboard crashes pytest workers (possibly due to fixture patching clipboard functionality) PYTEST_IGNORES="--ignore=tests/io/parser/common/test_read_errors.py \ ---ignore=tests/io/test_clipboard.py" +--ignore=tests/io/test_clipboard.py \ +--ignore=tests/io/test_pickle.py" mkdir -p pandas-testing cd pandas-testing diff --git a/python/cudf/cudf/tests/test_df_protocol.py b/python/cudf/cudf/tests/test_df_protocol.py deleted file mode 100644 index 371db773805..00000000000 --- a/python/cudf/cudf/tests/test_df_protocol.py +++ /dev/null @@ -1,286 +0,0 @@ -# Copyright (c) 2021-2025, NVIDIA CORPORATION. -from __future__ import annotations - -from typing import Any - -import cupy as cp -import pandas as pd -import pytest - -import cudf -from cudf.core.buffer import as_buffer -from cudf.core.column import as_column, build_column -from cudf.core.df_protocol import ( - DataFrameObject, - _CuDFBuffer, - _CuDFColumn, - _DtypeKind, - _MaskKind, - _protocol_buffer_to_cudf_buffer, - from_dataframe, - protocol_dtype_to_cupy_dtype, -) -from cudf.testing import assert_eq - -pytestmark = pytest.mark.filterwarnings( - "ignore:Using `__dataframe__` is deprecated:FutureWarning" -) - - -@pytest.fixture( - params=[ - {"a": [1, 2, 3], "b": ["x", "y", "z"]}, - {"a": [1, 2, None], "b": ["x", "y", "z"]}, - {"a": [1, 2, 3], "b": pd.Categorical(["x", "y", None])}, - ] -) -def pandas_df(request): - data = request.param - return pd.DataFrame(data) - - -def assert_validity_equal(protocol_buffer, cudf_buffer, size, null, valid): - if null == _MaskKind.BYTEMASK: - protocol_mask = _protocol_buffer_to_cudf_buffer(protocol_buffer) - assert_eq( - as_column(protocol_mask, dtype="bool"), - as_column(cudf_buffer, dtype="bool"), - ) - elif null == _MaskKind.BITMASK: - protocol_mask = _protocol_buffer_to_cudf_buffer(protocol_buffer) - cudf_mask = cudf_buffer - assert_eq( - build_column( - as_buffer(cp.zeros(10, dtype="int8")), - "int8", - size=size, - mask=protocol_mask, - children=(), - ), - build_column( - as_buffer(cp.zeros(10, dtype="int8")), - "int8", - size=size, - mask=cudf_mask, - children=(), - ), - ) - else: - raise NotImplementedError() - - -def assert_buffer_equal(buffer_and_dtype: tuple[_CuDFBuffer, Any], cudfcol): - buf, dtype = buffer_and_dtype - device_id = cp.asarray(cudfcol.data).device.id - assert buf.__dlpack_device__() == (2, device_id) - col_from_buf = build_column( - _protocol_buffer_to_cudf_buffer(buf), - protocol_dtype_to_cupy_dtype(dtype), - ) - # check that non null values are the equals as nulls are represented - # by sentinel values in the buffer. - # FIXME: In gh-10202 some minimal fixes were added to unblock CI. But - # currently only non-null values are compared, null positions are - # unchecked. - non_null_idxs = cudfcol.notnull() - assert_eq( - col_from_buf.apply_boolean_mask(non_null_idxs), - cudfcol.apply_boolean_mask(non_null_idxs), - ) - array_from_dlpack = cp.from_dlpack(buf.__dlpack__()).get() - col_array = cp.asarray(cudfcol.data_array_view(mode="read")).get() - assert_eq( - array_from_dlpack[non_null_idxs.values_host].flatten(), - col_array[non_null_idxs.values_host].flatten(), - ) - - -def assert_column_equal(col: _CuDFColumn, cudfcol): - assert col.size() == cudfcol.size - assert col.offset == 0 - assert col.null_count == cudfcol.null_count - assert col.num_chunks() == 1 - if col.null_count == 0: - pytest.raises(RuntimeError, col._get_validity_buffer) - assert col.get_buffers()["validity"] is None - else: - assert_validity_equal( - col.get_buffers()["validity"][0], - cudfcol.mask, - cudfcol.size, - *col.describe_null, - ) - - if col.dtype[0] == _DtypeKind.CATEGORICAL: - assert_buffer_equal(col.get_buffers()["data"], cudfcol.codes) - assert col.get_buffers()["offsets"] is None - - elif col.dtype[0] == _DtypeKind.STRING: - chars_col = build_column(data=cudfcol.data, dtype="int8") - assert_buffer_equal(col.get_buffers()["data"], chars_col) - assert_buffer_equal(col.get_buffers()["offsets"], cudfcol.children[0]) - - else: - assert_buffer_equal(col.get_buffers()["data"], cudfcol) - assert col.get_buffers()["offsets"] is None - - if col.null_count == 0: - assert col.describe_null == (0, None) - else: - assert col.describe_null == (3, 0) - - -def assert_dataframe_equal(dfo: DataFrameObject, df: cudf.DataFrame): - assert dfo.num_columns() == len(df.columns) - assert dfo.num_rows() == len(df) - assert dfo.num_chunks() == 1 - assert dfo.column_names() == tuple(df.columns) - for col in df.columns: - assert_column_equal(dfo.get_column_by_name(col), df[col]._column) - - -def assert_from_dataframe_equals(dfobj, allow_copy): - df2 = from_dataframe(dfobj, allow_copy=allow_copy) - - assert_dataframe_equal(dfobj.__dataframe__(allow_copy), df2) - if isinstance(dfobj, cudf.DataFrame): - assert_eq(dfobj, df2) - - elif isinstance(dfobj, pd.DataFrame): - assert_eq(cudf.DataFrame(dfobj), df2) - - else: - raise TypeError(f"{type(dfobj)} not supported yet.") - - -def test_from_dataframe_exception(pandas_df): - exception_msg = "This operation must copy data from CPU to GPU." - " Set `allow_copy=True` to allow it." - with pytest.raises(TypeError, match=exception_msg): - from_dataframe(pandas_df) - - -def assert_df_unique_dtype_cols(data): - cdf = cudf.DataFrame(data=data) - assert_from_dataframe_equals(cdf, allow_copy=False) - assert_from_dataframe_equals(cdf, allow_copy=True) - - -def test_from_dataframe(): - data = dict(a=[1, 2, 3], b=[9, 10, 11]) - df1 = cudf.DataFrame(data=data) - df2 = cudf.from_dataframe(df1) - assert_eq(df1, df2) - - df3 = cudf.from_dataframe(df2) - assert_eq(df1, df3) - - -def test_int_dtype(): - data_int = dict(a=[1, 2, 3], b=[9, 10, 11]) - assert_df_unique_dtype_cols(data_int) - - -def test_float_dtype(): - data_float = dict(a=[1.5, 2.5, 3.5], b=[9.2, 10.5, 11.8]) - assert_df_unique_dtype_cols(data_float) - - -def test_categorical_dtype(): - cdf = cudf.DataFrame({"A": [1, 2, 5, 1]}) - cdf["A"] = cdf["A"].astype("category") - col = cdf.__dataframe__().get_column_by_name("A") - assert col.dtype[0] == _DtypeKind.CATEGORICAL - assert col.describe_categorical == (False, True, {0: 1, 1: 2, 2: 5}) - assert_from_dataframe_equals(cdf, allow_copy=False) - assert_from_dataframe_equals(cdf, allow_copy=True) - - -def test_bool_dtype(): - data_bool = dict(a=[True, True, False], b=[False, True, False]) - assert_df_unique_dtype_cols(data_bool) - - -def test_string_dtype(): - data_string = dict(a=["a", "b", "cdef", "", "g"]) - assert_df_unique_dtype_cols(data_string) - - -def test_mixed_dtype(): - data_mixed = dict( - int=[1, 2, 3], - float=[1.5, 2.5, 3.5], - bool=[True, False, True], - categorical=[5, 1, 5], - string=["rapidsai-cudf ", "", "df protocol"], - ) - assert_df_unique_dtype_cols(data_mixed) - - -def test_NA_int_dtype(): - data_int = dict( - a=[1, None, 3, None, 5], - b=[9, 10, None, 7, 8], - c=[6, 19, 20, 100, 1000], - ) - assert_df_unique_dtype_cols(data_int) - - -def test_NA_float_dtype(): - data_float = dict( - a=[1.4, None, 3.6, None, 5.2], - b=[9.7, 10.9, None, 7.8, 8.2], - c=[6.1, 19.2, 20.3, 100.4, 1000.5], - ) - assert_df_unique_dtype_cols(data_float) - - -def test_NA_categorical_dtype(): - df = cudf.DataFrame({"A": [1, 2, 5, 1]}) - df["B"] = df["A"].astype("category") - df.at[[1, 3], "B"] = None # Set two items to null - - # Some detailed testing for correctness of dtype and null handling: - col = df.__dataframe__().get_column_by_name("B") - assert col.dtype[0] == _DtypeKind.CATEGORICAL - assert col.null_count == 2 - assert col.describe_null == (3, 0) - assert col.num_chunks() == 1 - assert col.describe_categorical == (False, True, {0: 1, 1: 2, 2: 5}) - assert_from_dataframe_equals(df, allow_copy=False) - assert_from_dataframe_equals(df, allow_copy=True) - - -def test_NA_bool_dtype(): - data_bool = dict(a=[None, True, False], b=[False, None, None]) - assert_df_unique_dtype_cols(data_bool) - - -def test_NA_string_dtype(): - df = cudf.DataFrame({"A": ["a", "b", "cdef", "", "g"]}) - df["B"] = df["A"].astype("object") - df.at[1, "B"] = cudf.NA # Set one item to null - - # Test for correctness and null handling: - col = df.__dataframe__().get_column_by_name("B") - assert col.dtype[0] == _DtypeKind.STRING - assert col.null_count == 1 - assert col.describe_null == (3, 0) - assert col.num_chunks() == 1 - assert_from_dataframe_equals(df, allow_copy=False) - assert_from_dataframe_equals(df, allow_copy=True) - - -def test_NA_mixed_dtype(): - data_mixed = dict( - int=[1, None, 2, 3, 1000], - float=[None, 1.5, 2.5, 3.5, None], - bool=[True, None, False, None, None], - categorical=[5, 1, 5, 3, None], - string=[None, None, None, "df protocol", None], - ) - assert_df_unique_dtype_cols(data_mixed) - - -def test_from_cpu_df(pandas_df): - cudf.from_dataframe(pandas_df, allow_copy=True) diff --git a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py index 3e8b6d5786c..81e0f09f795 100644 --- a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py +++ b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py @@ -1221,25 +1221,6 @@ def test_intermediates_are_proxied(): assert isinstance(grouper, xpd.core.groupby.generic.DataFrameGroupBy) -def test_from_dataframe(): - cudf = pytest.importorskip("cudf") - from cudf.testing import assert_eq - - data = {"foo": [1, 2, 3], "bar": [4, 5, 6]} - - cudf_pandas_df = xpd.DataFrame(data) - cudf_df = cudf.DataFrame(data) - - # test construction of a cuDF DataFrame from an cudf_pandas DataFrame - assert_eq(cudf_df, cudf.DataFrame.from_pandas(cudf_pandas_df)) - assert_eq(cudf_df, cudf.from_dataframe(cudf_pandas_df)) - - # ideally the below would work as well, but currently segfaults - - # pd_df = pd.DataFrame(data) - # assert_eq(pd_df, pd.api.interchange.from_dataframe(cudf_pandas_df)) - - def test_multiindex_values_returns_1d_tuples(): mi = xpd.MultiIndex.from_tuples([(1, 2), (3, 4)]) result = mi.values From 4323ae4084a81efa7fa1d7d9b4a57ce3e92e7290 Mon Sep 17 00:00:00 2001 From: Gil Forsyth Date: Thu, 6 Feb 2025 10:03:23 -0500 Subject: [PATCH 004/129] Add build_type input field for `test.yaml` (#17925) Exposes `build_type` as an input in `test.yaml` so that `test.yaml` can be manually run against a specific branch/commit as needed. The default value is still `nightly`, and without maintainer intervention, that is what will run each night. xref rapidsai/build-planning#147 supercedes #17906 Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/17925 --- .github/workflows/test.yaml | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index b6b2caddeb8..08784358087 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -12,13 +12,16 @@ on: sha: required: true type: string + build_type: + type: string + default: nightly jobs: conda-cpp-checks: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -27,7 +30,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -35,7 +38,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -47,7 +50,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -59,7 +62,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -69,7 +72,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -79,7 +82,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -88,7 +91,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -100,7 +103,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -112,7 +115,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -121,7 +124,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -130,7 +133,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -139,7 +142,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -151,7 +154,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} @@ -160,7 +163,7 @@ jobs: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners with: - build_type: nightly + build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} From bc41d1edecad9b73f158adbe87f3c785871835dc Mon Sep 17 00:00:00 2001 From: Michael Schellenberger Costa Date: Thu, 6 Feb 2025 20:56:31 +0100 Subject: [PATCH 005/129] Add missing standard includes (#17928) With the upcoming CCCL release we moved some includes around and it seems that we relied on transitively including `` in some files. Fix that and include all the headers used in at least those two files Authors: - Michael Schellenberger Costa (https://github.com/miscco) - David Wendt (https://github.com/davidwendt) Approvers: - Bradley Dice (https://github.com/bdice) - Yunsong Wang (https://github.com/PointKernel) - Muhammad Haseeb (https://github.com/mhaseeb123) - David Wendt (https://github.com/davidwendt) URL: https://github.com/rapidsai/cudf/pull/17928 --- cpp/include/cudf/column/column_device_view.cuh | 3 ++- cpp/src/copying/pack.cpp | 6 ++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/cpp/include/cudf/column/column_device_view.cuh b/cpp/include/cudf/column/column_device_view.cuh index aacb5ccfede..990dfee2d17 100644 --- a/cpp/include/cudf/column/column_device_view.cuh +++ b/cpp/include/cudf/column/column_device_view.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -39,6 +39,7 @@ #include #include +#include #include /** diff --git a/cpp/src/copying/pack.cpp b/cpp/src/copying/pack.cpp index 0c6b7977752..869a83cf369 100644 --- a/cpp/src/copying/pack.cpp +++ b/cpp/src/copying/pack.cpp @@ -20,6 +20,12 @@ #include +#include +#include +#include +#include +#include + namespace cudf { namespace detail { From 6a032290eb8224802f2be8f9c8d6acf422b647f5 Mon Sep 17 00:00:00 2001 From: GALI PREM SAGAR Date: Thu, 6 Feb 2025 17:43:29 -0600 Subject: [PATCH 006/129] Patch `__init__` of `cudf` constructors to parse through `cudf.pandas` proxy objects (#17878) Fixes: https://github.com/rapidsai/cuml/issues/6232 This PR patches `Series`, `Index` and `DataFrame` constructors in such a way that true objects are extracted from `cudf.pandas` proxy objects if they are passed to any of these constructors. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/17878 --- python/cudf/cudf/pandas/_wrappers/pandas.py | 81 +++++++++++++++++++ python/cudf/cudf/pandas/module_accelerator.py | 4 +- .../cudf_pandas_tests/test_cudf_pandas.py | 62 ++++++++++++++ 3 files changed, 145 insertions(+), 2 deletions(-) diff --git a/python/cudf/cudf/pandas/_wrappers/pandas.py b/python/cudf/cudf/pandas/_wrappers/pandas.py index 526778b4ecb..5ec2b4b4a03 100644 --- a/python/cudf/cudf/pandas/_wrappers/pandas.py +++ b/python/cudf/cudf/pandas/_wrappers/pandas.py @@ -3,6 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 import abc import copyreg +import functools import importlib import os import pickle @@ -37,6 +38,7 @@ _FunctionProxy, _maybe_wrap_result, _Unusable, + is_proxy_object, make_final_proxy_type as _make_final_proxy_type, make_intermediate_proxy_type as _make_intermediate_proxy_type, register_proxy_func, @@ -1734,6 +1736,85 @@ def _unpickle_obj(pickled_args): return obj +# Save the original __init__ methods +_original_Series_init = cudf.Series.__init__ +_original_DataFrame_init = cudf.DataFrame.__init__ +_original_Index_init = cudf.Index.__init__ +_original_IndexMeta_call = cudf.core.index.IndexMeta.__call__ + + +def wrap_init(original_init): + @functools.wraps(original_init) + def wrapped_init(self, data=None, *args, **kwargs): + if is_proxy_object(data): + data = data.as_gpu_object() + if ( + isinstance(data, type(self)) + and len(args) == 0 + and len(kwargs) == 0 + ): + # This short-circuits the constructor to avoid + # unnecessary work when the data is already a + # proxy object of the same type. + # It is a common case in `cuml` and `xgboost`. + # For perf impact see: + # https://github.com/rapidsai/cudf/pull/17878/files#r1936469215 + self.__dict__.update(data.__dict__) + return + original_init(self, data, *args, **kwargs) + + return wrapped_init + + +def wrap_call(original_call): + @functools.wraps(original_call) + def wrapped_call(cls, data, *args, **kwargs): + if is_proxy_object(data): + data = data.as_gpu_object() + return original_call(cls, data, *args, **kwargs) + + return wrapped_call + + +@functools.wraps(_original_DataFrame_init) +def DataFrame_init_(self, data, index=None, columns=None, *args, **kwargs): + data_is_proxy = is_proxy_object(data) + + if data_is_proxy: + data = data.as_gpu_object() + if is_proxy_object(index): + index = index.as_gpu_object() + if is_proxy_object(columns): + columns = columns.as_cpu_object() + if ( + ( + (data_is_proxy and isinstance(data, type(self))) + and (index is None) + and (columns is None) + ) + and len(args) == 0 + and len(kwargs) == 0 + ): + self.__dict__.update(data.__dict__) + return + _original_DataFrame_init(self, data, index, columns, *args, **kwargs) + + +def initial_setup(): + """ + This is a one-time setup function that can contain + any initialization code that needs to be run once + when the module is imported. Currently, it is used + to wrap the __init__ methods and enable pandas compatibility mode. + """ + cudf.Series.__init__ = wrap_init(_original_Series_init) + cudf.Index.__init__ = wrap_init(_original_Index_init) + cudf.DataFrame.__init__ = DataFrame_init_ + cudf.core.index.IndexMeta.__call__ = wrap_call(_original_IndexMeta_call) + + cudf.set_option("mode.pandas_compatible", True) + + copyreg.dispatch_table[pd.Timestamp] = _reduce_obj # same reducer/unpickler can be used for Timedelta: copyreg.dispatch_table[pd.Timedelta] = _reduce_obj diff --git a/python/cudf/cudf/pandas/module_accelerator.py b/python/cudf/cudf/pandas/module_accelerator.py index 818971105cb..c4020887907 100644 --- a/python/cudf/cudf/pandas/module_accelerator.py +++ b/python/cudf/cudf/pandas/module_accelerator.py @@ -595,10 +595,10 @@ def install( ) mode = deduce_cudf_pandas_mode(slow_lib, fast_lib) if mode.use_fast_lib: - pandas_wrappers = importlib.import_module( + lib_wrappers = importlib.import_module( f".._wrappers.{mode.slow_lib}", __name__ ) - pandas_wrappers.cudf.set_option("mode.pandas_compatible", True) + lib_wrappers.initial_setup() try: (self,) = ( p diff --git a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py index 81e0f09f795..800702a6544 100644 --- a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py +++ b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py @@ -5,11 +5,13 @@ import collections import contextlib import copy +import cProfile import datetime import operator import os import pathlib import pickle +import pstats import subprocess import tempfile import time @@ -1910,6 +1912,66 @@ def test_series_dtype_property(): assert expected == actual +def assert_functions_called(profiler, functions): + # Process profiling data + stream = StringIO() + stats = pstats.Stats(profiler, stream=stream) + + # Get all called functions as (filename, lineno, func_name) + called_functions = {func[2] for func in stats.stats.keys()} + print(called_functions) + for func_str in functions: + assert func_str in called_functions + + +def test_cudf_series_from_cudf_pandas(): + s = xpd.Series([1, 2, 3]) + + with cProfile.Profile() as profiler: + gs = cudf.Series(s) + + assert_functions_called( + profiler, ["as_gpu_object", ""] + ) + + tm.assert_equal(s.as_gpu_object(), gs) + + +def test_cudf_dataframe_from_cudf_pandas(): + df = xpd.DataFrame({"a": [1, 2, 3], "b": [1, 2, 3]}) + + with cProfile.Profile() as profiler: + gdf = cudf.DataFrame(df) + + assert_functions_called( + profiler, ["as_gpu_object", ""] + ) + tm.assert_frame_equal(df.as_gpu_object(), gdf) + + df = xpd.DataFrame({"a": [1, 2, 3], "b": [1, 2, 3]}) + gdf = cudf.DataFrame( + {"a": xpd.Series([1, 2, 3]), "b": xpd.Series([1, 2, 3])} + ) + + tm.assert_frame_equal(df.as_gpu_object(), gdf) + + df = xpd.DataFrame({0: [1, 2, 3], 1: [1, 2, 3]}) + gdf = cudf.DataFrame( + [xpd.Series([1, 1]), xpd.Series([2, 2]), xpd.Series([3, 3])] + ) + + tm.assert_frame_equal(df.as_gpu_object(), gdf) + + +def test_cudf_index_from_cudf_pandas(): + idx = xpd.Index([1, 2, 3]) + with cProfile.Profile() as profiler: + gidx = cudf.Index(idx) + assert_functions_called(profiler, ["as_gpu_object"]) + + tm.assert_equal(idx.as_gpu_object(), gidx) + + def test_numpy_data_access(): s = pd.Series([1, 2, 3]) xs = xpd.Series([1, 2, 3]) From 6dc4a536897b3156a3b549a400113e0373798dc9 Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Fri, 7 Feb 2025 08:47:59 -0500 Subject: [PATCH 007/129] Remove deprecated nvtext::minhash_permuted APIs (#17939) Removes `nvtext::minhash_permuted` and `nvtext::minhash64_permuted` APIs deprecated in the previous release. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Shruti Shivakumar (https://github.com/shrshi) URL: https://github.com/rapidsai/cudf/pull/17939 --- cpp/include/nvtext/minhash.hpp | 30 +----------------------------- cpp/src/text/minhash.cu | 26 +------------------------- 2 files changed, 2 insertions(+), 54 deletions(-) diff --git a/cpp/include/nvtext/minhash.hpp b/cpp/include/nvtext/minhash.hpp index f0d5d9ecb5d..43f060fdafa 100644 --- a/cpp/include/nvtext/minhash.hpp +++ b/cpp/include/nvtext/minhash.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2023-2024, NVIDIA CORPORATION. + * Copyright (c) 2023-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -78,20 +78,6 @@ std::unique_ptr minhash( rmm::cuda_stream_view stream = cudf::get_default_stream(), rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); -/** - * @copydoc nvtext::minhash - * - * @deprecated Use nvtext::minhash() - */ -[[deprecated]] std::unique_ptr minhash_permuted( - cudf::strings_column_view const& input, - uint32_t seed, - cudf::device_span parameter_a, - cudf::device_span parameter_b, - cudf::size_type width, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - /** * @brief Returns the minhash values for each string * @@ -139,19 +125,5 @@ std::unique_ptr minhash64( rmm::cuda_stream_view stream = cudf::get_default_stream(), rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); -/** - * @copydoc nvtext::minhash64 - * - * @deprecated Use nvtext::minhash64() - */ -[[deprecated]] std::unique_ptr minhash64_permuted( - cudf::strings_column_view const& input, - uint64_t seed, - cudf::device_span parameter_a, - cudf::device_span parameter_b, - cudf::size_type width, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - /** @} */ // end of group } // namespace CUDF_EXPORT nvtext diff --git a/cpp/src/text/minhash.cu b/cpp/src/text/minhash.cu index 9ce17c36b1f..50c16c8ba6c 100644 --- a/cpp/src/text/minhash.cu +++ b/cpp/src/text/minhash.cu @@ -72,7 +72,7 @@ constexpr cudf::size_type blocks_per_string = 64; * * This kernel computes the hashes for each string using the seed and the specified * hash function. The width is used to compute rolling substrings to hash over. - * The hashes are stored in d_hashes to be used in the minhash_permuted_kernel. + * The hashes are stored in d_hashes to be used in the minhash_kernel. * * This kernel also counts the number of strings above the wide_string_threshold * and proactively initializes the output values for those strings. @@ -454,18 +454,6 @@ std::unique_ptr minhash(cudf::strings_column_view const& input, return detail::minhash(input, seed, parameter_a, parameter_b, width, stream, mr); } -std::unique_ptr minhash_permuted(cudf::strings_column_view const& input, - uint32_t seed, - cudf::device_span parameter_a, - cudf::device_span parameter_b, - cudf::size_type width, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::minhash(input, seed, parameter_a, parameter_b, width, stream, mr); -} - std::unique_ptr minhash64(cudf::strings_column_view const& input, uint64_t seed, cudf::device_span parameter_a, @@ -478,16 +466,4 @@ std::unique_ptr minhash64(cudf::strings_column_view const& input, return detail::minhash64(input, seed, parameter_a, parameter_b, width, stream, mr); } -std::unique_ptr minhash64_permuted(cudf::strings_column_view const& input, - uint64_t seed, - cudf::device_span parameter_a, - cudf::device_span parameter_b, - cudf::size_type width, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::minhash64(input, seed, parameter_a, parameter_b, width, stream, mr); -} - } // namespace nvtext From fdb7e7dee3c061748fce871d324a01c63ebd5128 Mon Sep 17 00:00:00 2001 From: Bradley Dice Date: Fri, 7 Feb 2025 09:38:37 -0800 Subject: [PATCH 008/129] Use shared-workflows branch-25.04 (#17943) This completes the migration to NVKS runners now that all libraries have been tested and https://github.com/rapidsai/shared-workflows/pull/273 has been merged. xref: https://github.com/rapidsai/build-infra/issues/184 Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) URL: https://github.com/rapidsai/cudf/pull/17943 --- .github/workflows/build.yaml | 28 +++++----- .github/workflows/pandas-tests.yaml | 2 +- .github/workflows/pr.yaml | 54 +++++++++---------- .../workflows/pr_issue_status_automation.yml | 8 +-- .github/workflows/test.yaml | 30 +++++------ .../trigger-breaking-change-alert.yaml | 2 +- 6 files changed, 62 insertions(+), 62 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 9bcd3a65a9d..11104037c5e 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -28,7 +28,7 @@ concurrency: jobs: cpp-build: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -37,7 +37,7 @@ jobs: python-build: needs: [cpp-build] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -46,7 +46,7 @@ jobs: upload-conda: needs: [cpp-build, python-build] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -57,7 +57,7 @@ jobs: if: github.ref_type == 'branch' needs: python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: arch: "amd64" branch: ${{ inputs.branch }} @@ -69,7 +69,7 @@ jobs: sha: ${{ inputs.sha }} wheel-build-libcudf: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: # build for every combination of arch and CUDA version, but only for the latest Python matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) @@ -81,7 +81,7 @@ jobs: wheel-publish-libcudf: needs: wheel-build-libcudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -92,7 +92,7 @@ jobs: wheel-build-pylibcudf: needs: [wheel-build-libcudf] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -102,7 +102,7 @@ jobs: wheel-publish-pylibcudf: needs: wheel-build-pylibcudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -113,7 +113,7 @@ jobs: wheel-build-cudf: needs: wheel-build-pylibcudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -123,7 +123,7 @@ jobs: wheel-publish-cudf: needs: wheel-build-cudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -134,7 +134,7 @@ jobs: wheel-build-dask-cudf: needs: wheel-build-cudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))])) @@ -146,7 +146,7 @@ jobs: wheel-publish-dask-cudf: needs: wheel-build-dask-cudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -157,7 +157,7 @@ jobs: wheel-build-cudf-polars: needs: wheel-build-pylibcudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))])) @@ -169,7 +169,7 @@ jobs: wheel-publish-cudf-polars: needs: wheel-build-cudf-polars secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-25.04 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} diff --git a/.github/workflows/pandas-tests.yaml b/.github/workflows/pandas-tests.yaml index 8730ae43ddf..fea393c549e 100644 --- a/.github/workflows/pandas-tests.yaml +++ b/.github/workflows/pandas-tests.yaml @@ -17,7 +17,7 @@ jobs: pandas-tests: # run the Pandas unit tests secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))])) diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 5c1352298fb..623ac196f06 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -42,7 +42,7 @@ jobs: - pandas-tests-diff - telemetry-setup secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-25.04 if: always() with: needs: ${{ toJSON(needs) }} @@ -70,7 +70,7 @@ jobs: changed-files: secrets: inherit needs: telemetry-setup - uses: rapidsai/shared-workflows/.github/workflows/changed-files.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/changed-files.yaml@branch-25.04 with: files_yaml: | test_cpp: @@ -123,48 +123,48 @@ jobs: checks: secrets: inherit needs: telemetry-setup - uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@branch-25.04 with: enable_check_generated_files: false ignored_pr_jobs: "telemetry-summarize spark-rapids-jni" conda-cpp-build: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-25.04 with: build_type: pull-request node_type: "cpu16" cpp-linters: secrets: inherit needs: checks - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: pull-request run_script: "ci/cpp_linters.sh" conda-cpp-checks: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-25.04 with: build_type: pull-request enable_check_symbols: true conda-cpp-tests: needs: [conda-cpp-build, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_cpp with: build_type: pull-request conda-python-build: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-25.04 with: build_type: pull-request conda-python-cudf-tests: needs: [conda-python-build, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python with: build_type: pull-request @@ -173,7 +173,7 @@ jobs: # Tests for dask_cudf, custreamz, cudf_kafka are separated for CI parallelism needs: [conda-python-build, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python with: build_type: pull-request @@ -181,7 +181,7 @@ jobs: conda-java-tests: needs: [conda-cpp-build, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_java with: build_type: pull-request @@ -192,7 +192,7 @@ jobs: static-configure: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: pull-request # Use the wheel container so we can skip conda solves and since our @@ -202,7 +202,7 @@ jobs: conda-notebook-tests: needs: [conda-python-build, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_notebooks with: build_type: pull-request @@ -213,7 +213,7 @@ jobs: docs-build: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: pull-request node_type: "gpu-l4-latest-1" @@ -223,7 +223,7 @@ jobs: wheel-build-libcudf: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: # build for every combination of arch and CUDA version, but only for the latest Python matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) @@ -233,21 +233,21 @@ jobs: wheel-build-pylibcudf: needs: [checks, wheel-build-libcudf] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: build_type: pull-request script: "ci/build_wheel_pylibcudf.sh" wheel-build-cudf: needs: wheel-build-pylibcudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: build_type: pull-request script: "ci/build_wheel_cudf.sh" wheel-tests-cudf: needs: [wheel-build-cudf, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python with: build_type: pull-request @@ -255,7 +255,7 @@ jobs: wheel-build-cudf-polars: needs: wheel-build-pylibcudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))])) @@ -264,7 +264,7 @@ jobs: wheel-tests-cudf-polars: needs: [wheel-build-cudf-polars, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". @@ -274,7 +274,7 @@ jobs: cudf-polars-polars-tests: needs: wheel-build-cudf-polars secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))])) @@ -283,7 +283,7 @@ jobs: wheel-build-dask-cudf: needs: wheel-build-cudf secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04 with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))])) @@ -292,7 +292,7 @@ jobs: wheel-tests-dask-cudf: needs: [wheel-build-dask-cudf, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". @@ -302,7 +302,7 @@ jobs: devcontainer: secrets: inherit needs: telemetry-setup - uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@branch-25.04 with: node_type: "cpu32" arch: '["amd64"]' @@ -314,7 +314,7 @@ jobs: unit-tests-cudf-pandas: needs: [wheel-build-cudf, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python || fromJSON(needs.changed-files.outputs.changed_file_groups).test_cudf_pandas with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". @@ -325,7 +325,7 @@ jobs: # run the Pandas unit tests using PR branch needs: [wheel-build-cudf, changed-files] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python || fromJSON(needs.changed-files.outputs.changed_file_groups).test_cudf_pandas with: # This selects "ARCH=amd64 + the latest supported Python + CUDA". @@ -337,7 +337,7 @@ jobs: pandas-tests-diff: # diff the results of running the Pandas unit tests and publish a job summary needs: pandas-tests - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: node_type: "cpu4" build_type: pull-request diff --git a/.github/workflows/pr_issue_status_automation.yml b/.github/workflows/pr_issue_status_automation.yml index b1bd2d4e768..44e48f691a2 100644 --- a/.github/workflows/pr_issue_status_automation.yml +++ b/.github/workflows/pr_issue_status_automation.yml @@ -23,7 +23,7 @@ on: jobs: get-project-id: - uses: rapidsai/shared-workflows/.github/workflows/project-get-item-id.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/project-get-item-id.yaml@branch-25.04 if: github.event.pull_request.state == 'open' secrets: inherit permissions: @@ -34,7 +34,7 @@ jobs: update-status: # This job sets the PR and its linked issues to "In Progress" status - uses: rapidsai/shared-workflows/.github/workflows/project-get-set-single-select-field.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/project-get-set-single-select-field.yaml@branch-25.04 if: ${{ github.event.pull_request.state == 'open' && needs.get-project-id.outputs.ITEM_PROJECT_ID != '' }} needs: get-project-id with: @@ -50,7 +50,7 @@ jobs: update-sprint: # This job sets the PR and its linked issues to the current "Weekly Sprint" - uses: rapidsai/shared-workflows/.github/workflows/project-get-set-iteration-field.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/project-get-set-iteration-field.yaml@branch-25.04 if: ${{ github.event.pull_request.state == 'open' && needs.get-project-id.outputs.ITEM_PROJECT_ID != '' }} needs: get-project-id with: @@ -79,7 +79,7 @@ jobs: update-release: # This job sets the PR and its linked issues to the release they are targeting - uses: rapidsai/shared-workflows/.github/workflows/project-get-set-single-select-field.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/project-get-set-single-select-field.yaml@branch-25.04 if: ${{ github.event.pull_request.state == 'open' && needs.get-project-id.outputs.ITEM_PROJECT_ID != '' }} needs: [get-project-id, process-branch-name] with: diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 08784358087..12f6d751493 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -19,7 +19,7 @@ on: jobs: conda-cpp-checks: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -28,7 +28,7 @@ jobs: enable_check_symbols: true conda-cpp-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -36,7 +36,7 @@ jobs: sha: ${{ inputs.sha }} conda-cpp-memcheck-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -48,7 +48,7 @@ jobs: run_script: "ci/test_cpp_memcheck.sh" static-configure: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -60,7 +60,7 @@ jobs: run_script: "ci/configure_cpp_static.sh" cpp-linters: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -70,7 +70,7 @@ jobs: file_to_upload: iwyu_results.txt conda-python-cudf-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -80,7 +80,7 @@ jobs: conda-python-other-tests: # Tests for dask_cudf, custreamz, cudf_kafka are separated for CI parallelism secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -89,7 +89,7 @@ jobs: script: "ci/test_python_other.sh" conda-java-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -101,7 +101,7 @@ jobs: run_script: "ci/test_java.sh" conda-notebook-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -113,7 +113,7 @@ jobs: run_script: "ci/test_notebooks.sh" wheel-tests-cudf: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -122,7 +122,7 @@ jobs: script: ci/test_wheel_cudf.sh wheel-tests-dask-cudf: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -131,7 +131,7 @@ jobs: script: ci/test_wheel_dask_cudf.sh unit-tests-cudf-pandas: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -140,7 +140,7 @@ jobs: script: ci/cudf_pandas_scripts/run_tests.sh third-party-integration-tests-cudf-pandas: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -152,7 +152,7 @@ jobs: ci/cudf_pandas_scripts/third-party-integration/test.sh python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml wheel-tests-cudf-polars: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} @@ -161,7 +161,7 @@ jobs: script: "ci/test_wheel_cudf_polars.sh" cudf-polars-polars-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04 with: build_type: ${{ inputs.build_type }} branch: ${{ inputs.branch }} diff --git a/.github/workflows/trigger-breaking-change-alert.yaml b/.github/workflows/trigger-breaking-change-alert.yaml index 7b5b4810fb6..9764c62c15c 100644 --- a/.github/workflows/trigger-breaking-change-alert.yaml +++ b/.github/workflows/trigger-breaking-change-alert.yaml @@ -12,7 +12,7 @@ jobs: trigger-notifier: if: contains(github.event.pull_request.labels.*.name, 'breaking') secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/breaking-change-alert.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/breaking-change-alert.yaml@branch-25.04 with: sender_login: ${{ github.event.sender.login }} sender_avatar: ${{ github.event.sender.avatar_url }} From 61e47bb4c23022c1256f419fc483576385396773 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Fri, 7 Feb 2025 17:06:52 -0800 Subject: [PATCH 009/129] Fix DataFrame/Series.rank for int and null data in mode.pandas_compatible (#17954) closes https://github.com/rapidsai/cudf/issues/17948 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17954 --- python/cudf/cudf/core/indexed_frame.py | 22 ++++++++++++---------- python/cudf/cudf/tests/test_rank.py | 14 +++++++++++++- 2 files changed, 25 insertions(+), 11 deletions(-) diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index 742f1d43ee1..589dc580ba1 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -5027,7 +5027,7 @@ def repeat(self, repeats, axis=None): def astype( self, - dtype: dict[Any, Dtype], + dtype: Dtype | dict[Any, Dtype], copy: bool = False, errors: Literal["raise", "ignore"] = "raise", ) -> Self: @@ -6340,13 +6340,13 @@ def _preprocess_subset(self, subset) -> set[abc.Hashable]: @_performance_tracking def rank( self, - axis=0, - method="average", - numeric_only=False, - na_option="keep", - ascending=True, - pct=False, - ): + axis: Literal[0, "index"] = 0, + method: Literal["average", "min", "max", "first", "dense"] = "average", + numeric_only: bool = False, + na_option: Literal["keep", "top", "bottom"] = "keep", + ascending: bool = True, + pct: bool = False, + ) -> Self: """ Compute numerical data ranks (1 through n) along axis. @@ -6404,7 +6404,7 @@ def rank( if numeric_only: if isinstance( source, cudf.Series - ) and not _is_non_decimal_numeric_dtype(self.dtype): + ) and not _is_non_decimal_numeric_dtype(self.dtype): # type: ignore[attr-defined] raise TypeError( "Series.rank does not allow numeric_only=True with " "non-numeric dtype." @@ -6416,7 +6416,7 @@ def rank( ) source = self._get_columns_by_label(numeric_cols) if source.empty: - return source.astype("float64") + return source.astype(np.dtype(np.float64)) elif source._num_columns != num_cols: dropped_cols = True @@ -6449,6 +6449,8 @@ def rank( else plc.types.NullPolicy.INCLUDE ) + if cudf.get_option("mode.pandas_compatible"): + source = source.nans_to_nulls() with acquire_spill_lock(): result_columns = [ libcudf.column.Column.from_pylibcudf( diff --git a/python/cudf/cudf/tests/test_rank.py b/python/cudf/cudf/tests/test_rank.py index 1d9c6690f14..fdb005d0ba9 100644 --- a/python/cudf/cudf/tests/test_rank.py +++ b/python/cudf/cudf/tests/test_rank.py @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from itertools import chain, combinations_with_replacement, product @@ -6,6 +6,7 @@ import pandas as pd import pytest +import cudf from cudf import DataFrame from cudf.testing import assert_eq from cudf.testing._utils import assert_exceptions_equal @@ -151,3 +152,14 @@ def test_series_rank_combinations(elem, dtype): ranked_ps = df["a"].rank(method="first") # Check assert_eq(ranked_ps, ranked_gs) + + +@pytest.mark.parametrize("klass", ["Series", "DataFrame"]) +def test_int_nan_pandas_compatible(klass): + data = [3, 6, 1, 1, None, 6] + pd_obj = getattr(pd, klass)(data) + cudf_obj = getattr(cudf, klass)(data) + with cudf.option_context("mode.pandas_compatible", True): + result = cudf_obj.rank() + expected = pd_obj.rank() + assert_eq(result, expected) From 428dc188cab5a51c1e15fb90c93a231ad95b7be2 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Fri, 7 Feb 2025 18:44:09 -0800 Subject: [PATCH 010/129] More avoid cudf.dtype internally in favor of pre-defined, supported types (#17918) Continuation of https://github.com/rapidsai/cudf/pull/17839 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17918 --- python/cudf/cudf/core/buffer/buffer.py | 5 +- python/cudf/cudf/core/dtypes.py | 5 -- python/cudf/cudf/core/scalar.py | 8 +-- python/cudf/cudf/core/tools/numeric.py | 2 +- python/cudf/cudf/io/csv.py | 99 ++++++++++++++------------ python/cudf/cudf/io/parquet.py | 6 +- python/cudf/cudf/utils/dtypes.py | 15 ++-- 7 files changed, 72 insertions(+), 68 deletions(-) diff --git a/python/cudf/cudf/core/buffer/buffer.py b/python/cudf/cudf/core/buffer/buffer.py index 625938ca168..d19176c71c2 100644 --- a/python/cudf/cudf/core/buffer/buffer.py +++ b/python/cudf/cudf/core/buffer/buffer.py @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from __future__ import annotations @@ -13,7 +13,6 @@ import pylibcudf import rmm -import cudf from cudf.core.abc import Serializable from cudf.utils.string import format_bytes @@ -504,7 +503,7 @@ def get_ptr_and_size(array_interface: Mapping) -> tuple[int, int]: shape = array_interface["shape"] or (1,) strides = array_interface["strides"] - itemsize = cudf.dtype(array_interface["typestr"]).itemsize + itemsize = numpy.dtype(array_interface["typestr"]).itemsize if strides is None or pylibcudf.column.is_c_contiguous( shape, strides, itemsize ): diff --git a/python/cudf/cudf/core/dtypes.py b/python/cudf/cudf/core/dtypes.py index 6e83d4d7051..f7ad49aed9f 100644 --- a/python/cudf/cudf/core/dtypes.py +++ b/python/cudf/cudf/core/dtypes.py @@ -64,11 +64,6 @@ def dtype(arbitrary): raise TypeError(f"Unsupported type {np_dtype}") return np_dtype - if isinstance(arbitrary, str) and arbitrary in {"hex", "hex32", "hex64"}: - # read_csv only accepts "hex" - # e.g. test_csv_reader_hexadecimals, test_csv_reader_hexadecimal_overflow - return arbitrary - # use `pandas_dtype` to try and interpret # `arbitrary` as a Pandas extension type. # Return the corresponding NumPy/cuDF type. diff --git a/python/cudf/cudf/core/scalar.py b/python/cudf/cudf/core/scalar.py index 533eeafedd4..d78ea83d578 100644 --- a/python/cudf/cudf/core/scalar.py +++ b/python/cudf/cudf/core/scalar.py @@ -476,16 +476,16 @@ def __repr__(self) -> str: # https://github.com/numpy/numpy/issues/17552 return f"{self.__class__.__name__}({self.value!s}, dtype={self.dtype})" - def _binop_result_dtype_or_error(self, other, op): + def _binop_result_dtype_or_error(self, other, op) -> np.dtype: if op in {"__eq__", "__ne__", "__lt__", "__gt__", "__le__", "__ge__"}: - return np.bool_ + return np.dtype(np.bool_) out_dtype = get_allowed_combinations_for_operator( self.dtype, other.dtype, op ) # datetime handling - if out_dtype in {"M", "m"}: + if out_dtype.kind in {"M", "m"}: if self.dtype.char in {"M", "m"} and other.dtype.char not in { "M", "m", @@ -505,7 +505,7 @@ def _binop_result_dtype_or_error(self, other, op): return np.dtype(f"m8[{res}]") return np.result_type(self.dtype, other.dtype) - return cudf.dtype(out_dtype) + return out_dtype def _binaryop(self, other, op: str): if is_scalar(other): diff --git a/python/cudf/cudf/core/tools/numeric.py b/python/cudf/cudf/core/tools/numeric.py index 271f72556f3..9a4d773d5d6 100644 --- a/python/cudf/cudf/core/tools/numeric.py +++ b/python/cudf/cudf/core/tools/numeric.py @@ -174,7 +174,7 @@ def to_numeric( type_set = list(np.typecodes["UnsignedInteger"]) for t in type_set: - downcast_dtype = cudf.dtype(t) + downcast_dtype = np.dtype(t) if downcast_dtype.itemsize <= col.dtype.itemsize: if col.can_cast_safely(downcast_dtype): col = col.cast(downcast_dtype) diff --git a/python/cudf/cudf/io/csv.py b/python/cudf/cudf/io/csv.py index 56d83aa49a8..f83bbb5a8fa 100644 --- a/python/cudf/cudf/io/csv.py +++ b/python/cudf/cudf/io/csv.py @@ -7,7 +7,7 @@ import warnings from collections import abc from io import BytesIO, StringIO -from typing import cast +from typing import TYPE_CHECKING, cast import numpy as np import pandas as pd @@ -16,7 +16,7 @@ import cudf from cudf._lib.column import Column -from cudf.api.types import is_hashable, is_scalar +from cudf.api.types import is_scalar from cudf.core.buffer import acquire_spill_lock from cudf.core.column_accessor import ColumnAccessor from cudf.utils import ioutils @@ -26,6 +26,10 @@ ) from cudf.utils.performance_tracking import _performance_tracking +if TYPE_CHECKING: + from cudf._typing import DtypeObj + + _CSV_HEX_TYPE_MAP = { "hex": np.dtype("int64"), "hex64": np.dtype("int64"), @@ -158,33 +162,49 @@ def read_csv( header = 0 hex_cols: list[abc.Hashable] = [] - new_dtypes: list[plc.DataType] | dict[abc.Hashable, plc.DataType] = [] + cudf_dtypes: list[DtypeObj] | dict[abc.Hashable, DtypeObj] | DtypeObj = [] + plc_dtypes: list[plc.DataType] | dict[abc.Hashable, plc.DataType] = [] if dtype is not None: if isinstance(dtype, abc.Mapping): - new_dtypes = {} + plc_dtypes = {} + cudf_dtypes = {} for k, col_type in dtype.items(): - if is_hashable(col_type) and col_type in _CSV_HEX_TYPE_MAP: + if isinstance(col_type, str) and col_type in _CSV_HEX_TYPE_MAP: col_type = _CSV_HEX_TYPE_MAP[col_type] hex_cols.append(str(k)) - new_dtypes[k] = _get_plc_data_type_from_dtype( - cudf.dtype(col_type) - ) - elif cudf.api.types.is_scalar(dtype) or isinstance( - dtype, (np.dtype, pd.api.extensions.ExtensionDtype, type) + cudf_dtype = cudf.dtype(col_type) + cudf_dtypes[k] = cudf_dtype + plc_dtypes[k] = _get_plc_data_type_from_dtype(cudf_dtype) + elif isinstance( + dtype, + ( + str, + np.dtype, + pd.api.extensions.ExtensionDtype, + cudf.core.dtypes._BaseDtype, + type, + ), ): - if is_hashable(dtype) and dtype in _CSV_HEX_TYPE_MAP: + if isinstance(dtype, str) and dtype in _CSV_HEX_TYPE_MAP: dtype = _CSV_HEX_TYPE_MAP[dtype] hex_cols.append(0) - - cast(list, new_dtypes).append(_get_plc_data_type_from_dtype(dtype)) + else: + dtype = cudf.dtype(dtype) + cudf_dtypes = dtype + cast(list, plc_dtypes).append(_get_plc_data_type_from_dtype(dtype)) elif isinstance(dtype, abc.Collection): for index, col_dtype in enumerate(dtype): - if is_hashable(col_dtype) and col_dtype in _CSV_HEX_TYPE_MAP: + if ( + isinstance(col_dtype, str) + and col_dtype in _CSV_HEX_TYPE_MAP + ): col_dtype = _CSV_HEX_TYPE_MAP[col_dtype] hex_cols.append(index) - - new_dtypes.append(_get_plc_data_type_from_dtype(col_dtype)) + else: + col_dtype = cudf.dtype(col_dtype) + cudf_dtypes.append(col_dtype) + plc_dtypes.append(_get_plc_data_type_from_dtype(col_dtype)) else: raise ValueError( "dtype should be a scalar/str/list-like/dict-like" @@ -243,7 +263,7 @@ def read_csv( if hex_cols is not None: options.set_parse_hex(list(hex_cols)) - options.set_dtypes(new_dtypes) + options.set_dtypes(plc_dtypes) if true_values is not None: options.set_true_values([str(val) for val in true_values]) @@ -266,15 +286,21 @@ def read_csv( ca = ColumnAccessor(data, rangeindex=len(data) == 0) df = cudf.DataFrame._from_data(ca) - if isinstance(dtype, abc.Mapping): - for k, v in dtype.items(): - if isinstance(cudf.dtype(v), cudf.CategoricalDtype): - df._data[str(k)] = df._data[str(k)].astype(v) - elif dtype == "category" or isinstance(dtype, cudf.CategoricalDtype): + # Cast result to categorical if specified in dtype= + # since categorical is not handled in pylibcudf + if isinstance(cudf_dtypes, dict): + to_category = { + k: v + for k, v in cudf_dtypes.items() + if isinstance(v, cudf.CategoricalDtype) + } + if to_category: + df = df.astype(to_category) + elif isinstance(cudf_dtypes, cudf.CategoricalDtype): df = df.astype(dtype) - elif isinstance(dtype, abc.Collection) and not is_scalar(dtype): - for index, col_dtype in enumerate(dtype): - if isinstance(cudf.dtype(col_dtype), cudf.CategoricalDtype): + elif isinstance(cudf_dtypes, list): + for index, col_dtype in enumerate(cudf_dtypes): + if isinstance(col_dtype, cudf.CategoricalDtype): col_name = df._column_names[index] df._data[col_name] = df._data[col_name].astype(col_dtype) @@ -527,30 +553,11 @@ def _validate_args( ) -def _get_plc_data_type_from_dtype(dtype) -> plc.DataType: +def _get_plc_data_type_from_dtype(dtype: DtypeObj) -> plc.DataType: # TODO: Remove this work-around Dictionary types # in libcudf are fully mapped to categorical columns: # https://github.com/rapidsai/cudf/issues/3960 if isinstance(dtype, cudf.CategoricalDtype): + # TODO: should we do this generally in dtype_to_pylibcudf_type? dtype = dtype.categories.dtype - elif dtype == "category": - dtype = "str" - - if isinstance(dtype, str): - if dtype == "date32": - return plc.DataType(plc.types.TypeId.TIMESTAMP_DAYS) - elif dtype in ("date", "date64"): - return plc.DataType(plc.types.TypeId.TIMESTAMP_MILLISECONDS) - elif dtype == "timestamp": - return plc.DataType(plc.types.TypeId.TIMESTAMP_MILLISECONDS) - elif dtype == "timestamp[us]": - return plc.DataType(plc.types.TypeId.TIMESTAMP_MICROSECONDS) - elif dtype == "timestamp[s]": - return plc.DataType(plc.types.TypeId.TIMESTAMP_SECONDS) - elif dtype == "timestamp[ms]": - return plc.DataType(plc.types.TypeId.TIMESTAMP_MILLISECONDS) - elif dtype == "timestamp[ns]": - return plc.DataType(plc.types.TypeId.TIMESTAMP_NANOSECONDS) - - dtype = cudf.dtype(dtype) return dtype_to_pylibcudf_type(dtype) diff --git a/python/cudf/cudf/io/parquet.py b/python/cudf/cudf/io/parquet.py index a7c7136ad4c..bcc9aacd2a7 100644 --- a/python/cudf/cudf/io/parquet.py +++ b/python/cudf/cudf/io/parquet.py @@ -527,7 +527,7 @@ def write_to_dataset( return metadata -def _parse_metadata(meta) -> tuple[bool, Any, Any]: +def _parse_metadata(meta) -> tuple[bool, Any, None | np.dtype]: file_is_range_index = False file_index_cols = None file_column_dtype = None @@ -541,7 +541,7 @@ def _parse_metadata(meta) -> tuple[bool, Any, Any]: ): file_is_range_index = True if "column_indexes" in meta and len(meta["column_indexes"]) == 1: - file_column_dtype = meta["column_indexes"][0]["numpy_type"] + file_column_dtype = np.dtype(meta["column_indexes"][0]["numpy_type"]) return file_is_range_index, file_index_cols, file_column_dtype @@ -2368,6 +2368,6 @@ def _process_metadata( df.index.names = index_col if df._num_columns == 0 and column_index_type is not None: - df._data.label_dtype = cudf.dtype(column_index_type) + df._data.label_dtype = column_index_type return df diff --git a/python/cudf/cudf/utils/dtypes.py b/python/cudf/cudf/utils/dtypes.py index 20ec5dce12e..c545b840c0e 100644 --- a/python/cudf/cudf/utils/dtypes.py +++ b/python/cudf/cudf/utils/dtypes.py @@ -430,7 +430,9 @@ def _get_nan_for_dtype(dtype: DtypeObj) -> DtypeObj: return np.float64("nan") -def get_allowed_combinations_for_operator(dtype_l, dtype_r, op): +def get_allowed_combinations_for_operator( + dtype_l: np.dtype, dtype_r: np.dtype, op: str +) -> np.dtype: error = TypeError( f"{op} not supported between {dtype_l} and {dtype_r} scalars" ) @@ -456,18 +458,19 @@ def get_allowed_combinations_for_operator(dtype_l, dtype_r, op): # special rules for string if dtype_l == "object" or dtype_r == "object": if (dtype_l == dtype_r == "object") and op == "__add__": - return "str" + return CUDF_STRING_DTYPE else: raise error # Check if we can directly operate for valid_combo in allowed: - ltype, rtype, outtype = valid_combo - if np.can_cast(dtype_l.char, ltype) and np.can_cast( - dtype_r.char, rtype + ltype, rtype, outtype = valid_combo # type: ignore[misc] + if np.can_cast(dtype_l.char, ltype) and np.can_cast( # type: ignore[has-type] + dtype_r.char, + rtype, # type: ignore[has-type] ): - return outtype + return np.dtype(outtype) # type: ignore[has-type] raise error From 218d67da490224a24e20ad0a917fee2cb59bcb2c Mon Sep 17 00:00:00 2001 From: GALI PREM SAGAR Date: Mon, 10 Feb 2025 07:40:06 -0600 Subject: [PATCH 011/129] Enable third party library integration tests in CI with `cudf.pandas` (#17936) This PR enables 3rd party library integration tests that are run with `cudf.pandas` enabled. Fixes: https://github.com/rapidsai/cuml/issues/6301 Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Mike Sarahan (https://github.com/msarahan) - Matthew Murray (https://github.com/Matt711) URL: https://github.com/rapidsai/cudf/pull/17936 --- .github/workflows/pr.yaml | 14 ++++ .../third-party-integration/test.sh | 32 ++++++-- python/cudf/cudf/pandas/_wrappers/pandas.py | 82 ++++++++++++++++++- .../dependencies.yaml | 2 +- .../tests/test_stumpy_distributed.py | 4 +- 5 files changed, 122 insertions(+), 12 deletions(-) diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 623ac196f06..e36b3e2ede4 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -41,6 +41,7 @@ jobs: - pandas-tests - pandas-tests-diff - telemetry-setup + - third-party-integration-tests-cudf-pandas secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-25.04 if: always() @@ -321,6 +322,19 @@ jobs: matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))])) build_type: pull-request script: ci/cudf_pandas_scripts/run_tests.sh + third-party-integration-tests-cudf-pandas: + needs: conda-python-build + secrets: inherit + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + with: + build_type: pull-request + branch: ${{ inputs.branch }} + date: ${{ inputs.date }} + sha: ${{ inputs.sha }} + node_type: "gpu-l4-latest-1" + container_image: "rapidsai/ci-conda:latest" + run_script: | + ci/cudf_pandas_scripts/third-party-integration/test.sh python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml pandas-tests: # run the Pandas unit tests using PR branch needs: [wheel-build-cudf, changed-files] diff --git a/ci/cudf_pandas_scripts/third-party-integration/test.sh b/ci/cudf_pandas_scripts/third-party-integration/test.sh index cf0a16fb3cb..894ba204f24 100755 --- a/ci/cudf_pandas_scripts/third-party-integration/test.sh +++ b/ci/cudf_pandas_scripts/third-party-integration/test.sh @@ -21,22 +21,38 @@ main() { LIBS=${LIBS#[} LIBS=${LIBS%]} + if [ "$RAPIDS_BUILD_TYPE" == "pull-request" ]; then + rapids-logger "Downloading artifacts from this pr jobs" + CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) + PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python) + fi + ANY_FAILURES=0 for lib in ${LIBS//,/ }; do lib=$(echo "$lib" | tr -d '""') echo "Running tests for library $lib" - CUDA_VERSION=$(if [ "$lib" = "tensorflow" ]; then echo "11.8"; else echo "${RAPIDS_CUDA_VERSION%.*}"; fi) . /opt/conda/etc/profile.d/conda.sh - - rapids-logger "Generate Python testing dependencies" - rapids-dependency-file-generator \ - --config "$dependencies_yaml" \ - --output conda \ - --file-key "test_${lib}" \ - --matrix "cuda=${CUDA_VERSION};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml + # Check the value of RAPIDS_BUILD_TYPE + if [ "$RAPIDS_BUILD_TYPE" == "pull-request" ]; then + rapids-logger "Generate Python testing dependencies" + rapids-dependency-file-generator \ + --config "$dependencies_yaml" \ + --output conda \ + --file-key "test_${lib}" \ + --matrix "cuda=${CUDA_VERSION};arch=$(arch);py=${RAPIDS_PY_VERSION}" \ + --prepend-channel "${CPP_CHANNEL}" \ + --prepend-channel "${PYTHON_CHANNEL}" | tee env.yaml + else + rapids-logger "Generate Python testing dependencies" + rapids-dependency-file-generator \ + --config "$dependencies_yaml" \ + --output conda \ + --file-key "test_${lib}" \ + --matrix "cuda=${CUDA_VERSION};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml + fi rapids-mamba-retry env create --yes -f env.yaml -n test diff --git a/python/cudf/cudf/pandas/_wrappers/pandas.py b/python/cudf/cudf/pandas/_wrappers/pandas.py index 5ec2b4b4a03..c65e058cd62 100644 --- a/python/cudf/cudf/pandas/_wrappers/pandas.py +++ b/python/cudf/cudf/pandas/_wrappers/pandas.py @@ -1741,6 +1741,11 @@ def _unpickle_obj(pickled_args): _original_DataFrame_init = cudf.DataFrame.__init__ _original_Index_init = cudf.Index.__init__ _original_IndexMeta_call = cudf.core.index.IndexMeta.__call__ +_original_from_pandas = cudf.from_pandas +_original_DataFrame_from_pandas = cudf.DataFrame.from_pandas +_original_Series_from_pandas = cudf.Series.from_pandas +_original_Index_from_pandas = cudf.BaseIndex.from_pandas +_original_MultiIndex_from_pandas = cudf.MultiIndex.from_pandas def wrap_init(original_init): @@ -1776,8 +1781,69 @@ def wrapped_call(cls, data, *args, **kwargs): return wrapped_call +def wrap_from_pandas(original_call): + @functools.wraps(original_call) + def wrapped_from_pandas(obj, *args, **kwargs): + if is_proxy_object(obj): + obj = obj.as_gpu_object() + return obj + return original_call(obj, *args, **kwargs) + + return wrapped_from_pandas + + +def wrap_from_pandas_dataframe(original_call): + @functools.wraps(original_call) + def wrapped_from_pandas_dataframe(dataframe, *args, **kwargs): + if is_proxy_object(dataframe): + dataframe = dataframe.as_gpu_object() + if isinstance(dataframe, cudf.DataFrame): + return dataframe + return original_call(dataframe, *args, **kwargs) + + return wrapped_from_pandas_dataframe + + +def wrap_from_pandas_series(original_call): + @functools.wraps(original_call) + def wrapped_from_pandas_series(s, *args, **kwargs): + if is_proxy_object(s): + s = s.as_gpu_object() + if isinstance(s, cudf.Series): + return s + return original_call(s, *args, **kwargs) + + return wrapped_from_pandas_series + + +def wrap_from_pandas_index(original_call): + @functools.wraps(original_call) + def wrapped_from_pandas_index(index, *args, **kwargs): + if is_proxy_object(index): + index = index.as_gpu_object() + if isinstance(index, cudf.core.index.BaseIndex): + return index + return original_call(index, *args, **kwargs) + + return wrapped_from_pandas_index + + +def wrap_from_pandas_multiindex(original_call): + @functools.wraps(original_call) + def wrapped_from_pandas_multiindex(multiindex, *args, **kwargs): + if is_proxy_object(multiindex): + multiindex = multiindex.as_gpu_object() + if isinstance(multiindex, cudf.MultiIndex): + return multiindex + return original_call(multiindex, *args, **kwargs) + + return wrapped_from_pandas_multiindex + + @functools.wraps(_original_DataFrame_init) -def DataFrame_init_(self, data, index=None, columns=None, *args, **kwargs): +def DataFrame_init_( + self, data=None, index=None, columns=None, *args, **kwargs +): data_is_proxy = is_proxy_object(data) if data_is_proxy: @@ -1811,7 +1877,19 @@ def initial_setup(): cudf.Index.__init__ = wrap_init(_original_Index_init) cudf.DataFrame.__init__ = DataFrame_init_ cudf.core.index.IndexMeta.__call__ = wrap_call(_original_IndexMeta_call) - + cudf.from_pandas = wrap_from_pandas(_original_from_pandas) + cudf.DataFrame.from_pandas = wrap_from_pandas_dataframe( + _original_DataFrame_from_pandas + ) + cudf.Series.from_pandas = wrap_from_pandas_series( + _original_Series_from_pandas + ) + cudf.BaseIndex.from_pandas = wrap_from_pandas_index( + _original_Index_from_pandas + ) + cudf.MultiIndex.from_pandas = wrap_from_pandas_multiindex( + _original_MultiIndex_from_pandas + ) cudf.set_option("mode.pandas_compatible", True) diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml index 977d25184b5..356e7ac4494 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml @@ -262,7 +262,7 @@ dependencies: packages: - pip - pip: - - ibis-framework[pandas] + - ibis-framework[pandas]<10.0.0 test_hvplot: common: - output_types: conda diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_stumpy_distributed.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_stumpy_distributed.py index f275659288e..6736608015e 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_stumpy_distributed.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_stumpy_distributed.py @@ -1,4 +1,4 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. import numpy as np import pandas as pd @@ -30,6 +30,7 @@ def dask_client(): yield dask_client +@pytest.mark.skip(reason="TODO: Fix these stumpy tests to work with dask") def test_1d_distributed(dask_client): rng = np.random.default_rng(seed=42) ts = pd.Series(rng.random(100)) @@ -37,6 +38,7 @@ def test_1d_distributed(dask_client): return stumpy.stumped(dask_client, ts, m) +@pytest.mark.skip(reason="TODO: Fix these stumpy tests to work with dask") def test_multidimensional_distributed_timeseries(dask_client): rng = np.random.default_rng(seed=42) # Each row represents data from a different dimension while each column represents From 94ac29e9174aa8165f2ed3b6e1af33f90b607e52 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Mon, 10 Feb 2025 10:47:05 -0500 Subject: [PATCH 012/129] Remove pandas backend from `cudf.pandas` - ibis integration tests (#17945) ibis has removed their pandas backend in version 10.0.0. In their release notes, > pandas: The pandas backend is removed. Note that pandas DataFrames are STILL VALID INPUTS AND OUTPUTS and will remain so for the foreseeable future. Please use one of the other local backends like DuckDB, Polars, or DataFusion to perform operations directly on pandas DataFrames. This PR removes the pandas backend from the integration tests. And asserts that the inputs and outputs to ibis APIs are proxy objects. --- .../third_party_integration_tests/tests/test_ibis.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py index 2a8cf7c6ac2..fe512f36866 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py @@ -1,15 +1,19 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. import ibis import numpy as np import pandas as pd import pytest -ibis.set_backend("pandas") +from cudf.pandas import is_proxy_object + ibis.options.interactive = False def ibis_assert_equal(expect, got, rtol: float = 1e-7, atol: float = 0.0): + assert is_proxy_object(got), ( + "The result from cudf.pandas must be a proxy object" + ) pd._testing.assert_almost_equal(expect, got, rtol=rtol, atol=atol) From 300d8165dc5d27a197aa655e34781713465c41ec Mon Sep 17 00:00:00 2001 From: jakirkham Date: Mon, 10 Feb 2025 08:15:48 -0800 Subject: [PATCH 013/129] Use Conda XGBoost (#17959) Switch to Conda XGBoost in Conda-based test of third-party dependencies. Authors: - https://github.com/jakirkham - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Murray (https://github.com/Matt711) - GALI PREM SAGAR (https://github.com/galipremsagar) - Jake Awe (https://github.com/AyodeAwe) URL: https://github.com/rapidsai/cudf/pull/17959 --- .../third_party_integration_tests/dependencies.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml index 356e7ac4494..53a8001c750 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml @@ -242,8 +242,7 @@ dependencies: - scipy - scikit-learn - pip - - pip: - - xgboost>=2.0.1 + - xgboost>=2.0.1 test_cuml: common: - output_types: conda From 1643e0a2793c20411e96e14fbec923db1e4567d0 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Mon, 10 Feb 2025 11:24:31 -0800 Subject: [PATCH 014/129] Raise NotImplementedError for groupby.agg if duplicate columns would be created (#17956) xref https://github.com/rapidsai/cudf/issues/17649 For `cudf.pandas`, we will dispatch to pandas instead of silently dropping the duplicate column Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17956 --- python/cudf/cudf/core/groupby/groupby.py | 15 ++++++++++++++- python/cudf/cudf/tests/test_groupby.py | 17 +++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/python/cudf/cudf/core/groupby/groupby.py b/python/cudf/cudf/core/groupby/groupby.py index bc55f4788a3..7b423f9d8a5 100644 --- a/python/cudf/cudf/core/groupby/groupby.py +++ b/python/cudf/cudf/core/groupby/groupby.py @@ -1643,12 +1643,25 @@ def _normalize_aggs( the keys. The aggs are applied to the corresponding column in the tuple. Each agg can be string or lambda functions. """ - aggs_per_column: Iterable[AggType | Iterable[AggType]] # TODO: Remove isinstance condition when the legacy dask_cudf API is removed. # See https://github.com/rapidsai/cudf/pull/16528#discussion_r1715482302 for information. if aggs or isinstance(aggs, dict): if isinstance(aggs, dict): + if any( + is_list_like(values) and len(set(values)) != len(values) # type: ignore[arg-type] + for values in aggs.values() + ): + if cudf.get_option("mode.pandas_compatible"): + raise NotImplementedError( + "Duplicate aggregations per column are currently not supported." + ) + else: + warnings.warn( + "Duplicate aggregations per column found. " + "The resulting duplicate columns will be dropped.", + UserWarning, + ) column_names, aggs_per_column = aggs.keys(), aggs.values() columns = tuple(self.obj._data[col] for col in column_names) else: diff --git a/python/cudf/cudf/tests/test_groupby.py b/python/cudf/cudf/tests/test_groupby.py index 50135db1344..2eac853a2e6 100644 --- a/python/cudf/cudf/tests/test_groupby.py +++ b/python/cudf/cudf/tests/test_groupby.py @@ -4115,3 +4115,20 @@ def test_scan_int_null_pandas_compatible(op): with cudf.option_context("mode.pandas_compatible", True): result = getattr(df_cudf.groupby("b")["a"], op)() assert_eq(result, expected) + + +def test_agg_duplicate_aggs_pandas_compat_raises(): + agg = {"b": ["mean", "mean"]} + dfgb = cudf.DataFrame({"a": [1, 1, 2], "b": [4, 5, 6]}).groupby(["a"]) + with cudf.option_context("mode.pandas_compatible", True): + with pytest.raises(NotImplementedError): + dfgb.agg(agg) + + with pytest.warns(UserWarning): + result = dfgb.agg(agg) + expected = cudf.DataFrame( + [4.5, 6.0], + index=cudf.Index([1, 2], name="a"), + columns=pd.MultiIndex.from_tuples([("b", "mean")]), + ) + assert_eq(result, expected) From d1a55582a13d1e1085a76e589785060839e811dd Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Mon, 10 Feb 2025 16:49:10 -0500 Subject: [PATCH 015/129] Pin `ibis` version in the cudf.pandas integration tests <10.0.0 (#17975) Follows up #17972. This PR is intended to get 25.02 nightly CI passing, which has been failing for few days. --- .../third_party_integration_tests/dependencies.yaml | 2 +- .../third_party_integration_tests/tests/test_ibis.py | 5 +---- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml index 03068d2268a..0b543a6d7d8 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml @@ -262,7 +262,7 @@ dependencies: packages: - pip - pip: - - ibis-framework[pandas] + - ibis-framework[pandas]<10.0.0 test_hvplot: common: - output_types: conda diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py index fe512f36866..70f20b2810e 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py @@ -5,15 +5,12 @@ import pandas as pd import pytest -from cudf.pandas import is_proxy_object +ibis.set_backend("pandas") ibis.options.interactive = False def ibis_assert_equal(expect, got, rtol: float = 1e-7, atol: float = 0.0): - assert is_proxy_object(got), ( - "The result from cudf.pandas must be a proxy object" - ) pd._testing.assert_almost_equal(expect, got, rtol=rtol, atol=atol) From 2d0c85c17152791a3e8a1020607546cb06e10c53 Mon Sep 17 00:00:00 2001 From: Yunsong Wang Date: Mon, 10 Feb 2025 14:29:48 -0800 Subject: [PATCH 016/129] Get rid of the deprecated `thrust::identity` (#17942) This PR removes the usage of the deprecated `thrust::identity` and replaces it with `cuda::std::identity` where applicable. In cases where `thrust::identity` was used as a type conversion callable and `cuda::std::identity` is not suitable, it uses a new utility called `cast_fn` in the `detail` namespace. Authors: - Yunsong Wang (https://github.com/PointKernel) Approvers: - Michael Schellenberger Costa (https://github.com/miscco) - David Wendt (https://github.com/davidwendt) - MithunR (https://github.com/mythrocks) URL: https://github.com/rapidsai/cudf/pull/17942 --- cpp/benchmarks/common/generate_input.cu | 11 +++++------ cpp/benchmarks/join/join_common.hpp | 6 +++--- .../cudf/detail/utilities/cast_functor.cuh | 19 ++++++++++++++++++- .../reduction/detail/reduction_operators.cuh | 15 +++++++-------- cpp/include/cudf_test/column_wrapper.hpp | 11 +++++++---- cpp/src/groupby/sort/group_merge_m2.cu | 6 +++--- cpp/src/groupby/sort/group_scan_util.cuh | 8 ++++---- .../sort/group_single_pass_reduction_util.cuh | 8 ++++---- cpp/src/groupby/sort/sort_helper.cu | 4 ++-- cpp/src/io/json/json_normalization.cu | 5 +++-- cpp/src/join/hash_join.cu | 6 +++--- cpp/src/join/join_utils.cu | 6 +++--- .../combine/concatenate_list_elements.cu | 5 ++--- cpp/src/lists/interleave_columns.cu | 7 +++---- cpp/src/reductions/scan/scan_inclusive.cu | 6 +++--- cpp/src/sort/rank.cu | 7 +++---- cpp/src/stream_compaction/unique.cu | 5 +++-- cpp/src/strings/search/contains_multiple.cu | 4 ++-- cpp/tests/iterator/iterator_tests.cuh | 6 +++--- .../iterator/value_iterator_test_transform.cu | 17 +++++++++++++---- cpp/tests/rolling/collect_ops_test.cpp | 5 ++--- .../apply_boolean_mask_tests.cpp | 8 ++++---- cpp/tests/transform/row_bit_count_test.cu | 6 +++--- cpp/tests/utilities/column_utilities.cu | 2 +- 24 files changed, 104 insertions(+), 79 deletions(-) diff --git a/cpp/benchmarks/common/generate_input.cu b/cpp/benchmarks/common/generate_input.cu index 8bce718c7d8..8d6aacd2ef1 100644 --- a/cpp/benchmarks/common/generate_input.cu +++ b/cpp/benchmarks/common/generate_input.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -46,7 +46,6 @@ #include #include #include -#include #include #include #include @@ -511,7 +510,7 @@ std::unique_ptr create_random_column(data_profile const& profile, auto [result_bitmask, null_count] = cudf::detail::valid_if(null_mask.begin(), null_mask.end(), - thrust::identity{}, + cuda::std::identity{}, cudf::get_default_stream(), cudf::get_current_device_resource_ref()); @@ -600,7 +599,7 @@ std::unique_ptr create_random_utf8_string_column(data_profile cons auto [result_bitmask, null_count] = profile.get_null_probability().has_value() ? cudf::detail::valid_if( - null_mask.begin(), null_mask.end() - 1, thrust::identity{}, stream, mr) + null_mask.begin(), null_mask.end() - 1, cuda::std::identity{}, stream, mr) : std::pair{rmm::device_buffer{}, 0}; return cudf::make_strings_column( @@ -693,7 +692,7 @@ std::unique_ptr create_random_column(data_profi auto valids = valid_dist(engine, num_rows); return cudf::detail::valid_if(valids.begin(), valids.end(), - thrust::identity{}, + cuda::std::identity{}, cudf::get_default_stream(), cudf::get_current_device_resource_ref()); } @@ -787,7 +786,7 @@ std::unique_ptr create_random_column(data_profile auto [null_mask, null_count] = cudf::detail::valid_if(valids.begin(), valids.end(), - thrust::identity{}, + cuda::std::identity{}, cudf::get_default_stream(), cudf::get_current_device_resource_ref()); list_column = cudf::make_lists_column( diff --git a/cpp/benchmarks/join/join_common.hpp b/cpp/benchmarks/join/join_common.hpp index 1f1ca414ad1..adb7cd26754 100644 --- a/cpp/benchmarks/join/join_common.hpp +++ b/cpp/benchmarks/join/join_common.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -31,7 +31,7 @@ #include #include -#include +#include #include #include #include @@ -85,7 +85,7 @@ void BM_join(state_type& state, Join JoinFunc) thrust::make_transform_iterator(thrust::make_counting_iterator(0), null75_generator{}); return cudf::detail::valid_if(validity, validity + size, - thrust::identity{}, + cuda::std::identity{}, cudf::get_default_stream(), cudf::get_current_device_resource_ref()); }; diff --git a/cpp/include/cudf/detail/utilities/cast_functor.cuh b/cpp/include/cudf/detail/utilities/cast_functor.cuh index d5209942c8a..079f9269071 100644 --- a/cpp/include/cudf/detail/utilities/cast_functor.cuh +++ b/cpp/include/cudf/detail/utilities/cast_functor.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2023, NVIDIA CORPORATION. + * Copyright (c) 2023-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -31,6 +31,23 @@ namespace cudf { namespace detail { +/** + * @brief Functor that casts a primitive input value to a specified type + */ +template +struct cast_fn { + template + CUDF_HOST_DEVICE constexpr T operator()(U&& val) const + { + return static_cast(cuda::std::forward(val)); + } + + CUDF_HOST_DEVICE constexpr T&& operator()(T&& val) const noexcept + { + return cuda::std::forward(val); + } +}; + /** * @brief Functor that casts another functor's result to a specified type. * diff --git a/cpp/include/cudf/reduction/detail/reduction_operators.cuh b/cpp/include/cudf/reduction/detail/reduction_operators.cuh index 5694362af8f..2425e5075aa 100644 --- a/cpp/include/cudf/reduction/detail/reduction_operators.cuh +++ b/cpp/include/cudf/reduction/detail/reduction_operators.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -17,12 +17,11 @@ #pragma once #include +#include #include #include #include //for CUDF_HOST_DEVICE -#include - #include namespace cudf { @@ -156,7 +155,7 @@ struct sum : public simple_op { using op = cudf::DeviceSum; template - using transformer = thrust::identity; + using transformer = cudf::detail::cast_fn; }; // operator for `product` @@ -164,7 +163,7 @@ struct product : public simple_op { using op = cudf::DeviceProduct; template - using transformer = thrust::identity; + using transformer = cudf::detail::cast_fn; }; // operator for `sum_of_squares` @@ -180,7 +179,7 @@ struct min : public simple_op { using op = cudf::DeviceMin; template - using transformer = thrust::identity; + using transformer = cudf::detail::cast_fn; }; // operator for `max` @@ -188,7 +187,7 @@ struct max : public simple_op { using op = cudf::DeviceMax; template - using transformer = thrust::identity; + using transformer = cudf::detail::cast_fn; }; /** @@ -246,7 +245,7 @@ struct mean : public compound_op { using op = cudf::DeviceSum; template - using transformer = thrust::identity; + using transformer = cudf::detail::cast_fn; template struct intermediate { diff --git a/cpp/include/cudf_test/column_wrapper.hpp b/cpp/include/cudf_test/column_wrapper.hpp index 6300bb87572..3a4b5bd0c96 100644 --- a/cpp/include/cudf_test/column_wrapper.hpp +++ b/cpp/include/cudf_test/column_wrapper.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -39,8 +39,8 @@ #include +#include #include -#include #include #include #include @@ -1645,8 +1645,11 @@ class lists_column_wrapper : public detail::column_wrapper { // concatenate them together, skipping children that are null. std::vector children; - thrust::copy_if( - std::cbegin(cols), std::cend(cols), valids, std::back_inserter(children), thrust::identity{}); + thrust::copy_if(std::cbegin(cols), + std::cend(cols), + valids, + std::back_inserter(children), + cuda::std::identity{}); auto data = children.empty() ? cudf::empty_like(expected_hierarchy) : cudf::concatenate(children, diff --git a/cpp/src/groupby/sort/group_merge_m2.cu b/cpp/src/groupby/sort/group_merge_m2.cu index 746c3fe3962..3984c0b6181 100644 --- a/cpp/src/groupby/sort/group_merge_m2.cu +++ b/cpp/src/groupby/sort/group_merge_m2.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -27,7 +27,7 @@ #include #include -#include +#include #include #include #include @@ -180,7 +180,7 @@ std::unique_ptr group_merge_m2(column_view const& values, // Generate bitmask for the output. // Only mean and M2 values can be nullable. Count column must be non-nullable. auto [null_mask, null_count] = - cudf::detail::valid_if(validities.begin(), validities.end(), thrust::identity{}, stream, mr); + cudf::detail::valid_if(validities.begin(), validities.end(), cuda::std::identity{}, stream, mr); if (null_count > 0) { result_means->set_null_mask(null_mask, null_count, stream); // copy null_mask result_M2s->set_null_mask(std::move(null_mask), null_count); // take over null_mask diff --git a/cpp/src/groupby/sort/group_scan_util.cuh b/cpp/src/groupby/sort/group_scan_util.cuh index 5082ad01327..a90445fabe1 100644 --- a/cpp/src/groupby/sort/group_scan_util.cuh +++ b/cpp/src/groupby/sort/group_scan_util.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -36,7 +37,6 @@ #include #include -#include #include #include #include @@ -129,12 +129,12 @@ struct group_scan_functor() if (values.has_nulls()) { auto input = thrust::make_transform_iterator( make_null_replacement_iterator(*values_view, OpType::template identity()), - thrust::identity{}); + cudf::detail::cast_fn{}); do_scan(input, result_view->begin(), OpType{}); result->set_null_mask(cudf::detail::copy_bitmask(values, stream, mr), values.null_count()); } else { auto input = thrust::make_transform_iterator(values_view->begin(), - thrust::identity{}); + cudf::detail::cast_fn{}); do_scan(input, result_view->begin(), OpType{}); } return result; diff --git a/cpp/src/groupby/sort/group_single_pass_reduction_util.cuh b/cpp/src/groupby/sort/group_single_pass_reduction_util.cuh index f9adfc6060e..662c380eff5 100644 --- a/cpp/src/groupby/sort/group_single_pass_reduction_util.cuh +++ b/cpp/src/groupby/sort/group_single_pass_reduction_util.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -33,7 +33,7 @@ #include #include -#include +#include #include #include #include @@ -204,7 +204,7 @@ struct group_reduction_functor< thrust::logical_or{}); auto [null_mask, null_count] = - cudf::detail::valid_if(validity.begin(), validity.end(), thrust::identity{}, stream, mr); + cudf::detail::valid_if(validity.begin(), validity.end(), cuda::std::identity{}, stream, mr); result->set_null_mask(std::move(null_mask), null_count); } return result; @@ -257,7 +257,7 @@ struct group_reduction_functor< thrust::logical_or{}); auto [null_mask, null_count] = - cudf::detail::valid_if(validity.begin(), validity.end(), thrust::identity{}, stream, mr); + cudf::detail::valid_if(validity.begin(), validity.end(), cuda::std::identity{}, stream, mr); result->set_null_mask(std::move(null_mask), null_count); } diff --git a/cpp/src/groupby/sort/sort_helper.cu b/cpp/src/groupby/sort/sort_helper.cu index 35e3e05a364..ae04113223d 100644 --- a/cpp/src/groupby/sort/sort_helper.cu +++ b/cpp/src/groupby/sort/sort_helper.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -165,7 +165,7 @@ sort_groupby_helper::index_vector const& sort_groupby_helper::group_offsets( itr + size, result.begin(), group_offsets->begin(), - thrust::identity{}); + cuda::std::identity{}); } else { auto const d_key_equal = comparator.equal_to( cudf::nullate::DYNAMIC{cudf::has_nested_nulls(_keys)}, null_equality::EQUAL); diff --git a/cpp/src/io/json/json_normalization.cu b/cpp/src/io/json/json_normalization.cu index 1b61be20202..b27ec24deae 100644 --- a/cpp/src/io/json/json_normalization.cu +++ b/cpp/src/io/json/json_normalization.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -30,6 +30,7 @@ #include #include +#include #include #include #include @@ -442,7 +443,7 @@ std:: inbuf.begin(), inbuf.end(), stencil.begin(), - thrust::identity()); + cuda::std::identity{}); inbuf.resize(inbuf_size - num_deletions, stream); thrust::exclusive_scan(rmm::exec_policy_nosync(stream), diff --git a/cpp/src/join/hash_join.cu b/cpp/src/join/hash_join.cu index 05b85fed1a8..86c0bbc0385 100644 --- a/cpp/src/join/hash_join.cu +++ b/cpp/src/join/hash_join.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -31,8 +31,8 @@ #include #include +#include #include -#include #include #include #include @@ -357,7 +357,7 @@ std::size_t get_full_join_size( left_join_complement_size = thrust::count_if(rmm::exec_policy(stream), invalid_index_map->begin(), invalid_index_map->end(), - thrust::identity()); + cuda::std::identity()); } return join_size + left_join_complement_size; } diff --git a/cpp/src/join/join_utils.cu b/cpp/src/join/join_utils.cu index 16302657ac2..76fa4831c19 100644 --- a/cpp/src/join/join_utils.cu +++ b/cpp/src/join/join_utils.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -20,8 +20,8 @@ #include +#include #include -#include #include #include #include @@ -141,7 +141,7 @@ get_left_join_indices_complement(std::unique_ptr> thrust::make_counting_iterator(end_counter), invalid_index_map->begin(), right_indices_complement->begin(), - thrust::identity{}) - + cuda::std::identity{}) - right_indices_complement->begin(); right_indices_complement->resize(indices_count, stream); } diff --git a/cpp/src/lists/combine/concatenate_list_elements.cu b/cpp/src/lists/combine/concatenate_list_elements.cu index 7ae5db3e84b..667872a3491 100644 --- a/cpp/src/lists/combine/concatenate_list_elements.cu +++ b/cpp/src/lists/combine/concatenate_list_elements.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -35,7 +35,6 @@ #include #include #include -#include #include #include #include @@ -227,7 +226,7 @@ std::unique_ptr concatenate_lists_nullifying_rows(column_view const& inp auto list_entries = gather_list_entries(input, offsets_view, num_rows, num_output_entries, stream, mr); auto [null_mask, null_count] = cudf::detail::valid_if( - list_validities.begin(), list_validities.end(), thrust::identity{}, stream, mr); + list_validities.begin(), list_validities.end(), cuda::std::identity{}, stream, mr); return make_lists_column(num_rows, std::move(list_offsets), diff --git a/cpp/src/lists/interleave_columns.cu b/cpp/src/lists/interleave_columns.cu index 3d6fdda957b..85157e4417f 100644 --- a/cpp/src/lists/interleave_columns.cu +++ b/cpp/src/lists/interleave_columns.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -35,7 +35,6 @@ #include #include #include -#include #include #include #include @@ -282,7 +281,7 @@ struct interleave_list_entries_impl( if (data_has_null_mask) { auto [null_mask, null_count] = cudf::detail::valid_if( - validities.begin(), validities.end(), thrust::identity{}, stream, mr); + validities.begin(), validities.end(), cuda::std::identity{}, stream, mr); if (null_count > 0) { output->set_null_mask(std::move(null_mask), null_count); } } @@ -381,7 +380,7 @@ std::unique_ptr interleave_columns(table_view const& input, } auto [null_mask, null_count] = cudf::detail::valid_if( - list_validities.begin(), list_validities.end(), thrust::identity{}, stream, mr); + list_validities.begin(), list_validities.end(), cuda::std::identity{}, stream, mr); return make_lists_column(num_output_lists, std::move(list_offsets), std::move(list_entries), diff --git a/cpp/src/reductions/scan/scan_inclusive.cu b/cpp/src/reductions/scan/scan_inclusive.cu index a876d54d45f..7509045f950 100644 --- a/cpp/src/reductions/scan/scan_inclusive.cu +++ b/cpp/src/reductions/scan/scan_inclusive.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -33,8 +33,8 @@ #include #include +#include #include -#include #include #include @@ -57,7 +57,7 @@ std::pair mask_scan(column_view const& input_view auto first_null_position = [&] { size_type const first_null = thrust::find_if_not( - rmm::exec_policy(stream), valid_itr, valid_itr + input_view.size(), thrust::identity{}) - + rmm::exec_policy(stream), valid_itr, valid_itr + input_view.size(), cuda::std::identity{}) - valid_itr; size_type const exclusive_offset = (inclusive == scan_type::EXCLUSIVE) ? 1 : 0; return std::min(input_view.size(), first_null + exclusive_offset); diff --git a/cpp/src/sort/rank.cu b/cpp/src/sort/rank.cu index cbde87198bd..e7dca2277ec 100644 --- a/cpp/src/sort/rank.cu +++ b/cpp/src/sort/rank.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -34,7 +34,6 @@ #include #include -#include #include #include #include @@ -204,7 +203,7 @@ void rank_min(cudf::device_span group_keys, sorted_order_view, rank_mutable_view.begin(), thrust::minimum{}, - thrust::identity{}, + cuda::std::identity{}, stream); } @@ -222,7 +221,7 @@ void rank_max(cudf::device_span group_keys, sorted_order_view, rank_mutable_view.begin(), thrust::maximum{}, - thrust::identity{}, + cuda::std::identity{}, stream); } diff --git a/cpp/src/stream_compaction/unique.cu b/cpp/src/stream_compaction/unique.cu index eaabc6f1272..4c6f3e9e6f0 100644 --- a/cpp/src/stream_compaction/unique.cu +++ b/cpp/src/stream_compaction/unique.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -37,6 +37,7 @@ #include #include +#include #include #include #include @@ -87,7 +88,7 @@ std::unique_ptr unique(table_view const& input, itr + num_rows, d_results.begin(), mutable_view->begin(), - thrust::identity{}); + cuda::std::identity{}); return static_cast(thrust::distance(mutable_view->begin(), result_end)); } else { // Using thrust::unique_copy with the comparator directly will compile more slowly but diff --git a/cpp/src/strings/search/contains_multiple.cu b/cpp/src/strings/search/contains_multiple.cu index 1183e3e4038..22b417fbc41 100644 --- a/cpp/src/strings/search/contains_multiple.cu +++ b/cpp/src/strings/search/contains_multiple.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -180,7 +180,7 @@ CUDF_KERNEL void multi_contains_kernel(column_device_view const d_strings, for (auto target_idx = lane_idx; target_idx < num_targets; target_idx += tile_size) { auto const begin = bools + (target_idx * tile_size); d_results[target_idx][str_idx] = - thrust::any_of(thrust::seq, begin, begin + tile_size, thrust::identity{}); + thrust::any_of(thrust::seq, begin, begin + tile_size, cuda::std::identity{}); // cooperative_group any() implementation was almost 3x slower than this parallel reduce } } diff --git a/cpp/tests/iterator/iterator_tests.cuh b/cpp/tests/iterator/iterator_tests.cuh index 5c9f6114eb5..119d8e7b138 100644 --- a/cpp/tests/iterator/iterator_tests.cuh +++ b/cpp/tests/iterator/iterator_tests.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -28,10 +28,10 @@ #include #include +#include #include #include #include -#include #include #include #include @@ -102,7 +102,7 @@ struct IteratorTest : public cudf::test::BaseFixture { auto result = thrust::all_of(rmm::exec_policy(cudf::get_default_stream()), dev_results.begin(), dev_results.end(), - thrust::identity{}); + cuda::std::identity{}); EXPECT_TRUE(result) << "thrust test"; } diff --git a/cpp/tests/iterator/value_iterator_test_transform.cu b/cpp/tests/iterator/value_iterator_test_transform.cu index 417233e759b..e6fb8ef0bb0 100644 --- a/cpp/tests/iterator/value_iterator_test_transform.cu +++ b/cpp/tests/iterator/value_iterator_test_transform.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -16,12 +16,21 @@ #include -#include +#include #include #include struct TransformedIteratorTest : public IteratorTest {}; +template +struct cast_fn { + template + __device__ T operator()(U const& val) const + { + return static_cast(val); + } +}; + // Tests up cast reduction with null iterator. // The up cast iterator will be created by transform_iterator and // cudf::detail::make_null_replacement_iterator(col, T{0}) @@ -57,7 +66,7 @@ TEST_F(TransformedIteratorTest, null_iterator_upcast) // GPU test auto it_dev = cudf::detail::make_null_replacement_iterator(*d_col, T{0}); - auto it_dev_upcast = thrust::make_transform_iterator(it_dev, thrust::identity()); + auto it_dev_upcast = thrust::make_transform_iterator(it_dev, cast_fn{}); this->iterator_test_thrust(replaced_array, it_dev_upcast, d_col->size()); this->iterator_test_cub(expected_value, it_dev, d_col->size()); } @@ -100,7 +109,7 @@ TEST_F(TransformedIteratorTest, null_iterator_square) // GPU test auto it_dev = cudf::detail::make_null_replacement_iterator(*d_col, T{0}); - auto it_dev_upcast = thrust::make_transform_iterator(it_dev, thrust::identity()); + auto it_dev_upcast = thrust::make_transform_iterator(it_dev, cast_fn{}); auto it_dev_squared = thrust::make_transform_iterator(it_dev_upcast, transformer); this->iterator_test_thrust(replaced_array, it_dev_squared, d_col->size()); this->iterator_test_cub(expected_value, it_dev_squared, d_col->size()); diff --git a/cpp/tests/rolling/collect_ops_test.cpp b/cpp/tests/rolling/collect_ops_test.cpp index cb4e6945c07..b4f31ae57dd 100644 --- a/cpp/tests/rolling/collect_ops_test.cpp +++ b/cpp/tests/rolling/collect_ops_test.cpp @@ -27,7 +27,7 @@ #include #include -#include +#include #include @@ -455,8 +455,7 @@ TEST_F(CollectListTest, RollingWindowHonoursMinPeriodsWithDecimal) { // Test that when the number of observations is fewer than min_periods, // the result is null. - auto const input_iter = - cudf::detail::make_counting_transform_iterator(0, thrust::identity{}); + auto const input_iter = thrust::counting_iterator{0}; auto const input_column = cudf::test::fixed_point_column_wrapper{ input_iter, input_iter + 6, numeric::scale_type{0}}; diff --git a/cpp/tests/stream_compaction/apply_boolean_mask_tests.cpp b/cpp/tests/stream_compaction/apply_boolean_mask_tests.cpp index 1204b019739..b1c51d05efb 100644 --- a/cpp/tests/stream_compaction/apply_boolean_mask_tests.cpp +++ b/cpp/tests/stream_compaction/apply_boolean_mask_tests.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -29,7 +29,7 @@ #include #include -#include +#include struct ApplyBooleanMask : public cudf::test::BaseFixture {}; @@ -208,13 +208,13 @@ TEST_F(ApplyBooleanMask, FixedPointLargeColumnTest) dec32_data.cend(), mask_data.cbegin(), std::back_inserter(expect_dec32_data), - thrust::identity{}); + cuda::std::identity{}); thrust::copy_if(thrust::seq, dec64_data.cbegin(), dec64_data.cend(), mask_data.cbegin(), std::back_inserter(expect_dec64_data), - thrust::identity{}); + cuda::std::identity{}); decimal32_wrapper expect_col32( expect_dec32_data.begin(), expect_dec32_data.end(), numeric::scale_type{-3}); diff --git a/cpp/tests/transform/row_bit_count_test.cu b/cpp/tests/transform/row_bit_count_test.cu index 7e203086fca..5b96633788c 100644 --- a/cpp/tests/transform/row_bit_count_test.cu +++ b/cpp/tests/transform/row_bit_count_test.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -27,8 +27,8 @@ #include +#include #include -#include #include #include #include @@ -348,7 +348,7 @@ TEST_F(RowBitCount, StructsWithLists_RowsExceedingASingleBlock) thrust::tabulate(rmm::exec_policy(cudf::get_default_stream()), ints_view.begin(), ints_view.end(), - thrust::identity{}); + cuda::std::identity{}); // List offsets = {0, 2, 4, 6, 8, ..., num_rows*2}; auto list_offsets = diff --git a/cpp/tests/utilities/column_utilities.cu b/cpp/tests/utilities/column_utilities.cu index 6888f26fd16..b97afacb14b 100644 --- a/cpp/tests/utilities/column_utilities.cu +++ b/cpp/tests/utilities/column_utilities.cu @@ -554,7 +554,7 @@ struct column_comparator_impl { input_iter + lhs_row_indices.size(), diff_map.begin(), differences.begin(), - thrust::identity{}); + cuda::std::identity{}); differences.resize(thrust::distance(differences.begin(), diff_iter), cudf::test::get_default_stream()); // shrink back down From 1a891e6cfd1daef5bb56990cd18b4e3c7640fb53 Mon Sep 17 00:00:00 2001 From: Vyas Ramasubramani Date: Mon, 10 Feb 2025 19:36:44 -0800 Subject: [PATCH 017/129] Use new rapids-logger library (#17899) Contributes to https://github.com/rapidsai/build-planning/issues/104. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Matthew Murray (https://github.com/Matt711) - Gil Forsyth (https://github.com/gforsyth) - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) URL: https://github.com/rapidsai/cudf/pull/17899 --- .github/workflows/pr.yaml | 1 + ci/build_wheel_cudf.sh | 1 + ci/build_wheel_libcudf.sh | 1 + ci/build_wheel_pylibcudf.sh | 1 + .../all_cuda-118_arch-x86_64.yaml | 2 +- .../all_cuda-128_arch-x86_64.yaml | 2 +- conda/recipes/libcudf/conda_build_config.yaml | 3 - conda/recipes/libcudf/meta.yaml | 3 +- cpp/CMakeLists.txt | 22 ++---- cpp/cmake/thirdparty/get_fmt.cmake | 22 ------ cpp/include/cudf/logger.hpp | 50 +++++++++++++ cpp/src/utilities/logger.cpp | 46 ++++++++++++ cpp/tests/CMakeLists.txt | 1 - cpp/tests/utilities_tests/logger_tests.cpp | 74 ------------------- dependencies.yaml | 14 +++- python/libcudf/libcudf/load.py | 9 ++- python/libcudf/pyproject.toml | 3 + 17 files changed, 134 insertions(+), 121 deletions(-) delete mode 100644 cpp/cmake/thirdparty/get_fmt.cmake create mode 100644 cpp/include/cudf/logger.hpp create mode 100644 cpp/src/utilities/logger.cpp delete mode 100644 cpp/tests/utilities_tests/logger_tests.cpp diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index e36b3e2ede4..07c00b3e13c 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -142,6 +142,7 @@ jobs: with: build_type: pull-request run_script: "ci/cpp_linters.sh" + node_type: "cpu16" conda-cpp-checks: needs: conda-cpp-build secrets: inherit diff --git a/ci/build_wheel_cudf.sh b/ci/build_wheel_cudf.sh index 0f373547660..9d2f97e4261 100755 --- a/ci/build_wheel_cudf.sh +++ b/ci/build_wheel_cudf.sh @@ -24,6 +24,7 @@ python -m auditwheel repair \ --exclude libcudf.so \ --exclude libnvcomp.so \ --exclude libkvikio.so \ + --exclude librapids_logger.so \ -w ${package_dir}/final_dist \ ${package_dir}/dist/* diff --git a/ci/build_wheel_libcudf.sh b/ci/build_wheel_libcudf.sh index ca2ecb24a75..06be943fd93 100755 --- a/ci/build_wheel_libcudf.sh +++ b/ci/build_wheel_libcudf.sh @@ -34,6 +34,7 @@ mkdir -p ${package_dir}/final_dist python -m auditwheel repair \ --exclude libnvcomp.so.4 \ --exclude libkvikio.so \ + --exclude librapids_logger.so \ -w ${package_dir}/final_dist \ ${package_dir}/dist/* diff --git a/ci/build_wheel_pylibcudf.sh b/ci/build_wheel_pylibcudf.sh index 9091f59d57b..e32c32d4262 100755 --- a/ci/build_wheel_pylibcudf.sh +++ b/ci/build_wheel_pylibcudf.sh @@ -22,6 +22,7 @@ python -m auditwheel repair \ --exclude libcudf.so \ --exclude libnvcomp.so \ --exclude libkvikio.so \ + --exclude librapids_logger.so \ -w ${package_dir}/final_dist \ ${package_dir}/dist/* diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index 190533abc51..09eb9949f1d 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -31,7 +31,6 @@ dependencies: - doxygen=1.9.1 - fastavro>=0.22.9 - flatbuffers==24.3.25 -- fmt>=11.0.2,<12 - fsspec>=0.6.0 - gcc_linux-64=11.* - hypothesis @@ -83,6 +82,7 @@ dependencies: - python>=3.10,<3.13 - rapids-build-backend>=0.3.0,<0.4.0.dev0 - rapids-dask-dependency==25.4.*,>=0.0.0a0 +- rapids-logger==0.1.*,>=0.0.0a0 - rich - rmm==25.4.*,>=0.0.0a0 - s3fs>=2022.3.0 diff --git a/conda/environments/all_cuda-128_arch-x86_64.yaml b/conda/environments/all_cuda-128_arch-x86_64.yaml index e719fd51573..56cef28ac61 100644 --- a/conda/environments/all_cuda-128_arch-x86_64.yaml +++ b/conda/environments/all_cuda-128_arch-x86_64.yaml @@ -32,7 +32,6 @@ dependencies: - doxygen=1.9.1 - fastavro>=0.22.9 - flatbuffers==24.3.25 -- fmt>=11.0.2,<12 - fsspec>=0.6.0 - gcc_linux-64=13.* - hypothesis @@ -82,6 +81,7 @@ dependencies: - pytorch>=2.4.0 - rapids-build-backend>=0.3.0,<0.4.0.dev0 - rapids-dask-dependency==25.4.*,>=0.0.0a0 +- rapids-logger==0.1.*,>=0.0.0a0 - rich - rmm==25.4.*,>=0.0.0a0 - s3fs>=2022.3.0 diff --git a/conda/recipes/libcudf/conda_build_config.yaml b/conda/recipes/libcudf/conda_build_config.yaml index 181064465ef..1da96ebc072 100644 --- a/conda/recipes/libcudf/conda_build_config.yaml +++ b/conda/recipes/libcudf/conda_build_config.yaml @@ -25,9 +25,6 @@ dlpack_version: librdkafka_version: - ">=2.5.0,<2.6.0a0" -fmt_version: - - ">=11.0.2,<12" - flatbuffers_version: - "=24.3.25" diff --git a/conda/recipes/libcudf/meta.yaml b/conda/recipes/libcudf/meta.yaml index 55a1d9cbe72..f7bd7280f0f 100644 --- a/conda/recipes/libcudf/meta.yaml +++ b/conda/recipes/libcudf/meta.yaml @@ -66,8 +66,8 @@ requirements: - nvcomp {{ nvcomp_version }} - dlpack {{ dlpack_version }} - librdkafka {{ librdkafka_version }} - - fmt {{ fmt_version }} - flatbuffers {{ flatbuffers_version }} + - rapids-logger =0.1 - zlib {{ zlib_version }} outputs: @@ -99,6 +99,7 @@ outputs: - librmm ={{ minor_version }} - libkvikio ={{ minor_version }} - dlpack {{ dlpack_version }} + - rapids-logger =0.1 test: commands: - test -f $PREFIX/lib/libcudf.so diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index 354560998c5..56b97f6ce00 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -275,8 +275,8 @@ endif() rapids_cpm_init() include(${rapids-cmake-dir}/cpm/rapids_logger.cmake) -rapids_cpm_rapids_logger() -rapids_make_logger(cudf EXPORT_SET cudf-exports LOGGER_DEFAULT_LEVEL WARN) +rapids_cpm_rapids_logger(BUILD_EXPORT_SET cudf-exports INSTALL_EXPORT_SET cudf-exports) +create_logger_macros(CUDF "cudf::default_logger()" include/cudf) # find jitify include(cmake/thirdparty/get_jitify.cmake) @@ -302,8 +302,6 @@ endif() include(cmake/Modules/JitifyPreprocessKernels.cmake) # find KvikIO include(cmake/thirdparty/get_kvikio.cmake) -# find fmt -include(cmake/thirdparty/get_fmt.cmake) # find nanoarrow include(cmake/thirdparty/get_nanoarrow.cmake) # find thread_pool @@ -779,6 +777,7 @@ add_library( src/utilities/default_stream.cpp src/utilities/host_memory.cpp src/utilities/linked_column.cpp + src/utilities/logger.cpp src/utilities/prefetch.cpp src/utilities/stacktrace.cpp src/utilities/stream_pool.cpp @@ -943,16 +942,9 @@ add_dependencies(cudf jitify_preprocess_run) # Specify the target module library dependencies target_link_libraries( cudf - PUBLIC CCCL::CCCL rmm::rmm rmm::rmm_logger $ cudf_logger - PRIVATE $ - cuco::cuco - ZLIB::ZLIB - nvcomp::nvcomp - kvikio::kvikio - $ - nanoarrow - rmm::rmm_logger_impl - cudf_logger_impl + PUBLIC CCCL::CCCL rapids_logger::rapids_logger rmm::rmm $ + PRIVATE $ cuco::cuco ZLIB::ZLIB nvcomp::nvcomp + kvikio::kvikio $ nanoarrow ) # Add Conda library, and include paths if specified @@ -1108,7 +1100,7 @@ if(CUDF_BUILD_STREAMS_TEST_UTIL) ${_tgt} PRIVATE "$:${CUDF_CXX_FLAGS}>>" ) target_include_directories(${_tgt} PRIVATE "$") - target_link_libraries(${_tgt} PUBLIC CUDA::cudart rmm::rmm rmm::rmm_logger rmm::rmm_logger_impl) + target_link_libraries(${_tgt} PUBLIC CUDA::cudart rmm::rmm) if(CUDF_BUILD_STACKTRACE_DEBUG) target_link_libraries(${_tgt} PRIVATE cudf_backtrace) endif() diff --git a/cpp/cmake/thirdparty/get_fmt.cmake b/cpp/cmake/thirdparty/get_fmt.cmake deleted file mode 100644 index 083dd1d0631..00000000000 --- a/cpp/cmake/thirdparty/get_fmt.cmake +++ /dev/null @@ -1,22 +0,0 @@ -# ============================================================================= -# Copyright (c) 2023, NVIDIA CORPORATION. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except -# in compliance with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software distributed under the License -# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express -# or implied. See the License for the specific language governing permissions and limitations under -# the License. -# ============================================================================= - -# Use CPM to find or clone fmt -function(find_and_configure_fmt) - - include(${rapids-cmake-dir}/cpm/fmt.cmake) - rapids_cpm_fmt(INSTALL_EXPORT_SET cudf-exports BUILD_EXPORT_SET cudf-exports) -endfunction() - -find_and_configure_fmt() diff --git a/cpp/include/cudf/logger.hpp b/cpp/include/cudf/logger.hpp new file mode 100644 index 00000000000..fd9a0509496 --- /dev/null +++ b/cpp/include/cudf/logger.hpp @@ -0,0 +1,50 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include + +#include + +namespace CUDF_EXPORT cudf { + +/** + * @brief Returns the default sink for the global logger. + * + * If the environment variable `CUDF_DEBUG_LOG_FILE` is defined, the default sink is a sink to that + * file. Otherwise, the default is to dump to stderr. + * + * @return sink_ptr The sink to use + */ +rapids_logger::sink_ptr default_logger_sink(); + +/** + * @brief Returns the default log pattern for the global logger. + * + * @return std::string The default log pattern. + */ +std::string default_logger_pattern(); + +/** + * @brief Get the default logger. + * + * @return logger& The default logger + */ +rapids_logger::logger& default_logger(); + +} // namespace CUDF_EXPORT cudf diff --git a/cpp/src/utilities/logger.cpp b/cpp/src/utilities/logger.cpp new file mode 100644 index 00000000000..7340f2ec1a7 --- /dev/null +++ b/cpp/src/utilities/logger.cpp @@ -0,0 +1,46 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include + +#include + +namespace CUDF_EXPORT cudf { + +rapids_logger::sink_ptr default_logger_sink() +{ + auto* filename = std::getenv("CUDF_DEBUG_LOG_FILE"); + if (filename != nullptr) { + return std::make_shared(filename, true); + } + return std::make_shared(); +} + +std::string default_logger_pattern() { return "[%6t][%H:%M:%S:%f][%-6l] %v"; } + +rapids_logger::logger& default_logger() +{ + static rapids_logger::logger logger_ = [] { + rapids_logger::logger logger_{"CUDF", {default_logger_sink()}}; + logger_.set_pattern(default_logger_pattern()); + logger_.set_level(rapids_logger::level_enum::warn); + return logger_; + }(); + return logger_; +} + +} // namespace CUDF_EXPORT cudf diff --git a/cpp/tests/CMakeLists.txt b/cpp/tests/CMakeLists.txt index e031597ed18..117cd620679 100644 --- a/cpp/tests/CMakeLists.txt +++ b/cpp/tests/CMakeLists.txt @@ -405,7 +405,6 @@ ConfigureTest( utilities_tests/default_stream_tests.cpp utilities_tests/io_utilities_tests.cpp utilities_tests/lists_column_wrapper_tests.cpp - utilities_tests/logger_tests.cpp utilities_tests/pinned_memory_tests.cpp utilities_tests/type_check_tests.cpp utilities_tests/type_list_tests.cpp diff --git a/cpp/tests/utilities_tests/logger_tests.cpp b/cpp/tests/utilities_tests/logger_tests.cpp deleted file mode 100644 index b5d20325b75..00000000000 --- a/cpp/tests/utilities_tests/logger_tests.cpp +++ /dev/null @@ -1,74 +0,0 @@ -/* - * Copyright (c) 2023-2025, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include - -#include - -#include - -class LoggerTest : public cudf::test::BaseFixture { - std::ostringstream oss; - cudf::level_enum prev_level; - - public: - LoggerTest() : prev_level{cudf::default_logger().level()} - { - cudf::default_logger().sinks().push_back(std::make_shared(oss)); - cudf::default_logger().set_pattern("%v"); - } - ~LoggerTest() override - { - cudf::default_logger().set_pattern("[%6t][%H:%M:%S:%f][%-6l] %v"); - cudf::default_logger().set_level(prev_level); - cudf::default_logger().sinks().pop_back(); - } - - void clear_sink() { oss.str(""); } - std::string sink_content() { return oss.str(); } -}; - -TEST_F(LoggerTest, Basic) -{ - cudf::default_logger().critical("crit msg"); - ASSERT_EQ(this->sink_content(), "crit msg\n"); -} - -TEST_F(LoggerTest, DefaultLevel) -{ - cudf::default_logger().trace("trace"); - cudf::default_logger().debug("debug"); - cudf::default_logger().info("info"); - cudf::default_logger().warn("warn"); - cudf::default_logger().error("error"); - cudf::default_logger().critical("critical"); - ASSERT_EQ(this->sink_content(), "warn\nerror\ncritical\n"); -} - -TEST_F(LoggerTest, CustomLevel) -{ - cudf::default_logger().set_level(cudf::level_enum::warn); - cudf::default_logger().info("info"); - cudf::default_logger().warn("warn"); - ASSERT_EQ(this->sink_content(), "warn\n"); - - this->clear_sink(); - - cudf::default_logger().set_level(cudf::level_enum::debug); - cudf::default_logger().trace("trace"); - cudf::default_logger().debug("debug"); - ASSERT_EQ(this->sink_content(), "debug\n"); -} diff --git a/dependencies.yaml b/dependencies.yaml index b1378fae6d7..db3ce1e535d 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -24,6 +24,7 @@ files: - depends_on_libkvikio - depends_on_librmm - depends_on_nvcomp + - depends_on_rapids_logger - depends_on_rmm - develop - docs @@ -177,6 +178,7 @@ files: - build_cpp - depends_on_libkvikio - depends_on_librmm + - depends_on_rapids_logger py_run_libcudf: output: pyproject pyproject_dir: python/libcudf @@ -184,7 +186,9 @@ files: table: project includes: - depends_on_libkvikio + - depends_on_librmm - depends_on_nvcomp + - depends_on_rapids_logger py_build_pylibcudf: output: pyproject pyproject_dir: python/pylibcudf @@ -424,7 +428,6 @@ dependencies: common: - output_types: conda packages: - - fmt>=11.0.2,<12 - flatbuffers==24.3.25 - librdkafka>=2.5.0,<2.6.0a0 depends_on_nvcomp: @@ -1141,6 +1144,15 @@ dependencies: - matrix: packages: - *rmm_unsuffixed + depends_on_rapids_logger: + common: + - output_types: [conda, requirements, pyproject] + packages: + - rapids-logger==0.1.*,>=0.0.0a0 + - output_types: requirements + packages: + # pip recognizes the index as a global option for the requirements.txt file + - --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple test_python_pandas_cudf: common: - output_types: [requirements, pyproject] diff --git a/python/libcudf/libcudf/load.py b/python/libcudf/libcudf/load.py index c3ff5534e87..4198fcbe385 100644 --- a/python/libcudf/libcudf/load.py +++ b/python/libcudf/libcudf/load.py @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -47,9 +47,14 @@ def _load_wheel_installation(soname: str): def load_library(): """Dynamically load libcudf.so and its dependencies""" try: - # libkvikio must be loaded before libcudf because libcudf references its symbols + # librmm and libkvikio must be loaded before libcudf because libcudf references + # them. import libkvikio + import librmm + import rapids_logger + rapids_logger.load_library() + librmm.load_library() libkvikio.load_library() except ModuleNotFoundError: # libcudf's runtime dependency on libkvikio may be satisfied by a natively diff --git a/python/libcudf/pyproject.toml b/python/libcudf/pyproject.toml index d64765dcd42..a4e655ebbca 100644 --- a/python/libcudf/pyproject.toml +++ b/python/libcudf/pyproject.toml @@ -39,7 +39,9 @@ classifiers = [ ] dependencies = [ "libkvikio==25.4.*,>=0.0.0a0", + "librmm==25.4.*,>=0.0.0a0", "nvidia-nvcomp==4.1.0.6", + "rapids-logger==0.1.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. [project.urls] @@ -81,4 +83,5 @@ requires = [ "libkvikio==25.4.*,>=0.0.0a0", "librmm==25.4.*,>=0.0.0a0", "ninja", + "rapids-logger==0.1.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. From 18533b20ab249abc18fdd158c5563bf8b2817a71 Mon Sep 17 00:00:00 2001 From: GALI PREM SAGAR Date: Tue, 11 Feb 2025 11:00:26 -0600 Subject: [PATCH 018/129] Fix `to_pandas` writable bug for `datetime` and `timedelta` types (#17913) Fixes: #17743 This PR fixes writable flag for numpy arrays. This bug is only specific to these two types. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/17913 --- python/cudf/cudf/core/column/datetime.py | 22 ++++++++++++++++++++++ python/cudf/cudf/core/column/timedelta.py | 22 ++++++++++++++++++++++ python/cudf/cudf/tests/test_datetime.py | 16 +++++++++++++++- python/cudf/cudf/tests/test_timedelta.py | 14 ++++++++++++++ 4 files changed, 73 insertions(+), 1 deletion(-) diff --git a/python/cudf/cudf/core/column/datetime.py b/python/cudf/cudf/core/column/datetime.py index dd1662e2105..8fd2a2b68d5 100644 --- a/python/cudf/cudf/core/column/datetime.py +++ b/python/cudf/cudf/core/column/datetime.py @@ -982,6 +982,28 @@ def tz_convert(self, tz: str | None): "Cannot convert tz-naive timestamps, use tz_localize to localize" ) + def to_pandas( + self, + *, + nullable: bool = False, + arrow_type: bool = False, + ) -> pd.Index: + if arrow_type and nullable: + raise ValueError( + f"{arrow_type=} and {nullable=} cannot both be set." + ) + elif nullable: + raise NotImplementedError(f"{nullable=} is not implemented.") + pa_array = self.to_arrow() + if arrow_type: + return pd.Index(pd.arrays.ArrowExtensionArray(pa_array)) + else: + # Workaround for datetime types until the following issue is fixed: + # https://github.com/apache/arrow/issues/45341 + return pd.Index( + pa_array.to_numpy(zero_copy_only=False, writable=True) + ) + class DatetimeTZColumn(DatetimeColumn): def __init__( diff --git a/python/cudf/cudf/core/column/timedelta.py b/python/cudf/cudf/core/column/timedelta.py index 0237b1bb840..022cfe2fe2e 100644 --- a/python/cudf/cudf/core/column/timedelta.py +++ b/python/cudf/cudf/core/column/timedelta.py @@ -152,6 +152,28 @@ def element_indexing(self, index: int): return pd.Timedelta(result) return result + def to_pandas( + self, + *, + nullable: bool = False, + arrow_type: bool = False, + ) -> pd.Index: + if arrow_type and nullable: + raise ValueError( + f"{arrow_type=} and {nullable=} cannot both be set." + ) + elif nullable: + raise NotImplementedError(f"{nullable=} is not implemented.") + pa_array = self.to_arrow() + if arrow_type: + return pd.Index(pd.arrays.ArrowExtensionArray(pa_array)) + else: + # Workaround for timedelta types until the following issue is fixed: + # https://github.com/apache/arrow/issues/45341 + return pd.Index( + pa_array.to_numpy(zero_copy_only=False, writable=True) + ) + @acquire_spill_lock() def to_arrow(self) -> pa.Array: mask = None diff --git a/python/cudf/cudf/tests/test_datetime.py b/python/cudf/cudf/tests/test_datetime.py index b7403c12bcd..f8fb5ccae25 100644 --- a/python/cudf/cudf/tests/test_datetime.py +++ b/python/cudf/cudf/tests/test_datetime.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019-2024, NVIDIA CORPORATION. +# Copyright (c) 2019-2025, NVIDIA CORPORATION. import datetime import operator @@ -2563,3 +2563,17 @@ def test_date_range_start_end_divisible_by_freq(): result = cudf.date_range("2011-01-01", "2011-01-02", freq="h") expected = pd.date_range("2011-01-01", "2011-01-02", freq="h") assert_eq(result, expected) + + +def test_writable_numpy_array(): + gi = cudf.Index([1, 2, 3], dtype="datetime64[ns]") + expected_flags = pd.Index( + [1, 2, 3], dtype="datetime64[ns]" + )._data._ndarray.flags + + actual_flags = gi.to_pandas()._data._ndarray.flags + assert expected_flags.c_contiguous == actual_flags.c_contiguous + assert expected_flags.f_contiguous == actual_flags.f_contiguous + assert expected_flags.writeable == actual_flags.writeable + assert expected_flags.aligned == actual_flags.aligned + assert expected_flags.writebackifcopy == actual_flags.writebackifcopy diff --git a/python/cudf/cudf/tests/test_timedelta.py b/python/cudf/cudf/tests/test_timedelta.py index f1da2a060ec..35e0e375b46 100644 --- a/python/cudf/cudf/tests/test_timedelta.py +++ b/python/cudf/cudf/tests/test_timedelta.py @@ -1528,3 +1528,17 @@ def test_timedelta_index_total_seconds(request, data, dtype): expected = pi.total_seconds() actual = gi.total_seconds() assert_eq(expected, actual) + + +def test_writable_numpy_array(): + gi = cudf.Index([1, 2, 3], dtype="timedelta64[ns]") + expected_flags = pd.Index( + [1, 2, 3], dtype="timedelta64[ns]" + )._data._ndarray.flags + + actual_flags = gi.to_pandas()._data._ndarray.flags + assert expected_flags.c_contiguous == actual_flags.c_contiguous + assert expected_flags.f_contiguous == actual_flags.f_contiguous + assert expected_flags.writeable == actual_flags.writeable + assert expected_flags.aligned == actual_flags.aligned + assert expected_flags.writebackifcopy == actual_flags.writebackifcopy From 9ae04a792216b889eb64b398f1867cda283150d4 Mon Sep 17 00:00:00 2001 From: Vukasin Milovanovic Date: Tue, 11 Feb 2025 13:12:57 -0800 Subject: [PATCH 019/129] Fix the index type in the indexing operator of the span types (#17971) Closes https://github.com/rapidsai/cudf/issues/17949 Closes https://github.com/rapidsai/cudf/issues/17960 Derived span classes use `size_type` for the index type in their `operator[]` implementations. The intent was to use `base::size_type`, but the type actually resolves to `cudf::size_type`, which is `int32_t`, and does not allow access past `int32_t::max`. This PR fixes the type used by explicitly using `typename base::size_type`. Also added static_asserts to make sure the type has the right size for element indexing. --- cpp/include/cudf/utilities/span.hpp | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/cpp/include/cudf/utilities/span.hpp b/cpp/include/cudf/utilities/span.hpp index b5044a58934..9fd36ec8955 100644 --- a/cpp/include/cudf/utilities/span.hpp +++ b/cpp/include/cudf/utilities/span.hpp @@ -278,7 +278,11 @@ struct host_span : public cudf::detail::span_base_data[idx]; } + constexpr typename base::reference operator[](typename base::size_type idx) const + { + static_assert(sizeof(idx) >= sizeof(size_t), "index type must not be smaller than size_t"); + return this->_data[idx]; + } // not noexcept due to undefined behavior when size = 0 /** @@ -402,8 +406,9 @@ struct device_span : public cudf::detail::span_base= sizeof(size_t), "index type must not be smaller than size_t"); return this->_data[idx]; } @@ -512,7 +517,7 @@ class base_2dspan { * @param row the index of the element to access * @return A reference to the row-th element of the sequence, i.e., `data()[row]` */ - CUDF_HOST_DEVICE constexpr RowType operator[](size_t row) const + CUDF_HOST_DEVICE constexpr RowType operator[](std::size_t row) const { return _flat.subspan(row * _size.second, _size.second); } From 4b2ce98745187f1367cc8c461c6a6a18f42e2d1b Mon Sep 17 00:00:00 2001 From: Yunsong Wang Date: Tue, 11 Feb 2025 13:13:25 -0800 Subject: [PATCH 020/129] Fix race check failures in shared memory groupby (#17985) Replace https://github.com/rapidsai/cudf/pull/17976 This fixes the race check failures in shared memory groupby and resolves https://github.com/NVIDIA/spark-rapids/issues/11835. --- cpp/src/groupby/hash/compute_aggregations.cuh | 2 +- .../hash/compute_shared_memory_aggs.cu | 8 +++---- .../hash/compute_shared_memory_aggs.hpp | 10 ++++----- .../groupby/hash/shared_memory_aggregator.cuh | 21 ++++++++++++------- 4 files changed, 23 insertions(+), 18 deletions(-) diff --git a/cpp/src/groupby/hash/compute_aggregations.cuh b/cpp/src/groupby/hash/compute_aggregations.cuh index df8fcf4690f..b97c8ddf88d 100644 --- a/cpp/src/groupby/hash/compute_aggregations.cuh +++ b/cpp/src/groupby/hash/compute_aggregations.cuh @@ -76,7 +76,7 @@ rmm::device_uvector compute_aggregations( // for shared memory aggregations auto const size = cudf::type_dispatcher(request.values.type(), size_of_functor{}); - return static_cast(data_buffer_size) >= (size * GROUPBY_CARDINALITY_THRESHOLD); + return data_buffer_size >= (size * GROUPBY_CARDINALITY_THRESHOLD); }); // Performs naive global memory aggregations when the workload is not compatible with shared diff --git a/cpp/src/groupby/hash/compute_shared_memory_aggs.cu b/cpp/src/groupby/hash/compute_shared_memory_aggs.cu index ae7584da483..26c89aec14e 100644 --- a/cpp/src/groupby/hash/compute_shared_memory_aggs.cu +++ b/cpp/src/groupby/hash/compute_shared_memory_aggs.cu @@ -99,7 +99,6 @@ __device__ void initialize_shmem_aggregations(cooperative_groups::thread_block c idx); } } - block.sync(); } __device__ void compute_pre_aggregrations(cudf::size_type col_start, @@ -174,7 +173,6 @@ __device__ void compute_final_aggregations(cooperative_groups::thread_block cons idx); } } - block.sync(); } /* Takes the local_mapping_index and global_mapping_index to compute @@ -213,6 +211,7 @@ CUDF_KERNEL void single_pass_shmem_aggs_kernel(cudf::size_type num_rows, block.sync(); while (col_end < num_cols) { + block.sync(); if (block.thread_rank() == 0) { calculate_columns_to_aggregate(col_start, col_end, @@ -234,6 +233,7 @@ CUDF_KERNEL void single_pass_shmem_aggs_kernel(cudf::size_type num_rows, shmem_agg_mask_offsets, cardinality, d_agg_kinds); + block.sync(); compute_pre_aggregrations(col_start, col_end, @@ -263,7 +263,7 @@ CUDF_KERNEL void single_pass_shmem_aggs_kernel(cudf::size_type num_rows, } } // namespace -std::size_t get_available_shared_memory_size(cudf::size_type grid_size) +size_type get_available_shared_memory_size(cudf::size_type grid_size) { auto const active_blocks_per_sm = cudf::util::div_rounding_up_safe(grid_size, cudf::detail::num_multiprocessors()); @@ -276,7 +276,7 @@ std::size_t get_available_shared_memory_size(cudf::size_type grid_size) } void compute_shared_memory_aggs(cudf::size_type grid_size, - std::size_t available_shmem_size, + size_type available_shmem_size, cudf::size_type num_input_rows, bitmask_type const* row_bitmask, bool skip_rows_with_nulls, diff --git a/cpp/src/groupby/hash/compute_shared_memory_aggs.hpp b/cpp/src/groupby/hash/compute_shared_memory_aggs.hpp index 346956cdab0..b6ba6898c07 100644 --- a/cpp/src/groupby/hash/compute_shared_memory_aggs.hpp +++ b/cpp/src/groupby/hash/compute_shared_memory_aggs.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -22,15 +22,15 @@ #include namespace cudf::groupby::detail::hash { -std::size_t get_available_shared_memory_size(cudf::size_type grid_size); +size_type get_available_shared_memory_size(cudf::size_type grid_size); -std::size_t constexpr compute_shmem_offsets_size(cudf::size_type num_cols) +size_type constexpr compute_shmem_offsets_size(cudf::size_type num_cols) { - return sizeof(cudf::size_type) * num_cols; + return static_cast(sizeof(cudf::size_type) * num_cols); } void compute_shared_memory_aggs(cudf::size_type grid_size, - std::size_t available_shmem_size, + cudf::size_type available_shmem_size, cudf::size_type num_input_rows, bitmask_type const* row_bitmask, bool skip_rows_with_nulls, diff --git a/cpp/src/groupby/hash/shared_memory_aggregator.cuh b/cpp/src/groupby/hash/shared_memory_aggregator.cuh index 9cbeeb34b86..559702533e8 100644 --- a/cpp/src/groupby/hash/shared_memory_aggregator.cuh +++ b/cpp/src/groupby/hash/shared_memory_aggregator.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -25,6 +25,11 @@ #include namespace cudf::groupby::detail::hash { +__device__ constexpr void set_mask(bool* mask) +{ + if (not *mask) { cudf::detail::atomic_max(mask, true); } +} + template struct update_target_element_shmem { __device__ void operator()( @@ -52,7 +57,7 @@ struct update_target_element_shmem< cudf::detail::atomic_min(&target_casted[target_index], static_cast(source.element(source_index))); - if (!target_mask[target_index]) { target_mask[target_index] = true; } + set_mask(target_mask + target_index); } }; @@ -74,7 +79,7 @@ struct update_target_element_shmem< cudf::detail::atomic_max(&target_casted[target_index], static_cast(source.element(source_index))); - if (!target_mask[target_index]) { target_mask[target_index] = true; } + set_mask(target_mask + target_index); } }; @@ -97,7 +102,7 @@ struct update_target_element_shmem< cudf::detail::atomic_add(&target_casted[target_index], static_cast(source.element(source_index))); - if (!target_mask[target_index]) { target_mask[target_index] = true; } + set_mask(target_mask + target_index); } }; @@ -117,7 +122,7 @@ struct update_target_element_shmem< auto value = static_cast(source.element(source_index)); cudf::detail::atomic_add(&target_casted[target_index], value * value); - if (!target_mask[target_index]) { target_mask[target_index] = true; } + set_mask(target_mask + target_index); } }; @@ -137,7 +142,7 @@ struct update_target_element_shmem< cudf::detail::atomic_mul(&target_casted[target_index], static_cast(source.element(source_index))); - if (!target_mask[target_index]) { target_mask[target_index] = true; } + set_mask(target_mask + target_index); } }; @@ -202,7 +207,7 @@ struct update_target_element_shmem< } } - if (!target_mask[target_index]) { target_mask[target_index] = true; } + set_mask(target_mask + target_index); } }; @@ -228,7 +233,7 @@ struct update_target_element_shmem< } } - if (!target_mask[target_index]) { target_mask[target_index] = true; } + set_mask(target_mask + target_index); } }; From 4e79029a868bcda01d3c18dab4c3cc1c7ddea5bb Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Wed, 12 Feb 2025 08:42:40 -0500 Subject: [PATCH 021/129] Add missing include for calling std::iota() (#17983) Recent build failure in #17600 indicated undefined `std::iota`. This PR adds the appropriate `#include ` in the source files where this is called. Authors: - David Wendt (https://github.com/davidwendt) - Nghia Truong (https://github.com/ttnghia) Approvers: - Bradley Dice (https://github.com/bdice) - Yunsong Wang (https://github.com/PointKernel) - Muhammad Haseeb (https://github.com/mhaseeb123) URL: https://github.com/rapidsai/cudf/pull/17983 --- cpp/benchmarks/hashing/partition.cpp | 3 ++- cpp/benchmarks/io/text/multibyte_split.cpp | 3 ++- cpp/examples/billion_rows/brc_pipeline.cpp | 3 ++- cpp/tests/copying/gather_tests.cpp | 4 +++- cpp/tests/copying/reverse_tests.cpp | 4 +++- cpp/tests/fixed_point/fixed_point_tests.cu | 3 ++- cpp/tests/io/orc_test.cpp | 1 + cpp/tests/join/conditional_join_tests.cu | 3 ++- cpp/tests/join/distinct_join_tests.cpp | 1 + cpp/tests/join/mixed_join_tests.cu | 3 ++- cpp/tests/partitioning/hash_partition_test.cpp | 4 +++- cpp/tests/reductions/reduction_tests.cpp | 3 ++- cpp/tests/search/search_test.cpp | 4 +++- cpp/tests/sort/segmented_sort_tests.cpp | 3 ++- cpp/tests/streams/column_view_test.cpp | 1 + cpp/tests/strings/integers_tests.cpp | 3 ++- cpp/tests/unary/math_ops_test.cpp | 1 + 17 files changed, 34 insertions(+), 13 deletions(-) diff --git a/cpp/benchmarks/hashing/partition.cpp b/cpp/benchmarks/hashing/partition.cpp index 0bec4394216..2ef66a1e619 100644 --- a/cpp/benchmarks/hashing/partition.cpp +++ b/cpp/benchmarks/hashing/partition.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2023, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -21,6 +21,7 @@ #include #include +#include class Hashing : public cudf::benchmark {}; diff --git a/cpp/benchmarks/io/text/multibyte_split.cpp b/cpp/benchmarks/io/text/multibyte_split.cpp index 4bfef9767ca..228ebf3a756 100644 --- a/cpp/benchmarks/io/text/multibyte_split.cpp +++ b/cpp/benchmarks/io/text/multibyte_split.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -38,6 +38,7 @@ #include #include #include +#include #include temp_directory const temp_dir("cudf_nvbench"); diff --git a/cpp/examples/billion_rows/brc_pipeline.cpp b/cpp/examples/billion_rows/brc_pipeline.cpp index c65edc163e1..2b10f18f99e 100644 --- a/cpp/examples/billion_rows/brc_pipeline.cpp +++ b/cpp/examples/billion_rows/brc_pipeline.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -31,6 +31,7 @@ #include #include #include +#include using elapsed_t = std::chrono::duration; using byte_range = std::pair; diff --git a/cpp/tests/copying/gather_tests.cpp b/cpp/tests/copying/gather_tests.cpp index 908dcd67673..e709ca1e7f2 100644 --- a/cpp/tests/copying/gather_tests.cpp +++ b/cpp/tests/copying/gather_tests.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -27,6 +27,8 @@ #include #include +#include + template class GatherTest : public cudf::test::BaseFixture {}; diff --git a/cpp/tests/copying/reverse_tests.cpp b/cpp/tests/copying/reverse_tests.cpp index 46516436901..fc40193b3cf 100644 --- a/cpp/tests/copying/reverse_tests.cpp +++ b/cpp/tests/copying/reverse_tests.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -28,6 +28,8 @@ #include #include +#include + constexpr cudf::test::debug_output_level verbosity{cudf::test::debug_output_level::ALL_ERRORS}; template diff --git a/cpp/tests/fixed_point/fixed_point_tests.cu b/cpp/tests/fixed_point/fixed_point_tests.cu index ddc48c97012..9915af66eca 100644 --- a/cpp/tests/fixed_point/fixed_point_tests.cu +++ b/cpp/tests/fixed_point/fixed_point_tests.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -33,6 +33,7 @@ #include #include +#include #include using namespace numeric; diff --git a/cpp/tests/io/orc_test.cpp b/cpp/tests/io/orc_test.cpp index 362a6df913d..82ed068aff7 100644 --- a/cpp/tests/io/orc_test.cpp +++ b/cpp/tests/io/orc_test.cpp @@ -38,6 +38,7 @@ #include #include +#include #include namespace nvcomp = cudf::io::detail::nvcomp; diff --git a/cpp/tests/join/conditional_join_tests.cu b/cpp/tests/join/conditional_join_tests.cu index 7ab4a2ea465..69907b67f5a 100644 --- a/cpp/tests/join/conditional_join_tests.cu +++ b/cpp/tests/join/conditional_join_tests.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -34,6 +34,7 @@ #include #include +#include #include #include #include diff --git a/cpp/tests/join/distinct_join_tests.cpp b/cpp/tests/join/distinct_join_tests.cpp index e1ec8cda3ac..8f48ccce084 100644 --- a/cpp/tests/join/distinct_join_tests.cpp +++ b/cpp/tests/join/distinct_join_tests.cpp @@ -27,6 +27,7 @@ #include #include +#include #include template diff --git a/cpp/tests/join/mixed_join_tests.cu b/cpp/tests/join/mixed_join_tests.cu index 9041969bec7..f3678b996bf 100644 --- a/cpp/tests/join/mixed_join_tests.cu +++ b/cpp/tests/join/mixed_join_tests.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -32,6 +32,7 @@ #include #include +#include #include #include #include diff --git a/cpp/tests/partitioning/hash_partition_test.cpp b/cpp/tests/partitioning/hash_partition_test.cpp index 579d918a31d..30701d7ac16 100644 --- a/cpp/tests/partitioning/hash_partition_test.cpp +++ b/cpp/tests/partitioning/hash_partition_test.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -31,6 +31,8 @@ #include #include +#include + using cudf::test::fixed_width_column_wrapper; using cudf::test::strings_column_wrapper; using structs_col = cudf::test::structs_column_wrapper; diff --git a/cpp/tests/reductions/reduction_tests.cpp b/cpp/tests/reductions/reduction_tests.cpp index 67083f19b3a..db6adb8d118 100644 --- a/cpp/tests/reductions/reduction_tests.cpp +++ b/cpp/tests/reductions/reduction_tests.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -35,6 +35,7 @@ #include #include +#include #include using aggregation = cudf::aggregation; diff --git a/cpp/tests/search/search_test.cpp b/cpp/tests/search/search_test.cpp index 8d750be5677..ea566fcbd38 100644 --- a/cpp/tests/search/search_test.cpp +++ b/cpp/tests/search/search_test.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -24,6 +24,8 @@ #include +#include + struct SearchTest : public cudf::test::BaseFixture {}; using cudf::numeric_scalar; diff --git a/cpp/tests/sort/segmented_sort_tests.cpp b/cpp/tests/sort/segmented_sort_tests.cpp index 79421a1fa30..640dd8d578e 100644 --- a/cpp/tests/sort/segmented_sort_tests.cpp +++ b/cpp/tests/sort/segmented_sort_tests.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -26,6 +26,7 @@ #include #include +#include #include #include diff --git a/cpp/tests/streams/column_view_test.cpp b/cpp/tests/streams/column_view_test.cpp index c7483223973..f9dc5a86405 100644 --- a/cpp/tests/streams/column_view_test.cpp +++ b/cpp/tests/streams/column_view_test.cpp @@ -23,6 +23,7 @@ #include #include +#include #include #include diff --git a/cpp/tests/strings/integers_tests.cpp b/cpp/tests/strings/integers_tests.cpp index c08effdb969..92ba4231ea6 100644 --- a/cpp/tests/strings/integers_tests.cpp +++ b/cpp/tests/strings/integers_tests.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -28,6 +28,7 @@ #include #include +#include #include #include diff --git a/cpp/tests/unary/math_ops_test.cpp b/cpp/tests/unary/math_ops_test.cpp index bcb84d4574c..d80765a3ced 100644 --- a/cpp/tests/unary/math_ops_test.cpp +++ b/cpp/tests/unary/math_ops_test.cpp @@ -23,6 +23,7 @@ #include #include +#include #include using TypesToNegate = cudf::test::Types Date: Wed, 12 Feb 2025 12:45:49 -0600 Subject: [PATCH 022/129] Compatibility with Dask `main` (#17992) Moves the import of get_collection_type. It's been available from `dask.dataframe` since at least `dask>=2024.12.1` xref https://github.com/dask/dask/pull/11689 and the CI failure at https://github.com/rapidsai/cudf/actions/runs/13286927884/job/37100164792?pr=17929 --- python/dask_cudf/dask_cudf/_expr/__init__.py | 167 ++++++++---------- .../dask_cudf/dask_cudf/_expr/collection.py | 2 +- 2 files changed, 76 insertions(+), 93 deletions(-) diff --git a/python/dask_cudf/dask_cudf/_expr/__init__.py b/python/dask_cudf/dask_cudf/_expr/__init__.py index 329ad861081..1f757476ce5 100644 --- a/python/dask_cudf/dask_cudf/_expr/__init__.py +++ b/python/dask_cudf/dask_cudf/_expr/__init__.py @@ -1,96 +1,79 @@ # Copyright (c) 2024-2025, NVIDIA CORPORATION. -from packaging.version import Version - import dask +import dask.dataframe.dask_expr._shuffle as _shuffle_module +from dask.dataframe import get_collection_type +from dask.dataframe.dask_expr import ( + DataFrame as DXDataFrame, + FrameBase, + Index as DXIndex, + Series as DXSeries, + from_dict, + new_collection, +) +from dask.dataframe.dask_expr._cumulative import ( + CumulativeBlockwise, +) +from dask.dataframe.dask_expr._expr import ( + Elemwise, + Expr, + RenameAxis, + VarColumns, +) +from dask.dataframe.dask_expr._groupby import ( + DecomposableGroupbyAggregation, + GroupBy as DXGroupBy, + GroupbyAggregation, + SeriesGroupBy as DXSeriesGroupBy, + SingleAggregation, +) +from dask.dataframe.dask_expr._reductions import ( + Reduction, + Var, +) +from dask.dataframe.dask_expr._util import ( + _convert_to_list, + _raise_if_object_series, + is_scalar, +) +from dask.dataframe.dask_expr.io.io import ( + FusedIO, + FusedParquetIO, +) +from dask.dataframe.dask_expr.io.parquet import ( + FragmentWrapper, + ReadParquetFSSpec, + ReadParquetPyarrowFS, +) -if Version(dask.__version__) > Version("2024.12.1"): - import dask.dataframe.dask_expr._shuffle as _shuffle_module - from dask.dataframe.dask_expr import ( - DataFrame as DXDataFrame, - FrameBase, - Index as DXIndex, - Series as DXSeries, - from_dict, - get_collection_type, - new_collection, - ) - from dask.dataframe.dask_expr._cumulative import ( - CumulativeBlockwise, - ) - from dask.dataframe.dask_expr._expr import ( - Elemwise, - Expr, - RenameAxis, - VarColumns, - ) - from dask.dataframe.dask_expr._groupby import ( - DecomposableGroupbyAggregation, - GroupBy as DXGroupBy, - GroupbyAggregation, - SeriesGroupBy as DXSeriesGroupBy, - SingleAggregation, - ) - from dask.dataframe.dask_expr._reductions import ( - Reduction, - Var, - ) - from dask.dataframe.dask_expr._util import ( - _convert_to_list, - _raise_if_object_series, - is_scalar, - ) - from dask.dataframe.dask_expr.io.io import ( - FusedIO, - FusedParquetIO, - ) - from dask.dataframe.dask_expr.io.parquet import ( - FragmentWrapper, - ReadParquetFSSpec, - ReadParquetPyarrowFS, - ) -else: - import dask_expr._shuffle as _shuffle_module # noqa: F401 - from dask_expr import ( # noqa: F401 - DataFrame as DXDataFrame, - FrameBase, - Index as DXIndex, - Series as DXSeries, - from_dict, - get_collection_type, - new_collection, - ) - from dask_expr._cumulative import CumulativeBlockwise # noqa: F401 - from dask_expr._expr import ( # noqa: F401 - Elemwise, - Expr, - RenameAxis, - VarColumns, - ) - from dask_expr._groupby import ( # noqa: F401 - DecomposableGroupbyAggregation, - GroupBy as DXGroupBy, - GroupbyAggregation, - SeriesGroupBy as DXSeriesGroupBy, - SingleAggregation, - ) - from dask_expr._reductions import Reduction, Var # noqa: F401 - from dask_expr._util import ( # noqa: F401 - _convert_to_list, - _raise_if_object_series, - is_scalar, - ) - from dask_expr.io.io import FusedIO, FusedParquetIO # noqa: F401 - from dask_expr.io.parquet import ( # noqa: F401 - FragmentWrapper, - ReadParquetFSSpec, - ReadParquetPyarrowFS, - ) - - from dask.dataframe import _dask_expr_enabled - - if not _dask_expr_enabled(): - raise ValueError( - "The legacy DataFrame API is not supported for RAPIDS >24.12. " - "The 'dataframe.query-planning' config must be True or None." - ) +__all__ = [ + "CumulativeBlockwise", + "DXDataFrame", + "DXGroupBy", + "DXIndex", + "DXSeries", + "DXSeriesGroupBy", + "DecomposableGroupbyAggregation", + "Elemwise", + "Expr", + "FragmentWrapper", + "FrameBase", + "FusedIO", + "FusedParquetIO", + "GroupbyAggregation", + "ReadParquetFSSpec", + "ReadParquetPyarrowFS", + "Reduction", + "RenameAxis", + "SingleAggregation", + "Var", + "VarColumns", + "_convert_to_list", + "_raise_if_object_series", + "_shuffle_module", + "dask", + "from_dict", + "get_collection_type", + "is_scalar", + "new_collection", +] diff --git a/python/dask_cudf/dask_cudf/_expr/collection.py b/python/dask_cudf/dask_cudf/_expr/collection.py index c3e44567abf..9e42ea2d650 100644 --- a/python/dask_cudf/dask_cudf/_expr/collection.py +++ b/python/dask_cudf/dask_cudf/_expr/collection.py @@ -4,6 +4,7 @@ from functools import cached_property from dask import config +from dask.dataframe import get_collection_type from dask.dataframe.core import is_dataframe_like from dask.dataframe.dispatch import get_parallel_type from dask.typing import no_default @@ -16,7 +17,6 @@ DXSeries, FrameBase, _raise_if_object_series, - get_collection_type, new_collection, ) From aab7edb26b57755311220fca0dc2c933b1695061 Mon Sep 17 00:00:00 2001 From: Muhammad Haseeb <14217455+mhaseeb123@users.noreply.github.com> Date: Wed, 12 Feb 2025 14:10:43 -0800 Subject: [PATCH 023/129] Remove decimal32/64 to decimal128 conversion in Parquet writer (#17869) Fixes #17080. Related to #17422 This PR removes the decimal32/64 to decimal128 conversion in Parquet writer as it's no longer needed with Arrow v19. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17869 --- cpp/CMakeLists.txt | 1 - .../interop/decimal_conversion_utilities.cu | 70 ---------- .../interop/decimal_conversion_utilities.cuh | 44 ------ cpp/src/interop/to_arrow_device.cu | 1 - cpp/src/interop/to_arrow_host.cu | 1 - cpp/src/io/parquet/arrow_schema_writer.cpp | 29 ++-- cpp/src/io/parquet/ipc/Schema_generated.h | 8 +- cpp/src/io/parquet/ipc/schema/Schema.fbs | 7 +- cpp/src/io/parquet/writer_impl.cu | 130 ++---------------- cpp/tests/io/parquet_writer_test.cpp | 89 ++++-------- 10 files changed, 72 insertions(+), 308 deletions(-) delete mode 100644 cpp/src/interop/decimal_conversion_utilities.cu delete mode 100644 cpp/src/interop/decimal_conversion_utilities.cuh diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index 56b97f6ce00..a282c12d97f 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -464,7 +464,6 @@ add_library( src/hash/xxhash_64.cu src/interop/dlpack.cpp src/interop/arrow_utilities.cpp - src/interop/decimal_conversion_utilities.cu src/interop/to_arrow_device.cu src/interop/to_arrow_host.cu src/interop/from_arrow_device.cu diff --git a/cpp/src/interop/decimal_conversion_utilities.cu b/cpp/src/interop/decimal_conversion_utilities.cu deleted file mode 100644 index 2f81c754a30..00000000000 --- a/cpp/src/interop/decimal_conversion_utilities.cu +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Copyright (c) 2024, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include "decimal_conversion_utilities.cuh" - -#include -#include -#include - -#include - -#include - -#include - -namespace cudf { -namespace detail { - -template -std::unique_ptr convert_decimals_to_decimal128( - cudf::column_view const& column, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) -{ - static_assert(std::is_same_v or std::is_same_v, - "Only int32 and int64 decimal types can be converted to decimal128."); - - constexpr size_type BIT_WIDTH_RATIO = sizeof(__int128_t) / sizeof(DecimalType); - auto buf = std::make_unique(column.size() * sizeof(__int128_t), stream, mr); - - thrust::for_each(rmm::exec_policy_nosync(stream, mr), - thrust::make_counting_iterator(0), - thrust::make_counting_iterator(column.size()), - [in = column.begin(), - out = reinterpret_cast(buf->data()), - BIT_WIDTH_RATIO] __device__(auto in_idx) { - auto const out_idx = in_idx * BIT_WIDTH_RATIO; - // the lowest order bits are the value, the remainder - // simply matches the sign bit to satisfy the two's - // complement integer representation of negative numbers. - out[out_idx] = in[in_idx]; -#pragma unroll BIT_WIDTH_RATIO - 1 - for (auto i = 1; i < BIT_WIDTH_RATIO; ++i) { - out[out_idx + i] = in[in_idx] < 0 ? -1 : 0; - } - }); - - return buf; -} - -// Instantiate templates for int32_t and int64_t decimal types -template std::unique_ptr convert_decimals_to_decimal128( - cudf::column_view const& column, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr); - -template std::unique_ptr convert_decimals_to_decimal128( - cudf::column_view const& column, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr); - -} // namespace detail -} // namespace cudf diff --git a/cpp/src/interop/decimal_conversion_utilities.cuh b/cpp/src/interop/decimal_conversion_utilities.cuh deleted file mode 100644 index 6b62eb0fee4..00000000000 --- a/cpp/src/interop/decimal_conversion_utilities.cuh +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Copyright (c) 2024, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#pragma once - -#include -#include -#include - -#include - -#include - -namespace cudf::detail { - -/** - * @brief Convert decimal32 and decimal64 numeric data to decimal128 and return the device vector - * - * @tparam DecimalType to convert from - * - * @param column A view of the input columns - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource to use for device memory allocation - * - * @return A device vector containing the converted decimal128 data - */ -template -std::unique_ptr convert_decimals_to_decimal128( - cudf::column_view const& input, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr); - -} // namespace cudf::detail diff --git a/cpp/src/interop/to_arrow_device.cu b/cpp/src/interop/to_arrow_device.cu index 17eff1128f6..3683285a89b 100644 --- a/cpp/src/interop/to_arrow_device.cu +++ b/cpp/src/interop/to_arrow_device.cu @@ -15,7 +15,6 @@ */ #include "arrow_utilities.hpp" -#include "decimal_conversion_utilities.cuh" #include #include diff --git a/cpp/src/interop/to_arrow_host.cu b/cpp/src/interop/to_arrow_host.cu index e93fdda0c1a..6a6b046fa1d 100644 --- a/cpp/src/interop/to_arrow_host.cu +++ b/cpp/src/interop/to_arrow_host.cu @@ -15,7 +15,6 @@ */ #include "arrow_utilities.hpp" -#include "decimal_conversion_utilities.cuh" #include #include diff --git a/cpp/src/io/parquet/arrow_schema_writer.cpp b/cpp/src/io/parquet/arrow_schema_writer.cpp index a4536ac6a3b..010af677a0f 100644 --- a/cpp/src/io/parquet/arrow_schema_writer.cpp +++ b/cpp/src/io/parquet/arrow_schema_writer.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -250,13 +250,26 @@ struct dispatch_to_flatbuf { std::enable_if_t(), void> operator()() { field_type_id = flatbuf::Type_Decimal; - field_offset = flatbuf::CreateDecimal(fbb, - (col_meta.is_decimal_precision_set()) - ? col_meta.get_decimal_precision() - : MAX_DECIMAL128_PRECISION, - col->type().scale(), - 128) - .Union(); + + auto const [max_precision, bitwidth] = []() constexpr -> std::pair { + if constexpr (std::is_same_v) { + return {MAX_DECIMAL32_PRECISION, 32}; + } else if constexpr (std::is_same_v) { + return {MAX_DECIMAL64_PRECISION, 64}; + } else if constexpr (std::is_same_v) { + return {MAX_DECIMAL128_PRECISION, 128}; + } else { + CUDF_FAIL("Unsupported fixed point type for arrow schema writer"); + } + }(); + + field_offset = + flatbuf::CreateDecimal( + fbb, + (col_meta.is_decimal_precision_set()) ? col_meta.get_decimal_precision() : max_precision, + col->type().scale(), + bitwidth) + .Union(); } template diff --git a/cpp/src/io/parquet/ipc/Schema_generated.h b/cpp/src/io/parquet/ipc/Schema_generated.h index c091204417a..a1150c9f7e0 100644 --- a/cpp/src/io/parquet/ipc/Schema_generated.h +++ b/cpp/src/io/parquet/ipc/Schema_generated.h @@ -1393,9 +1393,9 @@ inline ::flatbuffers::Offset CreateRunEndEncoded( } /// Exact decimal value represented as an integer value in two's -/// complement. Currently only 128-bit (16-byte) and 256-bit (32-byte) integers -/// are used. The representation uses the endianness indicated -/// in the Schema. +/// complement. Currently 32-bit (4-byte), 64-bit (8-byte), +/// 128-bit (16-byte) and 256-bit (32-byte) integers are used. +/// The representation uses the endianness indicated in the Schema. struct Decimal FLATBUFFERS_FINAL_CLASS : private ::flatbuffers::Table { typedef DecimalBuilder Builder; enum FlatBuffersVTableOffset FLATBUFFERS_VTABLE_UNDERLYING_TYPE { @@ -1407,7 +1407,7 @@ struct Decimal FLATBUFFERS_FINAL_CLASS : private ::flatbuffers::Table { int32_t precision() const { return GetField(VT_PRECISION, 0); } /// Number of digits after the decimal point "." int32_t scale() const { return GetField(VT_SCALE, 0); } - /// Number of bits per value. The only accepted widths are 128 and 256. + /// Number of bits per value. The accepted widths are 32, 64, 128 and 256. /// We use bitWidth for consistency with Int::bitWidth. int32_t bitWidth() const { return GetField(VT_BITWIDTH, 128); } bool Verify(::flatbuffers::Verifier& verifier) const diff --git a/cpp/src/io/parquet/ipc/schema/Schema.fbs b/cpp/src/io/parquet/ipc/schema/Schema.fbs index 5f66e7bbd5e..41739240aa4 100644 --- a/cpp/src/io/parquet/ipc/schema/Schema.fbs +++ b/cpp/src/io/parquet/ipc/schema/Schema.fbs @@ -45,6 +45,7 @@ /// Version 1.3 - Add Run-End Encoded. /// Version 1.4 - Add BinaryView, Utf8View, variadicBufferCounts, ListView, and /// LargeListView. +/// Version 1.5 - Add 32-bit and 64-bit as allowed bit widths for Decimal namespace cudf.io.parquet.flatbuf; @@ -243,9 +244,9 @@ table RunEndEncoded { } /// Exact decimal value represented as an integer value in two's -/// complement. Currently only 128-bit (16-byte) and 256-bit (32-byte) integers -/// are used. The representation uses the endianness indicated -/// in the Schema. +/// complement. Currently 32-bit (4-byte), 64-bit (8-byte), +/// 128-bit (16-byte) and 256-bit (32-byte) integers are used. +/// The representation uses the endianness indicated in the Schema. table Decimal { /// Total number of decimal digits precision: int; diff --git a/cpp/src/io/parquet/writer_impl.cu b/cpp/src/io/parquet/writer_impl.cu index 1b67b53ae8e..9e50fafa8a7 100644 --- a/cpp/src/io/parquet/writer_impl.cu +++ b/cpp/src/io/parquet/writer_impl.cu @@ -22,7 +22,6 @@ #include "arrow_schema_writer.hpp" #include "compact_protocol_reader.hpp" #include "compact_protocol_writer.hpp" -#include "interop/decimal_conversion_utilities.cuh" #include "io/comp/comp.hpp" #include "io/parquet/parquet.hpp" #include "io/parquet/parquet_gpu.hpp" @@ -568,27 +567,24 @@ struct leaf_schema_fn { template std::enable_if_t(), void> operator()() { - // If writing arrow schema, then convert d32 and d64 to d128 - if (write_arrow_schema or std::is_same_v) { + if constexpr (std::is_same_v) { + col_schema.type = Type::INT32; + col_schema.stats_dtype = statistics_dtype::dtype_int32; + col_schema.decimal_precision = MAX_DECIMAL32_PRECISION; + col_schema.logical_type = LogicalType{DecimalType{0, MAX_DECIMAL32_PRECISION}}; + } else if constexpr (std::is_same_v) { + col_schema.type = Type::INT64; + col_schema.stats_dtype = statistics_dtype::dtype_decimal64; + col_schema.decimal_precision = MAX_DECIMAL64_PRECISION; + col_schema.logical_type = LogicalType{DecimalType{0, MAX_DECIMAL64_PRECISION}}; + } else if constexpr (std::is_same_v) { col_schema.type = Type::FIXED_LEN_BYTE_ARRAY; col_schema.type_length = sizeof(__int128_t); col_schema.stats_dtype = statistics_dtype::dtype_decimal128; col_schema.decimal_precision = MAX_DECIMAL128_PRECISION; col_schema.logical_type = LogicalType{DecimalType{0, MAX_DECIMAL128_PRECISION}}; } else { - if (std::is_same_v) { - col_schema.type = Type::INT32; - col_schema.stats_dtype = statistics_dtype::dtype_int32; - col_schema.decimal_precision = MAX_DECIMAL32_PRECISION; - col_schema.logical_type = LogicalType{DecimalType{0, MAX_DECIMAL32_PRECISION}}; - } else if (std::is_same_v) { - col_schema.type = Type::INT64; - col_schema.stats_dtype = statistics_dtype::dtype_decimal64; - col_schema.decimal_precision = MAX_DECIMAL64_PRECISION; - col_schema.logical_type = LogicalType{DecimalType{0, MAX_DECIMAL64_PRECISION}}; - } else { - CUDF_FAIL("Unsupported fixed point type for parquet writer"); - } + CUDF_FAIL("Unsupported fixed point type for parquet writer"); } // Write logical and converted types, decimal scale and precision @@ -1592,97 +1588,6 @@ size_t column_index_buffer_size(EncColumnChunk* ck, return ck->ck_stat_size * num_pages + column_index_truncate_length + padding + size_struct_size; } -/** - * @brief Function to convert decimal32 and decimal64 columns to decimal128 data, - * update the input table metadata, and return a new vector of column views. - * - * @param[in,out] table_meta The table metadata - * @param[in,out] d128_buffers Buffers containing the converted decimal128 data. - * @param input The input table - * @param stream CUDA stream used for device memory operations and kernel launches - * - * @return A device vector containing the converted decimal128 data - */ -std::vector convert_decimal_columns_and_metadata( - table_input_metadata& table_meta, - std::vector>& d128_buffers, - table_view const& table, - rmm::cuda_stream_view stream) -{ - // Lambda function to convert each decimal32/decimal64 column to decimal128. - std::function convert_column = - [&](column_view column, column_in_metadata& metadata) -> column_view { - // Vector of passable-by-reference children column views - std::vector converted_children; - - // Process children column views first - std::transform( - thrust::make_counting_iterator(0), - thrust::make_counting_iterator(column.num_children()), - std::back_inserter(converted_children), - [&](auto const idx) { return convert_column(column.child(idx), metadata.child(idx)); }); - - // Process this column view. Only convert if decimal32 and decimal64 column. - switch (column.type().id()) { - case type_id::DECIMAL32: - // Convert data to decimal128 type - d128_buffers.emplace_back(cudf::detail::convert_decimals_to_decimal128( - column, stream, cudf::get_current_device_resource_ref())); - // Update metadata - metadata.set_decimal_precision(MAX_DECIMAL32_PRECISION); - metadata.set_type_length(size_of(data_type{type_id::DECIMAL128, column.type().scale()})); - // Create a new column view from the d128 data vector - return {data_type{type_id::DECIMAL128, column.type().scale()}, - column.size(), - d128_buffers.back()->data(), - column.null_mask(), - column.null_count(), - column.offset(), - converted_children}; - case type_id::DECIMAL64: - // Convert data to decimal128 type - d128_buffers.emplace_back(cudf::detail::convert_decimals_to_decimal128( - column, stream, cudf::get_current_device_resource_ref())); - // Update metadata - metadata.set_decimal_precision(MAX_DECIMAL64_PRECISION); - metadata.set_type_length(size_of(data_type{type_id::DECIMAL128, column.type().scale()})); - // Create a new column view from the d128 data vector - return {data_type{type_id::DECIMAL128, column.type().scale()}, - column.size(), - d128_buffers.back()->data(), - column.null_mask(), - column.null_count(), - column.offset(), - converted_children}; - default: - // Update the children vector keeping everything else the same - return {column.type(), - column.size(), - column.head(), - column.null_mask(), - column.null_count(), - column.offset(), - converted_children}; - } - }; - - // Vector of converted column views - std::vector converted_column_views; - - // Convert each column view - std::transform( - thrust::make_zip_iterator( - thrust::make_tuple(table.begin(), table_meta.column_metadata.begin())), - thrust::make_zip_iterator(thrust::make_tuple(table.end(), table_meta.column_metadata.end())), - std::back_inserter(converted_column_views), - [&](auto elem) { return convert_column(thrust::get<0>(elem), thrust::get<1>(elem)); }); - - // Synchronize stream here to ensure all decimal128 buffers are ready. - stream.synchronize(); - - return converted_column_views; -} - /** * @brief Perform the processing steps needed to convert the input table into the output Parquet * data for writing, such as compression and encoding. @@ -1737,15 +1642,8 @@ auto convert_table_to_parquet_data(table_input_metadata& table_meta, host_span const> out_sink, rmm::cuda_stream_view stream) { - // Container to store decimal128 converted data if needed - std::vector> d128_buffers; - - // Convert decimal32/decimal64 data to decimal128 if writing arrow schema - // and initialize LinkedColVector - auto vec = table_to_linked_columns( - (write_arrow_schema) - ? table_view({convert_decimal_columns_and_metadata(table_meta, d128_buffers, input, stream)}) - : input); + // initialize LinkedColVector + auto vec = table_to_linked_columns(input); auto schema_tree = construct_parquet_schema_tree( vec, table_meta, write_mode, int96_timestamps, utc_timestamps, write_arrow_schema); diff --git a/cpp/tests/io/parquet_writer_test.cpp b/cpp/tests/io/parquet_writer_test.cpp index 6c5e9cdf07a..e201dc0565c 100644 --- a/cpp/tests/io/parquet_writer_test.cpp +++ b/cpp/tests/io/parquet_writer_test.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2023-2024, NVIDIA CORPORATION. + * Copyright (c) 2023-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -483,73 +483,42 @@ TEST_F(ParquetWriterTest, DecimalWrite) auto table = table_view({col0, col1}); - auto filepath = temp_env->get_temp_filepath("DecimalWrite.parquet"); - cudf::io::parquet_writer_options args = - cudf::io::parquet_writer_options::builder(cudf::io::sink_info{filepath}, table); - - cudf::io::table_input_metadata expected_metadata(table); - - // verify failure if too small a precision is given - expected_metadata.column_metadata[0].set_decimal_precision(7); - expected_metadata.column_metadata[1].set_decimal_precision(1); - args.set_metadata(expected_metadata); - EXPECT_THROW(cudf::io::write_parquet(args), cudf::logic_error); - - // verify success if equal precision is given - expected_metadata.column_metadata[0].set_decimal_precision(7); - expected_metadata.column_metadata[1].set_decimal_precision(9); - args.set_metadata(std::move(expected_metadata)); - cudf::io::write_parquet(args); - - cudf::io::parquet_reader_options read_opts = - cudf::io::parquet_reader_options::builder(cudf::io::source_info{filepath}); - auto result = cudf::io::read_parquet(read_opts); - - CUDF_TEST_EXPECT_TABLES_EQUAL(*result.tbl, table); -} - -TEST_F(ParquetWriterTest, DecimalWriteWithArrowSchema) -{ - constexpr cudf::size_type num_rows = 500; - auto seq_col0 = random_values(num_rows); - auto seq_col1 = random_values(num_rows); + auto const test_decimal_write = [&](bool is_write_arrow_schema) { + auto filepath = temp_env->get_temp_filepath("DecimalWrite.parquet"); + cudf::io::parquet_writer_options args = + cudf::io::parquet_writer_options::builder(cudf::io::sink_info{filepath}, table) + .write_arrow_schema(is_write_arrow_schema); - auto valids = - cudf::detail::make_counting_transform_iterator(0, [](auto i) { return i % 2 == 0; }); + cudf::io::table_input_metadata expected_metadata(table); - auto col0 = cudf::test::fixed_point_column_wrapper{ - seq_col0.begin(), seq_col0.end(), valids, numeric::scale_type{5}}; - auto col1 = cudf::test::fixed_point_column_wrapper{ - seq_col1.begin(), seq_col1.end(), valids, numeric::scale_type{-9}}; + // verify failure if too small a precision is given + expected_metadata.column_metadata[0].set_decimal_precision(7); + expected_metadata.column_metadata[1].set_decimal_precision(1); + args.set_metadata(expected_metadata); + EXPECT_THROW(cudf::io::write_parquet(args), cudf::logic_error); - auto table = table_view({col0, col1}); + // verify success if equal precision is given + expected_metadata.column_metadata[0].set_decimal_precision(7); + expected_metadata.column_metadata[1].set_decimal_precision(9); - auto filepath = temp_env->get_temp_filepath("DecimalWriteWithArrowSchema.parquet"); - cudf::io::parquet_writer_options args = - cudf::io::parquet_writer_options::builder(cudf::io::sink_info{filepath}, table) - .write_arrow_schema(true); - - cudf::io::table_input_metadata expected_metadata(table); - // verify success if equal precision is given - expected_metadata.column_metadata[0].set_decimal_precision( - cudf::io::parquet::detail::MAX_DECIMAL32_PRECISION); - expected_metadata.column_metadata[1].set_decimal_precision( - cudf::io::parquet::detail::MAX_DECIMAL64_PRECISION); - args.set_metadata(std::move(expected_metadata)); - cudf::io::write_parquet(args); + // just plain encoding + expected_metadata.column_metadata[0].set_encoding(cudf::io::column_encoding::PLAIN); + expected_metadata.column_metadata[1].set_encoding(cudf::io::column_encoding::PLAIN); - auto expected_col0 = cudf::test::fixed_point_column_wrapper<__int128_t>{ - seq_col0.begin(), seq_col0.end(), valids, numeric::scale_type{5}}; - auto expected_col1 = cudf::test::fixed_point_column_wrapper<__int128_t>{ - seq_col1.begin(), seq_col1.end(), valids, numeric::scale_type{-9}}; + args.set_metadata(std::move(expected_metadata)); + cudf::io::write_parquet(args); - auto expected_table = table_view({expected_col0, expected_col1}); + cudf::io::parquet_reader_options read_opts = + cudf::io::parquet_reader_options::builder(cudf::io::source_info{filepath}) + .use_arrow_schema(true); + auto result = cudf::io::read_parquet(read_opts); - cudf::io::parquet_reader_options read_opts = - cudf::io::parquet_reader_options::builder(cudf::io::source_info{filepath}); - auto result = cudf::io::read_parquet(read_opts); + CUDF_TEST_EXPECT_TABLES_EQUAL(*result.tbl, table); + }; - CUDF_TEST_EXPECT_TABLES_EQUAL(*result.tbl, expected_table); + // Test decimal write with and without arrow schema + test_decimal_write(true); + test_decimal_write(false); } TEST_F(ParquetWriterTest, RowGroupSizeInvalid) From 359d9360e129c826d3cc0d670a0983778e077687 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Wed, 12 Feb 2025 19:17:20 -0500 Subject: [PATCH 024/129] Continue on failures in cudf.pandas integration tests CI job (#17987) Make the third-party integration tests CI job optional for merging. This PR depends on https://github.com/rapidsai/shared-workflows/pull/280 Authors: - Matthew Murray (https://github.com/Matt711) - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/17987 --- .github/workflows/pr.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 07c00b3e13c..e7a37a477b7 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -326,13 +326,14 @@ jobs: third-party-integration-tests-cudf-pandas: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@nvks-runners + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 with: build_type: pull-request branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} node_type: "gpu-l4-latest-1" + continue-on-error: true container_image: "rapidsai/ci-conda:latest" run_script: | ci/cudf_pandas_scripts/third-party-integration/test.sh python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml From ee74e2de06e5a23579f165d8bc59bdbfb483147c Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Wed, 12 Feb 2025 21:20:49 -0500 Subject: [PATCH 025/129] Check null count too in sum aggregation (#17964) Closes #17963. We should also check the null count when doing a sum aggregation to match polars. Also adds `size` and `null_count` directly to cudf.polars Column class. Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/17964 --- .../cudf_polars/containers/column.py | 22 +++++++++++++------ .../dsl/expressions/aggregation.py | 6 ++--- .../cudf_polars/dsl/expressions/binaryop.py | 4 ++-- .../cudf_polars/dsl/expressions/boolean.py | 8 +++---- .../cudf_polars/dsl/expressions/selection.py | 4 ++-- .../cudf_polars/dsl/expressions/string.py | 8 +++---- .../cudf_polars/dsl/expressions/unary.py | 2 +- python/cudf_polars/cudf_polars/dsl/ir.py | 6 ++--- .../tests/containers/test_column.py | 4 ++-- .../cudf_polars/tests/expressions/test_agg.py | 5 +++-- .../cudf_polars/tests/utils/test_broadcast.py | 6 ++--- 11 files changed, 42 insertions(+), 33 deletions(-) diff --git a/python/cudf_polars/cudf_polars/containers/column.py b/python/cudf_polars/cudf_polars/containers/column.py index 93d95346a37..2c83e05fe9c 100644 --- a/python/cudf_polars/cudf_polars/containers/column.py +++ b/python/cudf_polars/cudf_polars/containers/column.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 """A column, with some properties.""" @@ -51,7 +51,7 @@ def __init__( name: str | None = None, ): self.obj = column - self.is_scalar = self.obj.size() == 1 + self.is_scalar = self.size == 1 self.name = name self.set_sorted(is_sorted=is_sorted, order=order, null_order=null_order) @@ -70,9 +70,7 @@ def obj_scalar(self) -> plc.Scalar: If the column is not length-1. """ if not self.is_scalar: - raise ValueError( - f"Cannot convert a column of length {self.obj.size()} to scalar" - ) + raise ValueError(f"Cannot convert a column of length {self.size} to scalar") return plc.copying.get_element(self.obj, 0) def rename(self, name: str | None, /) -> Self: @@ -242,7 +240,7 @@ def set_sorted( ------- Self with metadata set. """ - if self.obj.size() <= 1: + if self.size <= 1: is_sorted = plc.types.Sorted.YES self.is_sorted = is_sorted self.order = order @@ -268,7 +266,7 @@ def copy(self) -> Self: def mask_nans(self) -> Self: """Return a shallow copy of self with nans masked out.""" if plc.traits.is_floating_point(self.obj.type()): - old_count = self.obj.null_count() + old_count = self.null_count mask, new_count = plc.transform.nans_to_nulls(self.obj) result = type(self)(self.obj.with_mask(mask, new_count)) if old_count == new_count: @@ -288,3 +286,13 @@ def nan_count(self) -> int: ) ).as_py() return 0 + + @property + def size(self) -> int: + """Return the size of the column.""" + return self.obj.size() + + @property + def null_count(self) -> int: + """Return the number of Null values in the column.""" + return self.obj.null_count() diff --git a/python/cudf_polars/cudf_polars/dsl/expressions/aggregation.py b/python/cudf_polars/cudf_polars/dsl/expressions/aggregation.py index 92f39abe71e..054220d7738 100644 --- a/python/cudf_polars/cudf_polars/dsl/expressions/aggregation.py +++ b/python/cudf_polars/cudf_polars/dsl/expressions/aggregation.py @@ -172,7 +172,7 @@ def _count(self, column: Column) -> Column: plc.Column.from_scalar( plc.interop.from_arrow( pa.scalar( - column.obj.size() - column.obj.null_count(), + column.size - column.null_count, type=plc.interop.to_arrow(self.dtype), ), ), @@ -181,7 +181,7 @@ def _count(self, column: Column) -> Column: ) def _sum(self, column: Column) -> Column: - if column.obj.size() == 0: + if column.size == 0 or column.null_count == column.size: return Column( plc.Column.from_scalar( plc.interop.from_arrow( @@ -224,7 +224,7 @@ def _first(self, column: Column) -> Column: return Column(plc.copying.slice(column.obj, [0, 1])[0]) def _last(self, column: Column) -> Column: - n = column.obj.size() + n = column.size return Column(plc.copying.slice(column.obj, [n - 1, n])[0]) def do_evaluate( diff --git a/python/cudf_polars/cudf_polars/dsl/expressions/binaryop.py b/python/cudf_polars/cudf_polars/dsl/expressions/binaryop.py index 556847b4738..84fd179aedd 100644 --- a/python/cudf_polars/cudf_polars/dsl/expressions/binaryop.py +++ b/python/cudf_polars/cudf_polars/dsl/expressions/binaryop.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 # TODO: remove need for this # ruff: noqa: D101 @@ -98,7 +98,7 @@ def do_evaluate( ) lop = left.obj rop = right.obj - if left.obj.size() != right.obj.size(): + if left.size != right.size: if left.is_scalar: lop = left.obj_scalar elif right.is_scalar: diff --git a/python/cudf_polars/cudf_polars/dsl/expressions/boolean.py b/python/cudf_polars/cudf_polars/dsl/expressions/boolean.py index d5ca22dd8d5..f1786f6ddd8 100644 --- a/python/cudf_polars/cudf_polars/dsl/expressions/boolean.py +++ b/python/cudf_polars/cudf_polars/dsl/expressions/boolean.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 # TODO: remove need for this # ruff: noqa: D101 @@ -191,7 +191,7 @@ def do_evaluate( is_any = self.name is BooleanFunction.Name.Any agg = plc.aggregation.any() if is_any else plc.aggregation.all() result = plc.reduce.reduce(column.obj, agg, self.dtype) - if not ignore_nulls and column.obj.null_count() > 0: + if not ignore_nulls and column.null_count > 0: # Truth tables # Any All # | F U T | F U T @@ -218,14 +218,14 @@ def do_evaluate( (column,) = columns return Column( plc.unary.is_nan(column.obj).with_mask( - column.obj.null_mask(), column.obj.null_count() + column.obj.null_mask(), column.null_count ) ) elif self.name is BooleanFunction.Name.IsNotNan: (column,) = columns return Column( plc.unary.is_not_nan(column.obj).with_mask( - column.obj.null_mask(), column.obj.null_count() + column.obj.null_mask(), column.null_count ) ) elif self.name is BooleanFunction.Name.IsFirstDistinct: diff --git a/python/cudf_polars/cudf_polars/dsl/expressions/selection.py b/python/cudf_polars/cudf_polars/dsl/expressions/selection.py index 93ecd026eaf..e889bfd2e33 100644 --- a/python/cudf_polars/cudf_polars/dsl/expressions/selection.py +++ b/python/cudf_polars/cudf_polars/dsl/expressions/selection.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 # TODO: remove need for this # ruff: noqa: D101 @@ -50,7 +50,7 @@ def do_evaluate( n = df.num_rows if hi >= n or lo < -n: raise ValueError("gather indices are out of bounds") - if indices.obj.null_count(): + if indices.null_count: bounds_policy = plc.copying.OutOfBoundsPolicy.NULLIFY obj = plc.replace.replace_nulls( indices.obj, diff --git a/python/cudf_polars/cudf_polars/dsl/expressions/string.py b/python/cudf_polars/cudf_polars/dsl/expressions/string.py index aa32dc66bd9..76dd0385635 100644 --- a/python/cudf_polars/cudf_polars/dsl/expressions/string.py +++ b/python/cudf_polars/cudf_polars/dsl/expressions/string.py @@ -210,7 +210,7 @@ def do_evaluate( (child,) = self.children column = child.evaluate(df, context=context, mapping=mapping) delimiter, ignore_nulls = self.options - if column.obj.null_count() > 0 and not ignore_nulls: + if column.null_count > 0 and not ignore_nulls: return Column(plc.Column.all_null_like(column.obj, 1)) return Column( plc.strings.combine.join_strings( @@ -228,7 +228,7 @@ def do_evaluate( pat = arg.evaluate(df, context=context, mapping=mapping) pattern = ( pat.obj_scalar - if pat.is_scalar and pat.obj.size() != column.obj.size() + if pat.is_scalar and pat.size != column.size else pat.obj ) return Column(plc.strings.find.contains(column.obj, pattern)) @@ -298,7 +298,7 @@ def do_evaluate( plc.strings.find.ends_with( column.obj, suffix.obj_scalar - if column.obj.size() != suffix.obj.size() and suffix.is_scalar + if column.size != suffix.size and suffix.is_scalar else suffix.obj, ) ) @@ -308,7 +308,7 @@ def do_evaluate( plc.strings.find.starts_with( column.obj, prefix.obj_scalar - if column.obj.size() != prefix.obj.size() and prefix.is_scalar + if column.size != prefix.size and prefix.is_scalar else prefix.obj, ) ) diff --git a/python/cudf_polars/cudf_polars/dsl/expressions/unary.py b/python/cudf_polars/cudf_polars/dsl/expressions/unary.py index 3286c9ff8bc..76856aac63f 100644 --- a/python/cudf_polars/cudf_polars/dsl/expressions/unary.py +++ b/python/cudf_polars/cudf_polars/dsl/expressions/unary.py @@ -236,7 +236,7 @@ def do_evaluate( else plc.types.Order.DESCENDING ) null_order = plc.types.NullOrder.BEFORE - if column.obj.null_count() > 0 and (n := column.obj.size()) > 1: + if column.null_count > 0 and (n := column.size) > 1: # PERF: This invokes four stream synchronisations! has_nulls_first = not plc.copying.get_element(column.obj, 0).is_valid() has_nulls_last = not plc.copying.get_element( diff --git a/python/cudf_polars/cudf_polars/dsl/ir.py b/python/cudf_polars/cudf_polars/dsl/ir.py index 8ec14fb2b7f..78bf10fdac7 100644 --- a/python/cudf_polars/cudf_polars/dsl/ir.py +++ b/python/cudf_polars/cudf_polars/dsl/ir.py @@ -100,7 +100,7 @@ def broadcast(*columns: Column, target_length: int | None = None) -> list[Column """ if len(columns) == 0: return [] - lengths: set[int] = {column.obj.size() for column in columns} + lengths: set[int] = {column.size for column in columns} if lengths == {1}: if target_length is None: return list(columns) @@ -116,7 +116,7 @@ def broadcast(*columns: Column, target_length: int | None = None) -> list[Column ) return [ column - if column.obj.size() != 1 + if column.size != 1 else Column( plc.Column.from_scalar(column.obj_scalar, nrows), is_sorted=plc.types.Sorted.YES, @@ -820,7 +820,7 @@ def do_evaluate( ) -> DataFrame: # pragma: no cover; not exposed by polars yet """Evaluate and return a dataframe.""" columns = broadcast(*(e.evaluate(df) for e in exprs)) - assert all(column.obj.size() == 1 for column in columns) + assert all(column.size == 1 for column in columns) return DataFrame(columns) diff --git a/python/cudf_polars/tests/containers/test_column.py b/python/cudf_polars/tests/containers/test_column.py index 95541b4ecc3..86188de2abd 100644 --- a/python/cudf_polars/tests/containers/test_column.py +++ b/python/cudf_polars/tests/containers/test_column.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 from __future__ import annotations @@ -59,7 +59,7 @@ def test_mask_nans(typeid): values = pyarrow.array([0, 0, 0], type=plc.interop.to_arrow(dtype)) column = Column(plc.interop.from_arrow(values)) masked = column.mask_nans() - assert column.obj.null_count() == masked.obj.null_count() + assert column.null_count == masked.null_count def test_mask_nans_float(): diff --git a/python/cudf_polars/tests/expressions/test_agg.py b/python/cudf_polars/tests/expressions/test_agg.py index 15ad845ea78..482e168a851 100644 --- a/python/cudf_polars/tests/expressions/test_agg.py +++ b/python/cudf_polars/tests/expressions/test_agg.py @@ -150,7 +150,8 @@ def test_agg_singleton(op): assert_gpu_result_equal(q) -def test_sum_empty_zero(): - df = pl.LazyFrame({"a": pl.Series(values=[], dtype=pl.Int32())}) +@pytest.mark.parametrize("data", [[], [None], [None, 2, 3, None]]) +def test_sum_empty_zero(data): + df = pl.LazyFrame({"a": pl.Series(values=data, dtype=pl.Int32())}) q = df.select(pl.col("a").sum()) assert_gpu_result_equal(q) diff --git a/python/cudf_polars/tests/utils/test_broadcast.py b/python/cudf_polars/tests/utils/test_broadcast.py index 3b3b4f0f8db..281aa8ceee5 100644 --- a/python/cudf_polars/tests/utils/test_broadcast.py +++ b/python/cudf_polars/tests/utils/test_broadcast.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 from __future__ import annotations @@ -26,7 +26,7 @@ def test_broadcast_all_scalar(target): expected = 1 if target is None else target assert [c.name for c in result] == [f"col{i}" for i in range(3)] - assert all(column.obj.size() == expected for column in result) + assert all(column.size == expected for column in result) def test_invalid_target_length(): @@ -73,4 +73,4 @@ def test_broadcast_with_scalars(nrows): result = broadcast(*columns) assert [c.name for c in result] == [f"col{i}" for i in range(3)] - assert all(column.obj.size() == nrows for column in result) + assert all(column.size == nrows for column in result) From d6bfe3b6564bb1ef7940873c1bf93561853ea267 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Wed, 12 Feb 2025 18:49:09 -0800 Subject: [PATCH 026/129] Add `pylibcudf.Scalar.from_py` for construction from Python strings, bool, int, float (#17898) Towards https://github.com/rapidsai/cudf/issues/17054 Authors: - Matthew Roeschke (https://github.com/mroeschke) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Matthew Murray (https://github.com/Matt711) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17898 --- .../libcudf/scalar/scalar_factories.pxd | 7 +- python/pylibcudf/pylibcudf/scalar.pyx | 93 ++++++++++++++++++- .../pylibcudf/pylibcudf/tests/test_scalar.py | 30 ++++++ 3 files changed, 125 insertions(+), 5 deletions(-) create mode 100644 python/pylibcudf/pylibcudf/tests/test_scalar.py diff --git a/python/pylibcudf/pylibcudf/libcudf/scalar/scalar_factories.pxd b/python/pylibcudf/pylibcudf/libcudf/scalar/scalar_factories.pxd index 9fb907970de..6ba72f2b3d7 100644 --- a/python/pylibcudf/pylibcudf/libcudf/scalar/scalar_factories.pxd +++ b/python/pylibcudf/pylibcudf/libcudf/scalar/scalar_factories.pxd @@ -1,7 +1,8 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from libcpp.memory cimport unique_ptr from libcpp.string cimport string from pylibcudf.exception_handler cimport libcudf_exception_handler +from pylibcudf.libcudf.types cimport data_type from pylibcudf.libcudf.column.column_view cimport column_view from pylibcudf.libcudf.scalar.scalar cimport scalar @@ -13,7 +14,9 @@ cdef extern from "cudf/scalar/scalar_factories.hpp" namespace "cudf" nogil: cdef unique_ptr[scalar] make_fixed_width_scalar[T]( T value ) except +libcudf_exception_handler - + cdef unique_ptr[scalar] make_numeric_scalar( + data_type type_ + ) except +libcudf_exception_handler cdef unique_ptr[scalar] make_empty_scalar_like( const column_view & ) except +libcudf_exception_handler diff --git a/python/pylibcudf/pylibcudf/scalar.pyx b/python/pylibcudf/pylibcudf/scalar.pyx index 1ac014e891e..35abab7e838 100644 --- a/python/pylibcudf/pylibcudf/scalar.pyx +++ b/python/pylibcudf/pylibcudf/scalar.pyx @@ -1,16 +1,30 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. +from cpython cimport bool as py_bool, datetime from cython cimport no_gc_clear +from libc.stdint cimport int64_t +from libcpp cimport bool as cbool from libcpp.memory cimport unique_ptr from libcpp.utility cimport move -from pylibcudf.libcudf.scalar.scalar cimport scalar -from pylibcudf.libcudf.scalar.scalar_factories cimport make_empty_scalar_like +from pylibcudf.libcudf.scalar.scalar cimport ( + scalar, + numeric_scalar, +) +from pylibcudf.libcudf.scalar.scalar_factories cimport ( + make_empty_scalar_like, + make_string_scalar, + make_numeric_scalar, +) +from pylibcudf.libcudf.types cimport type_id + from rmm.pylibrmm.memory_resource cimport get_current_device_resource from .column cimport Column from .types cimport DataType +from functools import singledispatch + __all__ = ["Scalar"] @@ -79,3 +93,76 @@ cdef class Scalar: s.c_obj.swap(libcudf_scalar) s._data_type = DataType.from_libcudf(s.get().type()) return s + + @classmethod + def from_py(cls, py_val): + """ + Convert a Python standard library object to a Scalar. + + Parameters + ---------- + py_val: bool, int, float, str, datetime.datetime, datetime.timedelta, list, dict + Value to convert to a pylibcudf.Scalar + + Returns + ------- + Scalar + New pylibcudf.Scalar + """ + return _from_py(py_val) + +cdef Scalar _new_scalar(unique_ptr[scalar] c_obj, DataType dtype): + cdef Scalar s = Scalar.__new__(Scalar) + s.c_obj.swap(c_obj) + s._data_type = dtype + return s + + +@singledispatch +def _from_py(py_val): + raise TypeError(f"{type(py_val).__name__} cannot be converted to pylibcudf.Scalar") + + +@_from_py.register(dict) +@_from_py.register(list) +@_from_py.register(datetime.datetime) +@_from_py.register(datetime.timedelta) +def _(py_val): + raise NotImplementedError( + f"Conversion from {type(py_val).__name__} is currently not supported." + ) + + +@_from_py.register(float) +def _(py_val): + cdef DataType dtype = DataType(type_id.FLOAT64) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(py_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + +@_from_py.register(int) +def _(py_val): + cdef DataType dtype = DataType(type_id.INT64) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(py_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + +@_from_py.register(py_bool) +def _(py_val): + cdef DataType dtype = DataType(type_id.BOOL8) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(py_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + +@_from_py.register(str) +def _(py_val): + cdef DataType dtype = DataType(type_id.STRING) + cdef unique_ptr[scalar] c_obj = make_string_scalar(py_val.encode()) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr diff --git a/python/pylibcudf/pylibcudf/tests/test_scalar.py b/python/pylibcudf/pylibcudf/tests/test_scalar.py new file mode 100644 index 00000000000..45afae91c9a --- /dev/null +++ b/python/pylibcudf/pylibcudf/tests/test_scalar.py @@ -0,0 +1,30 @@ +# Copyright (c) 2024-2025, NVIDIA CORPORATION. +import datetime + +import pyarrow as pa +import pytest + +import pylibcudf as plc + + +@pytest.mark.parametrize( + "val", [True, False, -1, 0, 1 - 1.0, 0.0, 1.52, "", "a1!"] +) +def test_from_py(val): + result = plc.Scalar.from_py(val) + expected = pa.scalar(val) + assert plc.interop.to_arrow(result).equals(expected) + + +@pytest.mark.parametrize( + "val", [datetime.datetime(2020, 1, 1), datetime.timedelta(1), [1], {1: 1}] +) +def test_from_py_notimplemented(val): + with pytest.raises(NotImplementedError): + plc.Scalar.from_py(val) + + +@pytest.mark.parametrize("val", [object, None]) +def test_from_py_typeerror(val): + with pytest.raises(TypeError): + plc.Scalar.from_py(val) From e6b1c0f7ea121f6d40567331bf13c3198f96fca3 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Wed, 12 Feb 2025 22:08:31 -0500 Subject: [PATCH 027/129] Add catboost integration tests (#17931) Apart of #17490. This PR adds back the catboost integration tests, which were originally added in https://github.com/rapidsai/cudf/pull/17267 but were later removed due to ABI incompatability between the version of numpy catboost is compiled against and the version of numpy installed in the test environment. This PR adds the tests back but pins a compatible numpy version in the catboost tests. Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - James Lamb (https://github.com/jameslamb) - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/17931 --- .../dependencies.yaml | 16 +++ .../tests/test_catboost.py | 128 ++++++++++++++++++ 2 files changed, 144 insertions(+) create mode 100644 python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_catboost.py diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml index 53a8001c750..059a4ff3c98 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml @@ -76,6 +76,13 @@ files: - py_version - test_base - test_xgboost + test_catboost: + output: none + includes: + - cuda_version + - py_version + - test_base + - test_catboost test_cuml: output: none includes: @@ -243,6 +250,15 @@ dependencies: - scikit-learn - pip - xgboost>=2.0.1 + test_catboost: + common: + - output_types: conda + packages: + # TODO: Remove numpy pinning once https://github.com/catboost/catboost/issues/2671 is resolved + - numpy>=1.23,<2.0.0 + - scipy + - scikit-learn + - catboost test_cuml: common: - output_types: conda diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_catboost.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_catboost.py new file mode 100644 index 00000000000..94b6094beae --- /dev/null +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_catboost.py @@ -0,0 +1,128 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. +import numpy as np +import pandas as pd +import pytest +from catboost import CatBoostClassifier, CatBoostRegressor, Pool +from sklearn.datasets import make_classification, make_regression + +rng = np.random.default_rng(seed=42) + + +def assert_catboost_equal(expect, got, rtol=1e-7, atol=0.0): + if isinstance(expect, (tuple, list)): + assert len(expect) == len(got) + for e, g in zip(expect, got): + assert_catboost_equal(e, g, rtol, atol) + elif isinstance(expect, np.ndarray): + np.testing.assert_allclose(expect, got, rtol=rtol, atol=atol) + elif isinstance(expect, pd.DataFrame): + pd.testing.assert_frame_equal(expect, got) + elif isinstance(expect, pd.Series): + pd.testing.assert_series_equal(expect, got) + else: + assert expect == got + + +pytestmark = pytest.mark.assert_eq(fn=assert_catboost_equal) + + +@pytest.fixture +def regression_data(): + X, y = make_regression(n_samples=100, n_features=10, random_state=42) + return pd.DataFrame(X), pd.Series(y) + + +@pytest.fixture +def classification_data(): + X, y = make_classification( + n_samples=100, n_features=10, n_classes=2, random_state=42 + ) + return pd.DataFrame(X), pd.Series(y) + + +def test_catboost_regressor_with_dataframe(regression_data): + X, y = regression_data + model = CatBoostRegressor(iterations=10, verbose=0) + model.fit(X, y) + predictions = model.predict(X) + return predictions + + +def test_catboost_regressor_with_numpy(regression_data): + X, y = regression_data + model = CatBoostRegressor(iterations=10, verbose=0) + model.fit(X.values, y.values) + predictions = model.predict(X.values) + return predictions + + +def test_catboost_classifier_with_dataframe(classification_data): + X, y = classification_data + model = CatBoostClassifier(iterations=10, verbose=0) + model.fit(X, y) + predictions = model.predict(X) + return predictions + + +def test_catboost_classifier_with_numpy(classification_data): + X, y = classification_data + model = CatBoostClassifier(iterations=10, verbose=0) + model.fit(X.values, y.values) + predictions = model.predict(X.values) + return predictions + + +def test_catboost_with_pool_and_dataframe(regression_data): + X, y = regression_data + train_pool = Pool(X, y) + model = CatBoostRegressor(iterations=10, verbose=0) + model.fit(train_pool) + predictions = model.predict(X) + return predictions + + +def test_catboost_with_pool_and_numpy(regression_data): + X, y = regression_data + train_pool = Pool(X.values, y.values) + model = CatBoostRegressor(iterations=10, verbose=0) + model.fit(train_pool) + predictions = model.predict(X.values) + return predictions + + +def test_catboost_with_categorical_features(): + data = { + "numerical_feature": rng.standard_normal(100), + "categorical_feature": rng.choice(["A", "B", "C"], size=100), + "target": rng.integers(0, 2, size=100), + } + df = pd.DataFrame(data) + X = df[["numerical_feature", "categorical_feature"]] + y = df["target"] + cat_features = ["categorical_feature"] + model = CatBoostClassifier( + iterations=10, verbose=0, cat_features=cat_features + ) + model.fit(X, y) + predictions = model.predict(X) + return predictions + + +@pytest.mark.parametrize( + "X, y", + [ + ( + pd.DataFrame(rng.standard_normal((100, 5))), + pd.Series(rng.standard_normal(100)), + ), + (rng.standard_normal((100, 5)), rng.standard_normal(100)), + ], +) +def test_catboost_train_test_split(X, y): + from sklearn.model_selection import train_test_split + + X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42) + model = CatBoostRegressor(iterations=10, verbose=0) + model.fit(X_train, y_train) + predictions = model.predict(X_test) + return len(X_train), len(X_test), len(y_train), len(y_test), predictions From 725f9eb1aeea4c8b68bdf35914bf64f7ab0d879a Mon Sep 17 00:00:00 2001 From: Tianyu Liu Date: Wed, 12 Feb 2025 23:40:03 -0500 Subject: [PATCH 028/129] Use KvikIO to enable file's fast host read and host write (#17764) This PR makes the following improvements on I/O: - Remove legacy cuFile integration to simplify code maintenance. Use KvikIO to manage the GDS setting and compatibility mode. - Remove file utility classes and functions. Use KvikIO for all file-related operations. - Replace in-house implementation of `host_read` (for `file_source`) and `host_write` (for `file_sink`) with KvikIO's parallel counterpart. - Update the documentation on compatibility mode/GDS. Closes https://github.com/rapidsai/cudf/issues/16418 Issue https://github.com/rapidsai/cudf/issues/17228 Authors: - Tianyu Liu (https://github.com/kingcrimsontianyu) - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Shruti Shivakumar (https://github.com/shrshi) - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17764 --- cpp/CMakeLists.txt | 13 +- cpp/include/cudf/io/config_utils.hpp | 21 +- cpp/include/cudf/io/data_sink.hpp | 4 +- cpp/src/io/utilities/config_utils.cpp | 36 +- cpp/src/io/utilities/data_sink.cpp | 62 ++-- cpp/src/io/utilities/datasource.cpp | 71 ++-- cpp/src/io/utilities/file_io_utilities.cpp | 389 --------------------- cpp/src/io/utilities/file_io_utilities.hpp | 229 ------------ cpp/tests/CMakeLists.txt | 5 - cpp/tests/io/file_io_test.cpp | 43 --- docs/cudf/source/user_guide/io/io.md | 76 ++-- 11 files changed, 82 insertions(+), 867 deletions(-) delete mode 100644 cpp/src/io/utilities/file_io_utilities.cpp delete mode 100644 cpp/src/io/utilities/file_io_utilities.hpp delete mode 100644 cpp/tests/io/file_io_test.cpp diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index a282c12d97f..2e4dd21667e 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -79,7 +79,6 @@ option(CUDA_ENABLE_LINEINFO option(CUDA_WARNINGS_AS_ERRORS "Enable -Werror=all-warnings for all CUDA compilation" ON) # cudart can be statically linked or dynamically linked. The python ecosystem wants dynamic linking option(CUDA_STATIC_RUNTIME "Statically link the CUDA runtime" OFF) -option(CUDA_STATIC_CUFILE "Statically link cuFile" OFF) set(DEFAULT_CUDF_BUILD_STREAMS_TEST_UTIL ON) if(CUDA_STATIC_RUNTIME OR NOT BUILD_SHARED_LIBS) @@ -546,7 +545,6 @@ add_library( src/io/utilities/data_casting.cu src/io/utilities/data_sink.cpp src/io/utilities/datasource.cpp - src/io/utilities/file_io_utilities.cpp src/io/utilities/row_selection.cpp src/io/utilities/type_inference.cu src/io/utilities/trie.cu @@ -922,15 +920,6 @@ target_compile_definitions( # Enable remote IO through KvikIO target_compile_definitions(cudf PRIVATE $<$:CUDF_KVIKIO_REMOTE_IO>) -# Enable cuFile support -set(_cufile_suffix) -if(CUDA_STATIC_CUFILE) - set(_cufile_suffix _static) -endif() -if(TARGET CUDA::cuFile${_cufile_suffix}) - target_compile_definitions(cudf PRIVATE CUDF_CUFILE_FOUND) -endif() - # Remove this after upgrading to a CCCL that has a proper CMake option. See # https://github.com/NVIDIA/cccl/pull/2844 target_compile_definitions(cudf PRIVATE THRUST_FORCE_32_BIT_OFFSET_TYPE=1) @@ -943,7 +932,7 @@ target_link_libraries( cudf PUBLIC CCCL::CCCL rapids_logger::rapids_logger rmm::rmm $ PRIVATE $ cuco::cuco ZLIB::ZLIB nvcomp::nvcomp - kvikio::kvikio $ nanoarrow + kvikio::kvikio nanoarrow ) # Add Conda library, and include paths if specified diff --git a/cpp/include/cudf/io/config_utils.hpp b/cpp/include/cudf/io/config_utils.hpp index 070b59a117c..13953280c80 100644 --- a/cpp/include/cudf/io/config_utils.hpp +++ b/cpp/include/cudf/io/config_utils.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -19,22 +19,7 @@ namespace CUDF_EXPORT cudf { namespace io { -namespace cufile_integration { - -/** - * @brief Returns true if cuFile and its compatibility mode are enabled. - */ -bool is_always_enabled(); - -/** - * @brief Returns true if only direct IO through cuFile is enabled (compatibility mode is disabled). - */ -bool is_gds_enabled(); - -/** - * @brief Returns true if KvikIO is enabled. - */ -bool is_kvikio_enabled(); +namespace kvikio_integration { /** * @brief Set KvikIO parameters, including: @@ -45,7 +30,7 @@ bool is_kvikio_enabled(); */ void set_up_kvikio(); -} // namespace cufile_integration +} // namespace kvikio_integration namespace nvcomp_integration { diff --git a/cpp/include/cudf/io/data_sink.hpp b/cpp/include/cudf/io/data_sink.hpp index e1eb9c042c7..fe10f46d6b1 100644 --- a/cpp/include/cudf/io/data_sink.hpp +++ b/cpp/include/cudf/io/data_sink.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -122,7 +122,7 @@ class data_sink { * * In the case where the sink type is itself a memory buffered write, this ends up * being effectively a second memcpy. So a useful optimization for a "smart" - * custom data_sink is to do it's own internal management of the movement + * custom data_sink is to do its own internal management of the movement * of data between cpu and gpu; turning the internals of the writer into simply * * sink->device_write(device_buffer, size) diff --git a/cpp/src/io/utilities/config_utils.cpp b/cpp/src/io/utilities/config_utils.cpp index 726feca328b..46816604918 100644 --- a/cpp/src/io/utilities/config_utils.cpp +++ b/cpp/src/io/utilities/config_utils.cpp @@ -24,38 +24,17 @@ namespace cudf::io { -namespace cufile_integration { - -namespace { -/** - * @brief Defines which cuFile usage to enable. - */ -enum class usage_policy : uint8_t { OFF, GDS, ALWAYS, KVIKIO }; - -/** - * @brief Get the current usage policy. - */ -usage_policy get_env_policy() -{ - static auto const env_val = getenv_or("LIBCUDF_CUFILE_POLICY", "KVIKIO"); - if (env_val == "OFF") return usage_policy::OFF; - if (env_val == "GDS") return usage_policy::GDS; - if (env_val == "ALWAYS") return usage_policy::ALWAYS; - if (env_val == "KVIKIO") return usage_policy::KVIKIO; - CUDF_FAIL("Invalid LIBCUDF_CUFILE_POLICY value: " + env_val); -} -} // namespace - -bool is_always_enabled() { return get_env_policy() == usage_policy::ALWAYS; } - -bool is_gds_enabled() { return is_always_enabled() or get_env_policy() == usage_policy::GDS; } - -bool is_kvikio_enabled() { return get_env_policy() == usage_policy::KVIKIO; } +namespace kvikio_integration { void set_up_kvikio() { static std::once_flag flag{}; std::call_once(flag, [] { + // Workaround for https://github.com/rapidsai/cudf/issues/14140, where cuFileDriverOpen errors + // out if no CUDA calls have been made before it. This is a no-op if the CUDA context is already + // initialized. + cudaFree(nullptr); + auto const compat_mode = kvikio::getenv_or("KVIKIO_COMPAT_MODE", kvikio::CompatMode::ON); kvikio::defaults::compat_mode_reset(compat_mode); @@ -63,7 +42,8 @@ void set_up_kvikio() kvikio::defaults::thread_pool_nthreads_reset(nthreads); }); } -} // namespace cufile_integration + +} // namespace kvikio_integration namespace nvcomp_integration { diff --git a/cpp/src/io/utilities/data_sink.cpp b/cpp/src/io/utilities/data_sink.cpp index 975206646c6..e8a05f431bd 100644 --- a/cpp/src/io/utilities/data_sink.cpp +++ b/cpp/src/io/utilities/data_sink.cpp @@ -14,8 +14,6 @@ * limitations under the License. */ -#include "file_io_utilities.hpp" - #include #include #include @@ -25,8 +23,6 @@ #include -#include - namespace cudf { namespace io { @@ -37,18 +33,11 @@ class file_sink : public data_sink { public: explicit file_sink(std::string const& filepath) { - detail::force_init_cuda_context(); - _output_stream.open(filepath, std::ios::out | std::ios::binary | std::ios::trunc); - if (!_output_stream.is_open()) { detail::throw_on_file_open_failure(filepath, true); } - - if (cufile_integration::is_kvikio_enabled()) { - cufile_integration::set_up_kvikio(); - _kvikio_file = kvikio::FileHandle(filepath, "w"); - CUDF_LOG_INFO("Writing a file using kvikIO, with compatibility mode %s.", - _kvikio_file.is_compat_mode_preferred() ? "on" : "off"); - } else { - _cufile_out = detail::make_cufile_output(filepath); - } + kvikio_integration::set_up_kvikio(); + _kvikio_file = kvikio::FileHandle(filepath, "w"); + CUDF_EXPECTS(!_kvikio_file.closed(), "KvikIO did not open the file successfully."); + CUDF_LOG_INFO("Writing a file using kvikIO, with compatibility mode %s.", + _kvikio_file.is_compat_mode_preferred() ? "on" : "off"); } // Marked as NOLINT because we are calling a virtual method in the destructor @@ -56,28 +45,24 @@ class file_sink : public data_sink { void host_write(void const* data, size_t size) override { - _output_stream.seekp(_bytes_written); - _output_stream.write(static_cast(data), size); + _kvikio_file.pwrite(data, size, _bytes_written).get(); _bytes_written += size; } - void flush() override { _output_stream.flush(); } + void flush() override + { + // kvikio::FileHandle::pwrite() makes system calls that reach the kernel buffer cache. This + // process does not involve application buffer. Therefore calls to ::fflush() or + // ofstream::flush() do not apply. + } size_t bytes_written() override { return _bytes_written; } - [[nodiscard]] bool supports_device_write() const override - { - return !_kvikio_file.closed() || _cufile_out != nullptr; - } + [[nodiscard]] bool supports_device_write() const override { return true; } [[nodiscard]] bool is_device_write_preferred(size_t size) const override { - if (!supports_device_write()) { return false; } - - // Always prefer device writes if kvikio is enabled - if (!_kvikio_file.closed()) { return true; } - - return size >= _gds_write_preferred_threshold; + return supports_device_write(); } std::future device_write_async(void const* gpu_data, @@ -89,14 +74,11 @@ class file_sink : public data_sink { size_t const offset = _bytes_written; _bytes_written += size; - if (!_kvikio_file.closed()) { - // KvikIO's `pwrite()` returns a `std::future` so we convert it - // to `std::future` - return std::async(std::launch::deferred, [this, gpu_data, size, offset] { - _kvikio_file.pwrite(gpu_data, size, offset).get(); - }); - } - return _cufile_out->write_async(gpu_data, offset, size); + // KvikIO's `pwrite()` returns a `std::future` so we convert it + // to `std::future` + return std::async(std::launch::deferred, [this, gpu_data, size, offset]() -> void { + _kvikio_file.pwrite(gpu_data, size, offset).get(); + }); } void device_write(void const* gpu_data, size_t size, rmm::cuda_stream_view stream) override @@ -105,12 +87,8 @@ class file_sink : public data_sink { } private: - std::ofstream _output_stream; size_t _bytes_written = 0; - std::unique_ptr _cufile_out; kvikio::FileHandle _kvikio_file; - // The write size above which GDS is faster then d2h-copy + posix-write - static constexpr size_t _gds_write_preferred_threshold = 128 << 10; // 128KB }; /** @@ -162,7 +140,7 @@ class void_sink : public data_sink { rmm::cuda_stream_view stream) override { _bytes_written += size; - return std::async(std::launch::deferred, [] {}); + return std::async(std::launch::deferred, []() -> void {}); } void flush() override {} diff --git a/cpp/src/io/utilities/datasource.cpp b/cpp/src/io/utilities/datasource.cpp index 87b3c6facdf..14b6bc6f774 100644 --- a/cpp/src/io/utilities/datasource.cpp +++ b/cpp/src/io/utilities/datasource.cpp @@ -14,7 +14,6 @@ * limitations under the License. */ -#include "file_io_utilities.hpp" #include "getenv_or.hpp" #include @@ -49,58 +48,39 @@ namespace { */ class file_source : public datasource { public: - explicit file_source(char const* filepath) : _file(filepath, O_RDONLY) - { - detail::force_init_cuda_context(); - if (cufile_integration::is_kvikio_enabled()) { - cufile_integration::set_up_kvikio(); - _kvikio_file = kvikio::FileHandle(filepath); - CUDF_LOG_INFO("Reading a file using kvikIO, with compatibility mode %s.", - _kvikio_file.is_compat_mode_preferred() ? "on" : "off"); - } else { - _cufile_in = detail::make_cufile_input(filepath); - } + explicit file_source(char const* filepath) + { + kvikio_integration::set_up_kvikio(); + _kvikio_file = kvikio::FileHandle(filepath, "r"); + CUDF_EXPECTS(!_kvikio_file.closed(), "KvikIO did not open the file successfully."); + CUDF_LOG_INFO("Reading a file using kvikIO, with compatibility mode %s.", + _kvikio_file.is_compat_mode_preferred() ? "on" : "off"); } std::unique_ptr host_read(size_t offset, size_t size) override { - lseek(_file.desc(), offset, SEEK_SET); - // Clamp length to available data - ssize_t const read_size = std::min(size, _file.size() - offset); - + auto const read_size = std::min(size, this->size() - offset); std::vector v(read_size); - CUDF_EXPECTS(read(_file.desc(), v.data(), read_size) == read_size, "read failed"); + CUDF_EXPECTS(_kvikio_file.pread(v.data(), read_size, offset).get() == read_size, "read failed"); return buffer::create(std::move(v)); } size_t host_read(size_t offset, size_t size, uint8_t* dst) override { - lseek(_file.desc(), offset, SEEK_SET); - // Clamp length to available data - auto const read_size = std::min(size, _file.size() - offset); - - CUDF_EXPECTS(read(_file.desc(), dst, read_size) == static_cast(read_size), - "read failed"); + auto const read_size = std::min(size, this->size() - offset); + CUDF_EXPECTS(_kvikio_file.pread(dst, read_size, offset).get() == read_size, "read failed"); return read_size; } ~file_source() override = default; - [[nodiscard]] bool supports_device_read() const override - { - return !_kvikio_file.closed() || _cufile_in != nullptr; - } + [[nodiscard]] bool supports_device_read() const override { return true; } [[nodiscard]] bool is_device_read_preferred(size_t size) const override { - if (!supports_device_read()) { return false; } - - // Always prefer device reads if kvikio is enabled - if (!_kvikio_file.closed()) { return true; } - - return size >= _gds_read_preferred_threshold; + return supports_device_read(); } std::future device_read_async(size_t offset, @@ -110,9 +90,8 @@ class file_source : public datasource { { CUDF_EXPECTS(supports_device_read(), "Device reads are not supported for this file."); - auto const read_size = std::min(size, _file.size() - offset); - if (!_kvikio_file.closed()) { return _kvikio_file.pread(dst, read_size, offset); } - return _cufile_in->read_async(offset, read_size, dst, stream); + auto const read_size = std::min(size, this->size() - offset); + return _kvikio_file.pread(dst, read_size, offset); } size_t device_read(size_t offset, @@ -134,16 +113,10 @@ class file_source : public datasource { return datasource::buffer::create(std::move(out_data)); } - [[nodiscard]] size_t size() const override { return _file.size(); } + [[nodiscard]] size_t size() const override { return _kvikio_file.nbytes(); } protected: - detail::file_wrapper _file; - - private: - std::unique_ptr _cufile_in; kvikio::FileHandle _kvikio_file; - // The read size above which GDS is faster then posix-read + h2d-copy - static constexpr size_t _gds_read_preferred_threshold = 128 << 10; // 128KB }; /** @@ -157,9 +130,9 @@ class memory_mapped_source : public file_source { explicit memory_mapped_source(char const* filepath, size_t offset, size_t max_size_estimate) : file_source(filepath) { - if (_file.size() != 0) { + if (this->size() != 0) { // Memory mapping is not exclusive, so we can include the whole region we expect to read - map(_file.desc(), offset, max_size_estimate); + map(_kvikio_file.fd(), offset, max_size_estimate); } } @@ -171,7 +144,7 @@ class memory_mapped_source : public file_source { std::unique_ptr host_read(size_t offset, size_t size) override { // Clamp length to available data - auto const read_size = std::min(size, +_file.size() - offset); + auto const read_size = std::min(size, this->size() - offset); // If the requested range is outside of the mapped region, read from the file if (offset < _map_offset or offset + read_size > (_map_offset + _map_size)) { @@ -195,7 +168,7 @@ class memory_mapped_source : public file_source { size_t host_read(size_t offset, size_t size, uint8_t* dst) override { // Clamp length to available data - auto const read_size = std::min(size, +_file.size() - offset); + auto const read_size = std::min(size, this->size() - offset); // If the requested range is outside of the mapped region, read from the file if (offset < _map_offset or offset + read_size > (_map_offset + _map_size)) { @@ -210,12 +183,12 @@ class memory_mapped_source : public file_source { private: void map(int fd, size_t offset, size_t size) { - CUDF_EXPECTS(offset < _file.size(), "Offset is past end of file", std::overflow_error); + CUDF_EXPECTS(offset < this->size(), "Offset is past end of file", std::overflow_error); // Offset for `mmap()` must be page aligned _map_offset = offset & ~(sysconf(_SC_PAGESIZE) - 1); - if (size == 0 || (offset + size) > _file.size()) { size = _file.size() - offset; } + if (size == 0 || (offset + size) > this->size()) { size = this->size() - offset; } // Size for `mmap()` needs to include the page padding _map_size = size + (offset - _map_offset); diff --git a/cpp/src/io/utilities/file_io_utilities.cpp b/cpp/src/io/utilities/file_io_utilities.cpp deleted file mode 100644 index 28367c95430..00000000000 --- a/cpp/src/io/utilities/file_io_utilities.cpp +++ /dev/null @@ -1,389 +0,0 @@ -/* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include "file_io_utilities.hpp" - -#include "getenv_or.hpp" - -#include -#include -#include - -#include -#include - -#include -#include -#include -#include -#include - -namespace cudf { -namespace io { -namespace detail { -namespace { - -[[nodiscard]] int open_file_checked(std::string const& filepath, int flags, mode_t mode) -{ - auto const fd = open(filepath.c_str(), flags, mode); - if (fd == -1) { throw_on_file_open_failure(filepath, flags & O_CREAT); } - - return fd; -} - -[[nodiscard]] size_t get_file_size(int file_descriptor) -{ - struct stat st {}; - CUDF_EXPECTS(fstat(file_descriptor, &st) != -1, "Cannot query file size"); - return static_cast(st.st_size); -} - -} // namespace - -void force_init_cuda_context() -{ - // Workaround for https://github.com/rapidsai/cudf/issues/14140, where cuFileDriverOpen errors - // out if no CUDA calls have been made before it. This is a no-op if the CUDA context is already - // initialized. - cudaFree(nullptr); -} - -[[noreturn]] void throw_on_file_open_failure(std::string const& filepath, bool is_create) -{ - // save errno because it may be overwritten by subsequent calls - auto const err = errno; - - if (auto const path = std::filesystem::path(filepath); is_create) { - CUDF_EXPECTS(std::filesystem::exists(path.parent_path()), - "Cannot create output file; directory does not exist"); - - } else { - CUDF_EXPECTS(std::filesystem::exists(path), "Cannot open file; it does not exist"); - } - - std::array error_msg_buffer{}; - auto const error_msg = strerror_r(err, error_msg_buffer.data(), 1024); - CUDF_FAIL("Cannot open file; failed with errno: " + std::string{error_msg}); -} - -file_wrapper::file_wrapper(std::string const& filepath, int flags, mode_t mode) - : fd(open_file_checked(filepath.c_str(), flags, mode)), _size{get_file_size(fd)} -{ -} - -file_wrapper::~file_wrapper() { close(fd); } - -#ifdef CUDF_CUFILE_FOUND - -/** - * @brief Class that dynamically loads the cuFile library and manages the cuFile driver. - */ -class cufile_shim { - private: - cufile_shim(); - void modify_cufile_json() const; - void load_cufile_lib(); - - void* cf_lib = nullptr; - decltype(cuFileDriverOpen)* driver_open = nullptr; - decltype(cuFileDriverClose)* driver_close = nullptr; - - std::unique_ptr init_error; - [[nodiscard]] auto is_valid() const noexcept { return init_error == nullptr; } - - public: - cufile_shim(cufile_shim const&) = delete; - cufile_shim& operator=(cufile_shim const&) = delete; - - static cufile_shim const* instance(); - - ~cufile_shim() - { - // Explicit cuFile driver close should not be performed here to avoid segfault. However, in the - // absence of driver_close(), cuFile will implicitly do that, which in most cases causes - // segfault anyway. TODO: Revisit this conundrum once cuFile is fixed. - // https://github.com/rapidsai/cudf/issues/17121 - - if (cf_lib != nullptr) dlclose(cf_lib); - } - - decltype(cuFileHandleRegister)* handle_register = nullptr; - decltype(cuFileHandleDeregister)* handle_deregister = nullptr; - decltype(cuFileRead)* read = nullptr; - decltype(cuFileWrite)* write = nullptr; -}; - -void cufile_shim::modify_cufile_json() const -{ - std::string const json_path_env_var = "CUFILE_ENV_PATH_JSON"; - static temp_directory const tmp_config_dir{"cudf_cufile_config"}; - - // Modify the config file based on the policy - auto const config_file_path = getenv_or(json_path_env_var, "/etc/cufile.json"); - std::ifstream user_config_file(config_file_path); - // Modified config file is stored in a temporary directory - auto const cudf_config_path = tmp_config_dir.path() + "cufile.json"; - std::ofstream cudf_config_file(cudf_config_path); - - std::string line; - while (std::getline(user_config_file, line)) { - std::string const tag = "\"allow_compat_mode\""; - if (line.find(tag) != std::string::npos) { - // TODO: only replace the true/false value instead of replacing the whole line - // Enable compatibility mode when cuDF does not fall back to host path - cudf_config_file << tag << ": " - << (cufile_integration::is_always_enabled() ? "true" : "false") << ",\n"; - } else { - cudf_config_file << line << '\n'; - } - - // Point libcufile to the modified config file - CUDF_EXPECTS(setenv(json_path_env_var.c_str(), cudf_config_path.c_str(), 0) == 0, - "Failed to set the cuFile config file environment variable."); - } -} - -void cufile_shim::load_cufile_lib() -{ - for (auto&& name : {"libcufile.so.0", - // Prior to CUDA 11.7.1, although ABI - // compatibility was maintained, some (at least - // Debian) packages do not have the .0 symlink, - // instead request the exact version. - "libcufile.so.1.3.0" /* 11.7.0 */, - "libcufile.so.1.2.1" /* 11.6.2, 11.6.1 */, - "libcufile.so.1.2.0" /* 11.6.0 */, - "libcufile.so.1.1.1" /* 11.5.1 */, - "libcufile.so.1.1.0" /* 11.5.0 */, - "libcufile.so.1.0.2" /* 11.4.4, 11.4.3, 11.4.2 */, - "libcufile.so.1.0.1" /* 11.4.1 */, - "libcufile.so.1.0.0" /* 11.4.0 */}) { - cf_lib = dlopen(name, RTLD_LAZY | RTLD_LOCAL | RTLD_NODELETE); - if (cf_lib != nullptr) break; - } - CUDF_EXPECTS(cf_lib != nullptr, "Failed to load cuFile library"); - driver_open = reinterpret_cast(dlsym(cf_lib, "cuFileDriverOpen")); - CUDF_EXPECTS(driver_open != nullptr, "could not find cuFile cuFileDriverOpen symbol"); - driver_close = reinterpret_cast(dlsym(cf_lib, "cuFileDriverClose")); - CUDF_EXPECTS(driver_close != nullptr, "could not find cuFile cuFileDriverClose symbol"); - handle_register = - reinterpret_cast(dlsym(cf_lib, "cuFileHandleRegister")); - CUDF_EXPECTS(handle_register != nullptr, "could not find cuFile cuFileHandleRegister symbol"); - handle_deregister = - reinterpret_cast(dlsym(cf_lib, "cuFileHandleDeregister")); - CUDF_EXPECTS(handle_deregister != nullptr, "could not find cuFile cuFileHandleDeregister symbol"); - read = reinterpret_cast(dlsym(cf_lib, "cuFileRead")); - CUDF_EXPECTS(read != nullptr, "could not find cuFile cuFileRead symbol"); - write = reinterpret_cast(dlsym(cf_lib, "cuFileWrite")); - CUDF_EXPECTS(write != nullptr, "could not find cuFile cuFileWrite symbol"); -} - -cufile_shim::cufile_shim() -{ - try { - modify_cufile_json(); - load_cufile_lib(); - - CUDF_EXPECTS(driver_open().err == CU_FILE_SUCCESS, "Failed to initialize cuFile driver"); - } catch (cudf::logic_error const& err) { - init_error = std::make_unique(err); - } -} - -cufile_shim const* cufile_shim::instance() -{ - static cufile_shim _instance; - // Defer throwing to avoid repeated attempts to load the library - if (!_instance.is_valid()) CUDF_FAIL("" + std::string(_instance.init_error->what())); - - return &_instance; -} - -void cufile_registered_file::register_handle() -{ - CUfileDescr_t cufile_desc{}; - cufile_desc.handle.fd = _file.desc(); - cufile_desc.type = CU_FILE_HANDLE_TYPE_OPAQUE_FD; - CUDF_EXPECTS(shim->handle_register(&cf_handle, &cufile_desc).err == CU_FILE_SUCCESS, - "Cannot register file handle with cuFile"); -} - -cufile_registered_file::~cufile_registered_file() { shim->handle_deregister(cf_handle); } - -cufile_input_impl::cufile_input_impl(std::string const& filepath) - : shim{cufile_shim::instance()}, - cf_file(shim, filepath, O_RDONLY | O_DIRECT), - // The benefit from multithreaded read plateaus around 16 threads - pool(getenv_or("LIBCUDF_CUFILE_THREAD_COUNT", 16)) -{ -} - -namespace { - -template > -std::vector> make_sliced_tasks( - F function, DataT* ptr, size_t offset, size_t size, BS::thread_pool& pool) -{ - constexpr size_t default_max_slice_size = 4 * 1024 * 1024; - static auto const max_slice_size = getenv_or("LIBCUDF_CUFILE_SLICE_SIZE", default_max_slice_size); - auto const slices = make_file_io_slices(size, max_slice_size); - std::vector> slice_tasks; - std::transform(slices.cbegin(), slices.cend(), std::back_inserter(slice_tasks), [&](auto& slice) { - return pool.submit_task( - [=] { return function(ptr + slice.offset, slice.size, offset + slice.offset); }); - }); - return slice_tasks; -} - -} // namespace - -std::future cufile_input_impl::read_async(size_t offset, - size_t size, - uint8_t* dst, - rmm::cuda_stream_view stream) -{ - int device = 0; - CUDF_CUDA_TRY(cudaGetDevice(&device)); - - auto read_slice = [device, gds_read = shim->read, file_handle = cf_file.handle()]( - void* dst, size_t size, size_t offset) -> ssize_t { - CUDF_CUDA_TRY(cudaSetDevice(device)); - auto read_size = gds_read(file_handle, dst, size, offset, 0); - CUDF_EXPECTS(read_size != -1, "cuFile error reading from a file"); - return read_size; - }; - - auto slice_tasks = make_sliced_tasks(read_slice, dst, offset, size, pool); - - auto waiter = [](auto slice_tasks) -> size_t { - return std::accumulate(slice_tasks.begin(), slice_tasks.end(), 0, [](auto sum, auto& task) { - return sum + task.get(); - }); - }; - // The future returned from this function is deferred, not async because we want to avoid creating - // threads for each read_async call. This overhead is significant in case of multiple small reads. - return std::async(std::launch::deferred, waiter, std::move(slice_tasks)); -} - -cufile_output_impl::cufile_output_impl(std::string const& filepath) - : shim{cufile_shim::instance()}, - cf_file(shim, filepath, O_CREAT | O_RDWR | O_DIRECT, 0664), - pool(getenv_or("LIBCUDF_CUFILE_THREAD_COUNT", 16)) -{ -} - -std::future cufile_output_impl::write_async(void const* data, size_t offset, size_t size) -{ - int device = 0; - CUDF_CUDA_TRY(cudaGetDevice(&device)); - - auto write_slice = [device, gds_write = shim->write, file_handle = cf_file.handle()]( - void const* src, size_t size, size_t offset) -> void { - CUDF_CUDA_TRY(cudaSetDevice(device)); - auto write_size = gds_write(file_handle, src, size, offset, 0); - CUDF_EXPECTS(write_size != -1 and write_size == static_cast(size), - "cuFile error writing to a file"); - }; - - auto source = static_cast(data); - auto slice_tasks = make_sliced_tasks(write_slice, source, offset, size, pool); - - auto waiter = [](auto slice_tasks) -> void { - for (auto const& task : slice_tasks) { - task.wait(); - } - }; - // The future returned from this function is deferred, not async because we want to avoid creating - // threads for each write_async call. This overhead is significant in case of multiple small - // writes. - return std::async(std::launch::deferred, waiter, std::move(slice_tasks)); -} -#else -cufile_input_impl::cufile_input_impl(std::string const& filepath) -{ - CUDF_FAIL("Cannot create cuFile source, current build was compiled without cuFile headers"); -} - -cufile_output_impl::cufile_output_impl(std::string const& filepath) -{ - CUDF_FAIL("Cannot create cuFile sink, current build was compiled without cuFile headers"); -} -#endif - -std::unique_ptr make_cufile_input(std::string const& filepath) -{ - if (cufile_integration::is_gds_enabled()) { - try { - auto cufile_in = std::make_unique(filepath); - CUDF_LOG_INFO("File successfully opened for reading with GDS."); - return cufile_in; - } catch (...) { - if (cufile_integration::is_always_enabled()) { - CUDF_LOG_ERROR( - "Failed to open file for reading with GDS. Enable bounce buffer fallback to read this " - "file."); - throw; - } - CUDF_LOG_INFO( - "Failed to open file for reading with GDS. Data will be read from the file using a bounce " - "buffer (possible performance impact)."); - } - } - return {}; -} - -std::unique_ptr make_cufile_output(std::string const& filepath) -{ - if (cufile_integration::is_gds_enabled()) { - try { - auto cufile_out = std::make_unique(filepath); - CUDF_LOG_INFO("File successfully opened for writing with GDS."); - return cufile_out; - } catch (...) { - if (cufile_integration::is_always_enabled()) { - CUDF_LOG_ERROR( - "Failed to open file for writing with GDS. Enable bounce buffer fallback to write to " - "this file."); - throw; - } - CUDF_LOG_INFO( - "Failed to open file for writing with GDS. Data will be written to the file using a bounce " - "buffer (possible performance impact)."); - } - } - return {}; -} - -std::vector make_file_io_slices(size_t size, size_t max_slice_size) -{ - max_slice_size = std::max(1024ul, max_slice_size); - auto const n_slices = util::div_rounding_up_safe(size, max_slice_size); - std::vector slices; - slices.reserve(n_slices); - std::generate_n(std::back_inserter(slices), n_slices, [&, idx = 0]() mutable { - auto const slice_offset = idx++ * max_slice_size; - auto const slice_size = std::min(size - slice_offset, max_slice_size); - return file_io_slice{slice_offset, slice_size}; - }); - - return slices; -} - -} // namespace detail -} // namespace io -} // namespace cudf diff --git a/cpp/src/io/utilities/file_io_utilities.hpp b/cpp/src/io/utilities/file_io_utilities.hpp deleted file mode 100644 index c844a596869..00000000000 --- a/cpp/src/io/utilities/file_io_utilities.hpp +++ /dev/null @@ -1,229 +0,0 @@ -/* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#pragma once - -#ifdef CUDF_CUFILE_FOUND -#include - -#include -#include -#endif - -#include -#include -#include - -#include - -#include - -namespace cudf { -namespace io { -namespace detail { - -[[noreturn]] void throw_on_file_open_failure(std::string const& filepath, bool is_create); - -// Call before any cuFile API calls to ensure the CUDA context is initialized. -void force_init_cuda_context(); - -/** - * @brief Class that provides RAII for file handling. - */ -class file_wrapper { - int fd = -1; - size_t _size = 0; - - public: - explicit file_wrapper(std::string const& filepath, int flags, mode_t mode = 0); - ~file_wrapper(); - [[nodiscard]] auto size() const { return _size; } - [[nodiscard]] auto desc() const { return fd; } -}; - -/** - * @brief Interface class for cufile input. - */ -class cufile_input { - public: - /** - * @brief Asynchronously reads into existing device memory. - * - * @throws cudf::logic_error on cuFile error - * - * @param offset Number of bytes from the start - * @param size Number of bytes to read - * @param dst Address of the existing device memory - * @param stream CUDA stream to use - * - * @return The number of bytes read as an std::future - */ - virtual std::future read_async(size_t offset, - size_t size, - uint8_t* dst, - rmm::cuda_stream_view stream) = 0; -}; - -/** - * @brief Interface class for cufile output. - */ -class cufile_output { - public: - /** - * @brief Asynchronously writes the data from a device buffer into a file. - * - * It is the caller's responsibility to not invalidate `data` until the result from this function - * is synchronized. - * - * @throws cudf::logic_error on cuFile error - * - * @param data Pointer to the buffer to be written into the output file - * @param offset Number of bytes from the start - * @param size Number of bytes to write - */ - virtual std::future write_async(void const* data, size_t offset, size_t size) = 0; -}; - -#ifdef CUDF_CUFILE_FOUND - -class cufile_shim; - -/** - * @brief Class that provides RAII for cuFile file registration. - */ -class cufile_registered_file { - void register_handle(); - - public: - cufile_registered_file(cufile_shim const* shim, std::string const& filepath, int flags) - : _file(filepath, flags), shim{shim} - { - register_handle(); - } - - cufile_registered_file(cufile_shim const* shim, - std::string const& filepath, - int flags, - mode_t mode) - : _file(filepath, flags, mode), shim{shim} - { - register_handle(); - } - - [[nodiscard]] auto const& handle() const noexcept { return cf_handle; } - - ~cufile_registered_file(); - - private: - file_wrapper const _file; - CUfileHandle_t cf_handle = nullptr; - cufile_shim const* shim = nullptr; -}; - -/** - * @brief Adapter for the `cuFileRead` API. - * - * Exposes APIs to read directly from a file into device memory. - */ -class cufile_input_impl final : public cufile_input { - public: - cufile_input_impl(std::string const& filepath); - - std::future read_async(size_t offset, - size_t size, - uint8_t* dst, - rmm::cuda_stream_view stream) override; - - private: - cufile_shim const* shim = nullptr; - cufile_registered_file const cf_file; - BS::thread_pool pool; -}; - -/** - * @brief Adapter for the `cuFileWrite` API. - * - * Exposes an API to write directly into a file from device memory. - */ -class cufile_output_impl final : public cufile_output { - public: - cufile_output_impl(std::string const& filepath); - - std::future write_async(void const* data, size_t offset, size_t size) override; - - private: - cufile_shim const* shim = nullptr; - cufile_registered_file const cf_file; - BS::thread_pool pool; -}; -#else - -class cufile_input_impl final : public cufile_input { - public: - cufile_input_impl(std::string const& filepath); - std::future read_async(size_t offset, - size_t size, - uint8_t* dst, - rmm::cuda_stream_view stream) override - { - CUDF_FAIL("Only used to compile without cufile library, should not be called"); - } -}; - -class cufile_output_impl final : public cufile_output { - public: - cufile_output_impl(std::string const& filepath); - std::future write_async(void const* data, size_t offset, size_t size) override - { - CUDF_FAIL("Only used to compile without cufile library, should not be called"); - } -}; -#endif - -/** - * @brief Creates a `cufile_input_impl` object - * - * Returns a null pointer if an exception occurs in the `cufile_input_impl` constructor, or if the - * cuFile library is not installed. - */ -std::unique_ptr make_cufile_input(std::string const& filepath); - -/** - * @brief Creates a `cufile_output_impl` object - * - * Returns a null pointer if an exception occurs in the `cufile_output_impl` constructor, or if the - * cuFile library is not installed. - */ -std::unique_ptr make_cufile_output(std::string const& filepath); - -/** - * @brief Byte range to be read/written in a single operation. - */ -CUDF_EXPORT struct file_io_slice { - size_t offset; - size_t size; -}; - -/** - * @brief Split the total number of bytes to read/write into slices to enable parallel IO. - * - * If `max_slice_size` is below 1024, 1024 will be used instead to prevent potential misuse. - */ -CUDF_EXPORT std::vector make_file_io_slices(size_t size, size_t max_slice_size); - -} // namespace detail -} // namespace io -} // namespace cudf diff --git a/cpp/tests/CMakeLists.txt b/cpp/tests/CMakeLists.txt index 117cd620679..fd8cb3f22f2 100644 --- a/cpp/tests/CMakeLists.txt +++ b/cpp/tests/CMakeLists.txt @@ -306,11 +306,6 @@ ConfigureTest( GPUS 1 PERCENT 30 EXTRA_LIBS ${ARROW_LIBRARIES} ) -ConfigureTest( - FILE_IO_TEST io/file_io_test.cpp - GPUS 1 - PERCENT 30 -) ConfigureTest( ORC_TEST io/orc_chunked_reader_test.cu io/orc_test.cpp GPUS 1 diff --git a/cpp/tests/io/file_io_test.cpp b/cpp/tests/io/file_io_test.cpp deleted file mode 100644 index 1b85541687a..00000000000 --- a/cpp/tests/io/file_io_test.cpp +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include -#include - -#include - -// Base test fixture for tests -struct CuFileIOTest : public cudf::test::BaseFixture {}; - -TEST_F(CuFileIOTest, SliceSize) -{ - std::vector> test_cases{ - {1 << 20, 1 << 18}, {1 << 18, 1 << 20}, {1 << 20, 3333}, {0, 1 << 18}, {0, 0}, {1 << 20, 0}}; - for (auto const& test_case : test_cases) { - auto const slices = cudf::io::detail::make_file_io_slices(test_case.first, test_case.second); - if (slices.empty()) { - ASSERT_EQ(test_case.first, 0); - } else { - ASSERT_EQ(slices.front().offset, 0); - ASSERT_EQ(slices.back().offset + slices.back().size, test_case.first); - for (auto i = 1u; i < slices.size(); ++i) { - ASSERT_EQ(slices[i].offset, slices[i - 1].offset + slices[i - 1].size); - } - } - } -} - -CUDF_TEST_PROGRAM_MAIN() diff --git a/docs/cudf/source/user_guide/io/io.md b/docs/cudf/source/user_guide/io/io.md index 7d863d890e2..7d644770a0d 100644 --- a/docs/cudf/source/user_guide/io/io.md +++ b/docs/cudf/source/user_guide/io/io.md @@ -80,54 +80,39 @@ IO format. - \[¹\] - Not all orientations are GPU-accelerated. - \[²\] - Not GPU-accelerated. -## Magnum IO GPUDirect Storage Integration +## KvikIO Integration + +cuDF leverages the [KvikIO](https://github.com/rapidsai/kvikio) library for high-performance +I/O features, such as parallel I/O operations and NVIDIA Magnum IO GPUDirect Storage (GDS). Many IO APIs can use Magnum IO GPUDirect Storage (GDS) library to optimize IO operations. GDS enables a direct data path for direct memory access (DMA) transfers between GPU memory and storage, which avoids a bounce -buffer through the CPU. GDS also has a compatibility mode that allows -the library to fall back to copying through a CPU bounce buffer. The -SDK is available for download +buffer through the CPU. The SDK is available for download [here](https://developer.nvidia.com/gpudirect-storage). GDS is also included in CUDA Toolkit 11.4 and higher. -Use of GPUDirect Storage in cuDF is disabled by default, but can be -enabled through the environment variable `LIBCUDF_CUFILE_POLICY`. -This variable also controls the GDS compatibility mode. - -There are four valid values for the environment variable: - -- "GDS": Enable GDS use. If the cuFile library cannot be properly loaded, -fall back to the GDS compatibility mode. -- "ALWAYS": Enable GDS use. If the cuFile library cannot be properly loaded, -throw an exception. -- "KVIKIO": Enable GDS compatibility mode through [KvikIO](https://github.com/rapidsai/kvikio). -Note that KvikIO also provides the environment variable `KVIKIO_COMPAT_MODE` for GDS -control that may alter the effect of "KVIKIO" option in cuDF: - - By default, `KVIKIO_COMPAT_MODE` is unset. In this case, cuDF enforces - the GDS compatibility mode, and the system configuration check for GDS I/O - is never performed. - - If `KVIKIO_COMPAT_MODE=ON`, this is the same with the above case. - - If `KVIKIO_COMPAT_MODE=OFF`, KvikIO enforces GDS I/O without system - configuration check, and will error out if GDS requirements are not met. The - only exceptional case is that if the system does not support files being - opened with the `O_DIRECT` flag, the GDS compatibility mode will be used. -- "OFF": Completely disable GDS and kvikIO use. - -If no value is set, behavior will be the same as the "KVIKIO" option. - -This environment variable also affects how cuDF treats GDS errors. - -- When `LIBCUDF_CUFILE_POLICY` is set to "GDS" and a GDS API call - fails for any reason, cuDF falls back to the internal implementation - with bounce buffers. -- When `LIBCUDF_CUFILE_POLICY` is set to "ALWAYS" and a GDS API call -fails for any reason (unlikely, given that the compatibility mode is -on), cuDF throws an exception to propagate the error to the user. -- When `LIBCUDF_CUFILE_POLICY` is set to "KVIKIO" and a KvikIO API - call fails for any reason (unlikely, given that KvikIO implements - its own compatibility mode) cuDF throws an exception to propagate - the error to the user. +Use of GDS in cuDF is controlled by KvikIO's environment variable `KVIKIO_COMPAT_MODE`. It has +3 options (case-insensitive): + +- `ON` (aliases: `TRUE`, `YES`, `1`): Enable the compatibility mode, which enforces KvikIO POSIX I/O path. + This is the default option in cuDF. +- `OFF` (aliases: `FALSE`, `NO`, `0`): Force-enable KvikIO cuFile (the underlying API for GDS) I/O path. + GDS will be activated if the system requirements for cuFile are met and cuFile is properly + configured. However, if the system is not suited for cuFile, I/O operations under the `OFF` + option may error out. +- `AUTO`: Try KvikIO cuFile I/O path first, and fall back to KvikIO POSIX I/O if the system requirements + for cuFile are not met. + +Note that: +- Even if KvikIO cuFile I/O path is taken, it is possible that GDS is still not activated, where cuFile falls back + to its internal compatibility mode. This will happen, for example, on an ext4 file system whose journaling + mode has not been explicitly set to `data=ordered`. This may also happen if cuFile's environment variable + `CUFILE_FORCE_COMPAT_MODE` is set to true. For more details, refer to + [cuFile compatibility mode](https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html#cufile-compatibility-mode) + and [cuFile environment variables](https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html#environment-variables). +- Details of the GDS system requirements can be found in the [GDS documentation](https://docs.nvidia.com/gpudirect-storage/index.html). +- If a KvikIO API call fails for any reason, cuDF throws an exception to propagate the error to the user. For more information about error handling, compatibility mode, and tuning parameters in KvikIO see: @@ -143,15 +128,6 @@ Operations that support the use of GPUDirect Storage: - {py:meth}`cudf.DataFrame.to_parquet` - {py:meth}`cudf.DataFrame.to_orc` -Several parameters that can be used to tune the performance of -GDS-enabled I/O are exposed through environment variables: - -- `LIBCUDF_CUFILE_THREAD_COUNT`: Integral value, maximum number of - parallel reads/writes per file (default 16); -- `LIBCUDF_CUFILE_SLICE_SIZE`: Integral value, maximum size of each - GDS read/write, in bytes (default 4MB). Larger I/O operations are - split into multiple calls. - ## nvCOMP Integration Some types of compression/decompression can be performed using either From 83aafcde71c693c6869a825d07ec249400381ba3 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Thu, 13 Feb 2025 00:56:56 -0800 Subject: [PATCH 029/129] Avoid cudf.dtype calls in build_column/column_empty/.where (#17979) Continuation of https://github.com/rapidsai/cudf/pull/17918 Authors: - Matthew Roeschke (https://github.com/mroeschke) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17979 --- python/cudf/cudf/core/_internals/where.py | 27 +++++------- python/cudf/cudf/core/column/column.py | 44 +++++++++----------- python/cudf/cudf/core/column/datetime.py | 4 +- python/cudf/cudf/core/column/decimal.py | 3 +- python/cudf/cudf/core/column/timedelta.py | 2 +- python/cudf/cudf/core/dataframe.py | 12 ++++-- python/cudf/cudf/core/dtypes.py | 6 +-- python/cudf/cudf/core/groupby/groupby.py | 7 +--- python/cudf/cudf/core/indexed_frame.py | 2 +- python/cudf/cudf/core/single_column_frame.py | 5 ++- python/cudf/cudf/io/orc.py | 6 ++- python/cudf/cudf/io/parquet.py | 21 +++------- python/cudf/cudf/tests/test_apply_rows.py | 6 +-- python/cudf/cudf/tests/test_column.py | 14 ++++--- python/cudf/cudf/tests/test_parquet.py | 14 ++++--- python/cudf/cudf/tests/test_testing.py | 6 +-- python/cudf/cudf/utils/applyutils.py | 5 ++- python/cudf/cudf/utils/queryutils.py | 2 +- 18 files changed, 90 insertions(+), 96 deletions(-) diff --git a/python/cudf/cudf/core/_internals/where.py b/python/cudf/cudf/core/_internals/where.py index 2199d4d5ba5..73011d6ffe0 100644 --- a/python/cudf/cudf/core/_internals/where.py +++ b/python/cudf/cudf/core/_internals/where.py @@ -1,4 +1,4 @@ -# Copyright (c) 2021-2024, NVIDIA CORPORATION. +# Copyright (c) 2021-2025, NVIDIA CORPORATION. from __future__ import annotations import warnings @@ -12,7 +12,7 @@ from cudf.utils.dtypes import find_common_type, is_mixed_with_object_dtype if TYPE_CHECKING: - from cudf._typing import ScalarLike + from cudf._typing import DtypeObj, ScalarLike from cudf.core.column import ColumnBase @@ -106,20 +106,13 @@ def _check_and_cast_columns_with_other( return _normalize_categorical(source_col.astype(common_dtype), other) -def _can_cast(from_dtype, to_dtype): +def _can_cast(from_dtype: DtypeObj, to_dtype: DtypeObj) -> bool: """ Utility function to determine if we can cast from `from_dtype` to `to_dtype`. This function primarily calls `np.can_cast` but with some special handling around cudf specific dtypes. """ - if cudf.utils.utils.is_na_like(from_dtype): - return True - if isinstance(from_dtype, type): - from_dtype = cudf.dtype(from_dtype) - if isinstance(to_dtype, type): - to_dtype = cudf.dtype(to_dtype) - # TODO : Add precision & scale checking for # decimal types in future @@ -131,6 +124,8 @@ def _can_cast(from_dtype, to_dtype): return True else: return False + else: + return False elif isinstance(from_dtype, np.dtype): if isinstance(to_dtype, np.dtype): return np.can_cast(from_dtype, to_dtype) @@ -139,22 +134,22 @@ def _can_cast(from_dtype, to_dtype): return True else: return False - elif isinstance(to_dtype, cudf.core.types.CategoricalDtype): + elif isinstance(to_dtype, cudf.CategoricalDtype): return True else: return False - elif isinstance(from_dtype, cudf.core.dtypes.ListDtype): + elif isinstance(from_dtype, cudf.ListDtype): # TODO: Add level based checks too once casting of # list columns is supported - if isinstance(to_dtype, cudf.core.dtypes.ListDtype): + if isinstance(to_dtype, cudf.ListDtype): return np.can_cast(from_dtype.leaf_type, to_dtype.leaf_type) else: return False - elif isinstance(from_dtype, cudf.core.dtypes.CategoricalDtype): - if isinstance(to_dtype, cudf.core.dtypes.CategoricalDtype): + elif isinstance(from_dtype, cudf.CategoricalDtype): + if isinstance(to_dtype, cudf.CategoricalDtype): return True elif isinstance(to_dtype, np.dtype): - return np.can_cast(from_dtype._categories.dtype, to_dtype) + return np.can_cast(from_dtype.categories.dtype, to_dtype) else: return False else: diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index 4429de952ee..6268ffb356d 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -61,6 +61,7 @@ from cudf.core.scalar import pa_scalar_to_plc_scalar from cudf.errors import MixedTypeError from cudf.utils.dtypes import ( + CUDF_STRING_DTYPE, SIZE_TYPE_DTYPE, _maybe_convert_to_default_type, cudf_dtype_from_pa_type, @@ -78,7 +79,7 @@ if TYPE_CHECKING: import builtins - from cudf._typing import ColumnLike, Dtype, ScalarLike + from cudf._typing import ColumnLike, Dtype, DtypeObj, ScalarLike from cudf.core.column.numerical import NumericalColumn from cudf.core.column.strings import StringColumn @@ -1756,7 +1757,7 @@ def _has_any_nan(arbitrary: pd.Series | np.ndarray) -> bool: def column_empty( row_count: int, - dtype: Dtype = "object", + dtype: DtypeObj = CUDF_STRING_DTYPE, for_numba: bool = False, ) -> ColumnBase: """ @@ -1776,7 +1777,6 @@ def column_empty( for_numba : bool, default False If True, don't allocate a mask as it's not supported by numba. """ - dtype = cudf.dtype(dtype) children: tuple[ColumnBase, ...] = () if isinstance(dtype, StructDtype): @@ -1824,7 +1824,7 @@ def column_empty( def build_column( data: Buffer | None, - dtype: Dtype, + dtype: DtypeObj, *, size: int | None = None, mask: Buffer | None = None, @@ -1848,20 +1848,6 @@ def build_column( offset : int, optional children : tuple, optional """ - dtype = cudf.dtype(dtype) - - if _is_non_decimal_numeric_dtype(dtype): - assert data is not None - col = cudf.core.column.NumericalColumn( - data=data, - dtype=dtype, - mask=mask, - size=size, - offset=offset, - null_count=null_count, - ) - return col - if isinstance(dtype, CategoricalDtype): return cudf.core.column.CategoricalColumn( data=data, # type: ignore[arg-type] @@ -1872,8 +1858,8 @@ def build_column( null_count=null_count, children=children, # type: ignore[arg-type] ) - elif dtype.type is np.datetime64: - return cudf.core.column.DatetimeColumn( + elif isinstance(dtype, pd.DatetimeTZDtype): + return cudf.core.column.datetime.DatetimeTZColumn( data=data, # type: ignore[arg-type] dtype=dtype, mask=mask, @@ -1881,8 +1867,8 @@ def build_column( offset=offset, null_count=null_count, ) - elif isinstance(dtype, pd.DatetimeTZDtype): - return cudf.core.column.datetime.DatetimeTZColumn( + elif dtype.kind == "M": + return cudf.core.column.DatetimeColumn( data=data, # type: ignore[arg-type] dtype=dtype, mask=mask, @@ -1890,7 +1876,7 @@ def build_column( offset=offset, null_count=null_count, ) - elif dtype.type is np.timedelta64: + elif dtype.kind == "m": return cudf.core.column.TimeDeltaColumn( data=data, # type: ignore[arg-type] dtype=dtype, @@ -1968,6 +1954,15 @@ def build_column( null_count=null_count, children=children, ) + elif dtype.kind in "iufb": + return cudf.core.column.NumericalColumn( + data=data, # type: ignore[arg-type] + dtype=dtype, + mask=mask, + size=size, + offset=offset, + null_count=null_count, + ) else: raise TypeError(f"Unrecognized dtype: {dtype}") @@ -2560,8 +2555,7 @@ def deserialize_columns(headers: list[dict], frames: list) -> list[ColumnBase]: def concat_columns(objs: "MutableSequence[ColumnBase]") -> ColumnBase: """Concatenate a sequence of columns.""" if len(objs) == 0: - dtype = np.dtype(np.float64) - return column_empty(0, dtype=dtype) + return column_empty(0, dtype=np.dtype(np.float64)) # If all columns are `NumericalColumn` with different dtypes, # we cast them to a common dtype. diff --git a/python/cudf/cudf/core/column/datetime.py b/python/cudf/cudf/core/column/datetime.py index 8fd2a2b68d5..1373febb47d 100644 --- a/python/cudf/cudf/core/column/datetime.py +++ b/python/cudf/cudf/core/column/datetime.py @@ -601,12 +601,12 @@ def strftime(self, format: str) -> cudf.core.column.StringColumn: if len(self) == 0: return cast( cudf.core.column.StringColumn, - column.column_empty(0, dtype="object"), + column.column_empty(0, dtype=CUDF_STRING_DTYPE), ) if format in _DATETIME_SPECIAL_FORMATS: names = as_column(_DATETIME_NAMES) else: - names = column.column_empty(0, dtype="object") + names = column.column_empty(0, dtype=CUDF_STRING_DTYPE) with acquire_spill_lock(): return type(self).from_pylibcudf( # type: ignore[return-value] plc.strings.convert.convert_datetime.from_timestamps( diff --git a/python/cudf/cudf/core/column/decimal.py b/python/cudf/cudf/core/column/decimal.py index 0f233b5bdc4..3c603c8e6ef 100644 --- a/python/cudf/cudf/core/column/decimal.py +++ b/python/cudf/cudf/core/column/decimal.py @@ -25,6 +25,7 @@ DecimalDtype, ) from cudf.core.mixins import BinaryOperand +from cudf.utils.dtypes import CUDF_STRING_DTYPE from cudf.utils.utils import pa_mask_buffer_to_mask if TYPE_CHECKING: @@ -98,7 +99,7 @@ def as_string_column(self) -> cudf.core.column.StringColumn: else: return cast( cudf.core.column.StringColumn, - cudf.core.column.column_empty(0, dtype="object"), + cudf.core.column.column_empty(0, dtype=CUDF_STRING_DTYPE), ) def __pow__(self, other): diff --git a/python/cudf/cudf/core/column/timedelta.py b/python/cudf/cudf/core/column/timedelta.py index 022cfe2fe2e..b45c62589d7 100644 --- a/python/cudf/cudf/core/column/timedelta.py +++ b/python/cudf/cudf/core/column/timedelta.py @@ -338,7 +338,7 @@ def strftime(self, format: str) -> cudf.core.column.StringColumn: if len(self) == 0: return cast( cudf.core.column.StringColumn, - column.column_empty(0, dtype="object"), + column.column_empty(0, dtype=CUDF_STRING_DTYPE), ) else: with acquire_spill_lock(): diff --git a/python/cudf/cudf/core/dataframe.py b/python/cudf/cudf/core/dataframe.py index 5041c9be476..5225a4b97ec 100644 --- a/python/cudf/cudf/core/dataframe.py +++ b/python/cudf/cudf/core/dataframe.py @@ -85,6 +85,7 @@ from cudf.utils import applyutils, docutils, ioutils, queryutils from cudf.utils.docutils import copy_docstring from cudf.utils.dtypes import ( + CUDF_STRING_DTYPE, can_convert_to_column, cudf_dtype_from_pydata_dtype, find_common_type, @@ -778,7 +779,7 @@ def __init__( label_dtype = getattr(columns, "dtype", None) self._data = ColumnAccessor( { - k: column_empty(len(self), dtype="object") + k: column_empty(len(self), dtype=CUDF_STRING_DTYPE) for k in columns }, level_names=tuple(columns.names) @@ -984,7 +985,7 @@ def _init_from_series_list(self, data, columns, index): for col_name in columns: if col_name not in self._data: self._data[col_name] = column_empty( - row_count=len(self), dtype=None + row_count=len(self), dtype=np.dtype(np.float64) ) self._data._level_names = ( tuple(columns.names) @@ -1035,7 +1036,10 @@ def _init_from_list_like(self, data, index=None, columns=None): data = list(itertools.zip_longest(*data)) if columns is not None and len(data) == 0: - data = [column_empty(row_count=0, dtype=None) for _ in columns] + data = [ + column_empty(row_count=0, dtype=np.dtype(np.float64)) + for _ in columns + ] for col_name, col in enumerate(data): self._data[col_name] = column.as_column(col) self._data.rangeindex = True @@ -3344,7 +3348,7 @@ def _insert(self, loc, name, value, nan_as_null=None, ignore_index=True): dtype = value.dtype value = value.item() if _is_null_host_scalar(value): - dtype = "str" + dtype = CUDF_STRING_DTYPE value = as_column( value, length=len(self), diff --git a/python/cudf/cudf/core/dtypes.py b/python/cudf/cudf/core/dtypes.py index f7ad49aed9f..983950580d0 100644 --- a/python/cudf/cudf/core/dtypes.py +++ b/python/cudf/cudf/core/dtypes.py @@ -19,7 +19,7 @@ from cudf.core._compat import PANDAS_GE_210, PANDAS_LT_300 from cudf.core.abc import Serializable from cudf.utils.docutils import doc_apply -from cudf.utils.dtypes import CUDF_STRING_DTYPE +from cudf.utils.dtypes import CUDF_STRING_DTYPE, cudf_dtype_from_pa_type if PANDAS_GE_210: PANDAS_NUMPY_DTYPE = pd.core.dtypes.dtypes.NumpyEADtype @@ -188,7 +188,7 @@ def categories(self) -> cudf.Index: Index(['b', 'a'], dtype='object') """ if self._categories is None: - col = cudf.core.column.column_empty(0, dtype="object") + col = cudf.core.column.column_empty(0, dtype=CUDF_STRING_DTYPE) else: col = self._categories return cudf.Index._from_column(col) @@ -395,7 +395,7 @@ def element_type(self) -> Dtype: elif isinstance(self._typ.value_type, pa.StructType): return StructDtype.from_arrow(self._typ.value_type) else: - return cudf.dtype(self._typ.value_type.to_pandas_dtype()) + return cudf_dtype_from_pa_type(self._typ.value_type) @cached_property def leaf_type(self): diff --git a/python/cudf/cudf/core/groupby/groupby.py b/python/cudf/cudf/core/groupby/groupby.py index 7b423f9d8a5..94e0f9155f6 100644 --- a/python/cudf/cudf/core/groupby/groupby.py +++ b/python/cudf/cudf/core/groupby/groupby.py @@ -657,7 +657,7 @@ def size(self): """ Return the size of each group. """ - col = cudf.core.column.column_empty(len(self.obj), "int8") + col = column_empty(len(self.obj), np.dtype(np.int8)) result = ( cudf.Series._from_column(col, name=getattr(self.obj, "name", None)) .groupby(self.grouping, sort=self._sort, dropna=self._dropna) @@ -684,10 +684,7 @@ def cumcount(self, ascending: bool = True): ) return ( cudf.Series._from_column( - cudf.core.column.column_empty( - len(self.obj), - "int8", - ), + column_empty(len(self.obj), np.dtype(np.int8)), index=self.obj.index, ) .groupby(self.grouping, sort=self._sort) diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index 589dc580ba1..aaf73e122ed 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -3900,7 +3900,7 @@ def _reindex( df._data[name].copy(deep=deep) if name in df._data else cudf.core.column.column.column_empty( - dtype=dtypes.get(name, np.float64), + dtype=dtypes.get(name, np.dtype(np.float64)), row_count=len(index), ) ) diff --git a/python/cudf/cudf/core/single_column_frame.py b/python/cudf/cudf/core/single_column_frame.py index 9c8da020ddc..f9713ca62d1 100644 --- a/python/cudf/cudf/core/single_column_frame.py +++ b/python/cudf/cudf/core/single_column_frame.py @@ -1,4 +1,4 @@ -# Copyright (c) 2021-2024, NVIDIA CORPORATION. +# Copyright (c) 2021-2025, NVIDIA CORPORATION. """Base class for Frame types that only have a single column.""" from __future__ import annotations @@ -17,6 +17,7 @@ from cudf.core.column import ColumnBase, as_column from cudf.core.column_accessor import ColumnAccessor from cudf.core.frame import Frame +from cudf.utils.dtypes import SIZE_TYPE_DTYPE from cudf.utils.performance_tracking import _performance_tracking from cudf.utils.utils import NotIterable @@ -346,7 +347,7 @@ def _get_elements_from_column(self, arg) -> ScalarLike | ColumnBase: else: arg = as_column(arg) if len(arg) == 0: - arg = cudf.core.column.column_empty(0, dtype="int32") + arg = cudf.core.column.column_empty(0, dtype=SIZE_TYPE_DTYPE) if arg.dtype.kind in "iu": return self._column.take(arg) if arg.dtype.kind == "b": diff --git a/python/cudf/cudf/io/orc.py b/python/cudf/cudf/io/orc.py index 4f88a7fed2f..9fd40eff119 100644 --- a/python/cudf/cudf/io/orc.py +++ b/python/cudf/cudf/io/orc.py @@ -16,7 +16,7 @@ from cudf.core.column_accessor import ColumnAccessor from cudf.core.index import _index_from_data from cudf.utils import ioutils -from cudf.utils.dtypes import dtype_to_pylibcudf_type +from cudf.utils.dtypes import cudf_dtype_from_pa_type, dtype_to_pylibcudf_type try: import ujson as json # type: ignore[import-untyped] @@ -220,7 +220,9 @@ def read_orc( data={ col_name: cudf.core.column.column_empty( row_count=0, - dtype=schema.field(col_name).type.to_pandas_dtype(), + dtype=cudf_dtype_from_pa_type( + schema.field(col_name).type + ), ) for col_name in col_names } diff --git a/python/cudf/cudf/io/parquet.py b/python/cudf/cudf/io/parquet.py index bcc9aacd2a7..f2b174bc8ff 100644 --- a/python/cudf/cudf/io/parquet.py +++ b/python/cudf/cudf/io/parquet.py @@ -1114,7 +1114,7 @@ def _parquet_to_frame( codes = as_unsigned_codes( len(partition_categories[name]), codes ) - dfs[-1][name] = CategoricalColumn( + col = CategoricalColumn( data=None, size=codes.size, dtype=cudf.CategoricalDtype( @@ -1126,22 +1126,13 @@ def _parquet_to_frame( else: # Not building categorical columns, so # `value` is already what we want - _dtype = ( - partition_meta[name].dtype - if partition_meta is not None - else None - ) if pd.isna(value): - dfs[-1][name] = column_empty( - row_count=_len, - dtype=_dtype, - ) + col = column_empty(row_count=_len) else: - dfs[-1][name] = as_column( - value, - dtype=_dtype, - length=_len, - ) + col = as_column(value, length=_len) + if partition_meta is not None: + col = col.astype(partition_meta[name].dtype) + dfs[-1][name] = col if len(dfs) > 1: # Concatenate dfs and return. diff --git a/python/cudf/cudf/tests/test_apply_rows.py b/python/cudf/cudf/tests/test_apply_rows.py index f9b0d9c1e78..b250435f68d 100644 --- a/python/cudf/cudf/tests/test_apply_rows.py +++ b/python/cudf/cudf/tests/test_apply_rows.py @@ -1,5 +1,5 @@ -# Copyright (c) 2019-2024, NVIDIA CORPORATION. - +# Copyright (c) 2019-2025, NVIDIA CORPORATION. +import numpy as np import pytest import cudf @@ -13,7 +13,7 @@ def _kernel_multiply(a, b, out): out[i] = x * y -@pytest.mark.parametrize("dtype", ["float32", "float64"]) +@pytest.mark.parametrize("dtype", [np.dtype("float32"), np.dtype("float64")]) @pytest.mark.parametrize("has_nulls", [False, True]) @pytest.mark.parametrize("pessimistic", [False, True]) def test_dataframe_apply_rows(dtype, has_nulls, pessimistic): diff --git a/python/cudf/cudf/tests/test_column.py b/python/cudf/cudf/tests/test_column.py index c3c9a1c5338..2996a88c171 100644 --- a/python/cudf/cudf/tests/test_column.py +++ b/python/cudf/cudf/tests/test_column.py @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. import cupy as cp import numpy as np @@ -401,19 +401,23 @@ def test_column_view_string_slice(slc): [ ( np.array([1, 2, 3, 4, 5], dtype="uint8"), - cudf.core.column.as_column([1, 2, 3, 4, 5], dtype="uint8"), + cudf.core.column.as_column( + [1, 2, 3, 4, 5], dtype=np.dtype(np.uint8) + ), ), ( cp.array([1, 2, 3, 4, 5], dtype="uint8"), - cudf.core.column.as_column([1, 2, 3, 4, 5], dtype="uint8"), + cudf.core.column.as_column( + [1, 2, 3, 4, 5], dtype=np.dtype(np.uint8) + ), ), ( cp.array([], dtype="uint8"), - cudf.core.column.column_empty(0, dtype="uint8"), + cudf.core.column.column_empty(0, dtype=np.dtype(np.uint8)), ), ( cp.array([255], dtype="uint8"), - cudf.core.column.as_column([255], dtype="uint8"), + cudf.core.column.as_column([255], dtype=np.dtype(np.uint8)), ), ], ) diff --git a/python/cudf/cudf/tests/test_parquet.py b/python/cudf/cudf/tests/test_parquet.py index 39a47ee4ccd..4d94bc59cda 100644 --- a/python/cudf/cudf/tests/test_parquet.py +++ b/python/cudf/cudf/tests/test_parquet.py @@ -4166,11 +4166,15 @@ def test_parquet_reader_with_mismatched_schemas_error(): def test_parquet_roundtrip_zero_rows_no_column_mask(): expected = cudf.DataFrame._from_data( { - "int": cudf.core.column.column_empty(0, "int64"), - "float": cudf.core.column.column_empty(0, "float64"), - "datetime": cudf.core.column.column_empty(0, "datetime64[ns]"), - "timedelta": cudf.core.column.column_empty(0, "timedelta64[ns]"), - "bool": cudf.core.column.column_empty(0, "bool"), + "int": cudf.core.column.column_empty(0, np.dtype(np.int64)), + "float": cudf.core.column.column_empty(0, np.dtype(np.float64)), + "datetime": cudf.core.column.column_empty( + 0, np.dtype("datetime64[ns]") + ), + "timedelta": cudf.core.column.column_empty( + 0, np.dtype("timedelta64[ns]") + ), + "bool": cudf.core.column.column_empty(0, np.dtype(np.bool_)), "decimal": cudf.core.column.column_empty( 0, cudf.Decimal64Dtype(1) ), diff --git a/python/cudf/cudf/tests/test_testing.py b/python/cudf/cudf/tests/test_testing.py index 87734ebed58..5b50bbe9060 100644 --- a/python/cudf/cudf/tests/test_testing.py +++ b/python/cudf/cudf/tests/test_testing.py @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. import numpy as np import pandas as pd @@ -429,8 +429,8 @@ def test_assert_column_memory_basic_same(arrow_arrays): data = cudf.core.column.ColumnBase.from_arrow(arrow_arrays) buf = cudf.core.buffer.as_buffer(data.base_data) - left = cudf.core.column.build_column(buf, dtype=np.int8) - right = cudf.core.column.build_column(buf, dtype=np.int8) + left = cudf.core.column.build_column(buf, dtype=np.dtype(np.int8)) + right = cudf.core.column.build_column(buf, dtype=np.dtype(np.int8)) assert_column_memory_eq(left, right) with pytest.raises(AssertionError): diff --git a/python/cudf/cudf/utils/applyutils.py b/python/cudf/cudf/utils/applyutils.py index 4d6f4ea73a8..0f40da60949 100644 --- a/python/cudf/cudf/utils/applyutils.py +++ b/python/cudf/cudf/utils/applyutils.py @@ -1,10 +1,11 @@ -# Copyright (c) 2018-2024, NVIDIA CORPORATION. +# Copyright (c) 2018-2025, NVIDIA CORPORATION. from __future__ import annotations import functools from typing import Any import cupy as cp +import numpy as np from numba import cuda from numba.core.utils import pysignature @@ -159,7 +160,7 @@ def run(self, df, **launch_params): outputs = {} for k, dt in self.outcols.items(): outputs[k] = column.column_empty( - len(df), dt, False + len(df), np.dtype(dt), False ).data_array_view(mode="write") # Bind argument args = {} diff --git a/python/cudf/cudf/utils/queryutils.py b/python/cudf/cudf/utils/queryutils.py index c20b0e62d35..9cd8d070ab3 100644 --- a/python/cudf/cudf/utils/queryutils.py +++ b/python/cudf/cudf/utils/queryutils.py @@ -245,7 +245,7 @@ def query_execute(df, expr, callenv): # allocate output buffer nrows = len(df) - out = column_empty(nrows, dtype=np.bool_, for_numba=True) + out = column_empty(nrows, dtype=np.dtype(np.bool_), for_numba=True) # run kernel args = [out, *colarrays, *envargs] with _CUDFNumbaConfig(): From 7914858882d32d2fff97fa536befdfb986fa395e Mon Sep 17 00:00:00 2001 From: Basit Ayantunde Date: Thu, 13 Feb 2025 13:59:28 +0000 Subject: [PATCH 030/129] Added Multi-input & Scalar Support for Transform UDFs (#17881) This merge request implements multi-input and scalar support for UDF transforms. Authors: - Basit Ayantunde (https://github.com/lamarrr) Approvers: - David Wendt (https://github.com/davidwendt) - Vyas Ramasubramani (https://github.com/vyasr) - Matthew Murray (https://github.com/Matt711) - Nghia Truong (https://github.com/ttnghia) URL: https://github.com/rapidsai/cudf/pull/17881 --- cpp/include/cudf/transform.hpp | 27 +-- cpp/src/transform/jit/kernel.cu | 20 ++- cpp/src/transform/transform.cpp | 167 ++++++++++++++---- cpp/tests/streams/transform_test.cpp | 9 +- .../integration/unary_transform_test.cpp | 119 ++++++++++++- java/src/main/native/src/ColumnViewJni.cpp | 4 +- python/cudf/cudf/core/column/numerical.py | 2 +- .../pylibcudf/pylibcudf/libcudf/transform.pxd | 7 +- .../pylibcudf/tests/test_transform.py | 34 +++- python/pylibcudf/pylibcudf/transform.pxd | 7 +- python/pylibcudf/pylibcudf/transform.pyi | 5 +- python/pylibcudf/pylibcudf/transform.pyx | 29 +-- 12 files changed, 355 insertions(+), 75 deletions(-) diff --git a/cpp/include/cudf/transform.hpp b/cpp/include/cudf/transform.hpp index 82b8bee1acf..f03446fba05 100644 --- a/cpp/include/cudf/transform.hpp +++ b/cpp/include/cudf/transform.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -32,27 +32,32 @@ namespace CUDF_EXPORT cudf { */ /** - * @brief Creates a new column by applying a unary function against every - * element of an input column. + * @brief Creates a new column by applying a transform function against every + * element of the input columns. * * Computes: - * `out[i] = F(in[i])` + * `out[i] = F(inputs[i]...)`. * - * The output null mask is the same is the input null mask so if input[i] is - * null then output[i] is also null + * Note that for every scalar in `inputs` (columns of size 1), `input[i] == input[0]` * - * @param input An immutable view of the input column to transform - * @param unary_udf The PTX/CUDA string of the unary function to apply + * The output null mask is the same as the null mask of the input columns, so if input[i] is + * null then output[i] is also null. The size of the resulting column is the size of the largest + * column. + * All input columns must have equivalent null masks. + * + * + * @param inputs Immutable views of the input columns to transform + * @param transform_udf The PTX/CUDA string of the transform function to apply * @param output_type The output type that is compatible with the output type in the UDF * @param is_ptx true: the UDF is treated as PTX code; false: the UDF is treated as CUDA code * @param stream CUDA stream used for device memory operations and kernel launches * @param mr Device memory resource used to allocate the returned column's device memory - * @return The column resulting from applying the unary function to + * @return The column resulting from applying the transform function to * every element of the input */ std::unique_ptr transform( - column_view const& input, - std::string const& unary_udf, + std::vector const& inputs, + std::string const& transform_udf, data_type output_type, bool is_ptx, rmm::cuda_stream_view stream = cudf::get_default_stream(), diff --git a/cpp/src/transform/jit/kernel.cu b/cpp/src/transform/jit/kernel.cu index 9d96c11c3f2..80cf5121bed 100644 --- a/cpp/src/transform/jit/kernel.cu +++ b/cpp/src/transform/jit/kernel.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -33,8 +33,20 @@ namespace cudf { namespace transformation { namespace jit { -template -CUDF_KERNEL void kernel(cudf::size_type size, TypeOut* out_data, TypeIn* in_data) +/// @brief This class supports striding into columns of data as either scalars or actual +/// columns at no runtime cost. Although it implies the kernel will be recompiled if scalar and +/// column inputs are interchanged. +template +struct strided { + T data; + + __device__ T const& get(int64_t index) const { return (&data)[index * multiplier]; } + + __device__ T& get(int64_t index) { return (&data)[index * multiplier]; } +}; + +template +CUDF_KERNEL void kernel(cudf::size_type size, Out* __restrict__ out, In const* __restrict__... ins) { // cannot use global_thread_id utility due to a JIT build issue by including // the `cudf/detail/utilities/cuda.cuh` header @@ -43,7 +55,7 @@ CUDF_KERNEL void kernel(cudf::size_type size, TypeOut* out_data, TypeIn* in_data thread_index_type const stride = block_size * gridDim.x; for (auto i = start; i < static_cast(size); i += stride) { - GENERIC_UNARY_OP(&out_data[i], in_data[i]); + GENERIC_TRANSFORM_OP(&out->get(i), ins->get(i)...); } } diff --git a/cpp/src/transform/transform.cpp b/cpp/src/transform/transform.cpp index 7b42bb76a50..b457ce5a676 100644 --- a/cpp/src/transform/transform.cpp +++ b/cpp/src/transform/transform.cpp @@ -34,35 +34,105 @@ namespace cudf { namespace transformation { namespace jit { namespace { -void unary_operation(mutable_column_view output, - column_view input, - std::string const& udf, - data_type output_type, - bool is_ptx, - rmm::cuda_stream_view stream) + +using device_data_t = void*; + +std::vector build_jit_typenames(mutable_column_view output, + std::vector const& inputs) +{ + static constexpr auto SCALAR_STRIDE = 0; + static constexpr auto COLUMN_STRIDE = 1; + + auto const column_type_name = [](data_type data_type, bool is_scalar) { + return jitify2::reflection::Template("cudf::transformation::jit::strided") + .instantiate(type_to_name(data_type), is_scalar ? SCALAR_STRIDE : COLUMN_STRIDE); + }; + + std::vector typenames; + + typenames.push_back(column_type_name(output.type(), false)); + std::transform( + inputs.begin(), inputs.end(), std::back_inserter(typenames), [&](auto const& input) { + bool const is_scalar = input.size() != output.size(); + return column_type_name(input.type(), is_scalar); + }); + + return typenames; +} + +std::map build_ptx_params(mutable_column_view output, + std::vector const& inputs) +{ + std::map params; + uint32_t index = 0; + + auto const add_column = [&](bool is_output, data_type type) { + auto const param_type = type_to_name(type); + params.emplace(index++, is_output ? (param_type + "*") : param_type); + }; + + add_column(true, output.type()); + + for (auto& input : inputs) { + add_column(false, input.type()); + } + + return params; +} + +std::vector build_device_data(mutable_column_view output, + std::vector const& inputs) +{ + std::vector data; + + data.push_back(const_cast(cudf::jit::get_data_ptr(output))); + + std::transform(inputs.begin(), inputs.end(), std::back_inserter(data), [](auto const& input) { + return const_cast(cudf::jit::get_data_ptr(input)); + }); + + return data; +} + +std::vector build_launch_args(cudf::size_type& size, std::vector& device_data) { - std::string const kernel_name = - jitify2::reflection::Template("cudf::transformation::jit::kernel") // - .instantiate(cudf::type_to_name(output.type()), // list of template arguments - cudf::type_to_name(input.type())); - - std::string cuda_source = is_ptx ? cudf::jit::parse_single_function_ptx( - udf, // - "GENERIC_UNARY_OP", - { - {0, "void *"}, // output argument - {1, cudf::type_to_name(input.type())} // input argument - }) - : cudf::jit::parse_single_function_cuda(udf, // - "GENERIC_UNARY_OP"); + // JITIFY and NVRTC need non-const pointers even if they aren't written to + std::vector args; + args.push_back(&size); + std::transform( + device_data.begin(), device_data.end(), std::back_inserter(args), [](auto& data) -> void* { + return &data; + }); + + return args; +} + +void transform_operation(size_type base_column_size, + mutable_column_view output, + std::vector const& inputs, + std::string const& udf, + data_type output_type, + bool is_ptx, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) +{ + std::string const kernel_name = jitify2::reflection::Template("cudf::transformation::jit::kernel") + .instantiate(build_jit_typenames(output, inputs)); + + std::string const cuda_source = + is_ptx ? cudf::jit::parse_single_function_ptx( + udf, "GENERIC_TRANSFORM_OP", build_ptx_params(output, inputs)) + : cudf::jit::parse_single_function_cuda(udf, "GENERIC_TRANSFORM_OP"); + + auto device_data = build_device_data(output, inputs); + + auto args = build_launch_args(base_column_size, device_data); cudf::jit::get_program_cache(*transform_jit_kernel_cu_jit) .get_kernel( kernel_name, {}, {{"transform/jit/operation-udf.hpp", cuda_source}}, {"-arch=sm_."}) // ->configure_1d_max_occupancy(0, 0, nullptr, stream.value()) // - ->launch(output.size(), // - cudf::jit::get_data_ptr(output), - cudf::jit::get_data_ptr(input)); + ->launch(args.data()); } } // namespace @@ -70,39 +140,66 @@ void unary_operation(mutable_column_view output, } // namespace transformation namespace detail { -std::unique_ptr transform(column_view const& input, - std::string const& unary_udf, +std::unique_ptr transform(std::vector const& inputs, + std::string const& transform_udf, data_type output_type, bool is_ptx, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) { - CUDF_EXPECTS(is_fixed_width(input.type()), "Unexpected non-fixed-width type."); - - std::unique_ptr output = make_fixed_width_column( - output_type, input.size(), copy_bitmask(input, stream, mr), input.null_count(), stream, mr); - - if (input.is_empty()) { return output; } + CUDF_EXPECTS(is_fixed_width(output_type), "Transforms only support fixed-width types"); + CUDF_EXPECTS( + std::all_of( + inputs.begin(), inputs.end(), [](auto& input) { return is_fixed_width(input.type()); }), + "Transforms only support fixed-width types"); + + auto const base_column = std::max_element( + inputs.begin(), inputs.end(), [](auto& a, auto& b) { return a.size() < b.size(); }); + + CUDF_EXPECTS(std::all_of(inputs.begin(), + inputs.end(), + [&](auto const& input) { + return (input.size() == 1) || (input.size() == base_column->size()); + }), + "All transform input columns must have the same size or be scalar (have size 1)"); + + CUDF_EXPECTS(std::all_of(inputs.begin(), + inputs.end(), + [&](auto const& input) { + return (input.size() == 1 && input.null_count() == 0) || + (input.null_count() == base_column->null_count()); + }), + "All transform input columns must have the same null-count"); + + auto output = make_fixed_width_column(output_type, + base_column->size(), + copy_bitmask(*base_column, stream, mr), + base_column->null_count(), + stream, + mr); + + if (base_column->is_empty()) { return output; } mutable_column_view const output_view = *output; // transform - transformation::jit::unary_operation(output_view, input, unary_udf, output_type, is_ptx, stream); + transformation::jit::transform_operation( + base_column->size(), output_view, inputs, transform_udf, output_type, is_ptx, stream, mr); return output; } } // namespace detail -std::unique_ptr transform(column_view const& input, - std::string const& unary_udf, +std::unique_ptr transform(std::vector const& inputs, + std::string const& transform_udf, data_type output_type, bool is_ptx, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) { CUDF_FUNC_RANGE(); - return detail::transform(input, unary_udf, output_type, is_ptx, stream, mr); + return detail::transform(inputs, transform_udf, output_type, is_ptx, stream, mr); } } // namespace cudf diff --git a/cpp/tests/streams/transform_test.cpp b/cpp/tests/streams/transform_test.cpp index 9f168abcb31..725a46e045f 100644 --- a/cpp/tests/streams/transform_test.cpp +++ b/cpp/tests/streams/transform_test.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -32,8 +32,11 @@ void test_udf(char const* udf, Data data_init, cudf::size_type size, bool is_ptx auto data_iter = cudf::detail::make_counting_transform_iterator(0, data_init); cudf::test::fixed_width_column_wrapper in( data_iter, data_iter + size, all_valid); - cudf::transform( - in, udf, cudf::data_type(cudf::type_to_id()), is_ptx, cudf::test::get_default_stream()); + cudf::transform({in}, + udf, + cudf::data_type(cudf::type_to_id()), + is_ptx, + cudf::test::get_default_stream()); } TEST_F(TransformTest, Transform) diff --git a/cpp/tests/transform/integration/unary_transform_test.cpp b/cpp/tests/transform/integration/unary_transform_test.cpp index 0bdf5b321ac..f0986bef9ae 100644 --- a/cpp/tests/transform/integration/unary_transform_test.cpp +++ b/cpp/tests/transform/integration/unary_transform_test.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Copyright 2018-2019 BlazingDB, Inc. * Copyright 2018 Christian Noboa Mardini @@ -39,7 +39,7 @@ void test_udf(char const* udf, Op op, Data data_init, cudf::size_type size, bool data_iter, data_iter + size, all_valid); std::unique_ptr out = - cudf::transform(in, udf, cudf::data_type(cudf::type_to_id()), is_ptx); + cudf::transform({in}, udf, cudf::data_type(cudf::type_to_id()), is_ptx); ASSERT_UNARY(out->view(), in, op); } @@ -220,4 +220,119 @@ __device__ inline void f(cudf::timestamp_us* output, cudf::timestamp_us input) test_udf(cuda.c_str(), op, data_init, 500, false); } +struct TenaryOperationTest : public cudf::test::BaseFixture {}; + +TEST_F(TenaryOperationTest, TransformWithScalar) +{ + std::string const cuda = + R"***( +__device__ inline void transform( + float* out, + float a, + float b, + float c +) +{ + *out = (a + b) * c; +} +)***"; + + // Generated from NUMBA, using: + // + // ```py + // + // from numba import cuda, float32 + // from numba.cuda import compile_ptx_for_current_device + // + // # Define a CUDA device function + // + // @cuda.jit(device=True) + // def op(a, b, c): + // return (a + b) * c + // + // # Define argument types for the function + // arg_types = (float32, float32, float32) + // + // # Compile the device function as relocatable + // ptx, _ = cuda.compile_ptx_for_current_device(op, arg_types, device=True) + // + // + // # Print the PTX code + // print("Relocatable PTX Code:") + // print(ptx) + // + // + // ``` + // + std::string const ptx = + R"***( +// +// Generated by NVIDIA NVVM Compiler +// +// Compiler Build ID: CL-35404655 +// Cuda compilation tools, release 12.8, V12.8.61 +// Based on NVVM 7.0.1 +// + +.version 8.7 +.target sm_86 +.address_size 64 + + // .globl _ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff +.common .global .align 8 .u64 _ZN08NumbaEnv8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff; + +.visible .func (.param .b32 func_retval0) _ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff( + .param .b64 _ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff_param_0, + .param .b32 _ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff_param_1, + .param .b32 _ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff_param_2, + .param .b32 _ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff_param_3 +) +{ + .reg .f32 %f<6>; + .reg .b32 %r<2>; + .reg .b64 %rd<2>; + + + ld.param.u64 %rd1, [_ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff_param_0]; + ld.param.f32 %f1, [_ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff_param_1]; + ld.param.f32 %f2, [_ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff_param_2]; + ld.param.f32 %f3, [_ZN8__main__2opB2v1B96cw51cXTLSUwv1sCUt9Ww0FEw09RRQPKiLTj0gIGIFp_2b2oLQFEYYkHSQB1OQAk0Bynm21OizQ1K0UoIGvDpQE8oxrNQE_3dEfff_param_3]; + add.f32 %f4, %f1, %f2; + mul.f32 %f5, %f4, %f3; + st.f32 [%rd1], %f5; + mov.u32 %r1, 0; + st.param.b32 [func_retval0+0], %r1; + ret; + +} +)***"; + + using T = float; + + constexpr T A = 90; + constexpr T B = 100; + constexpr T C = 5; + constexpr T OUT = (A + B) * C; + + std::vector a_host(200, A); + std::vector b_host(200, B); + std::vector c_host(1, C); + std::vector expected_host(200, OUT); + + cudf::test::fixed_width_column_wrapper a(a_host.begin(), a_host.end()); + cudf::test::fixed_width_column_wrapper b(b_host.begin(), b_host.end()); + cudf::test::fixed_width_column_wrapper c(c_host.begin(), c_host.end()); + cudf::test::fixed_width_column_wrapper expected(expected_host.begin(), expected_host.end()); + + std::unique_ptr cuda_result = + cudf::transform({a, b, c}, cuda, cudf::data_type(cudf::type_to_id()), false); + + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*cuda_result, expected); + + std::unique_ptr ptx_result = + cudf::transform({a, b, c}, ptx, cudf::data_type(cudf::type_to_id()), true); + + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*ptx_result, expected); +} + } // namespace transformation diff --git a/java/src/main/native/src/ColumnViewJni.cpp b/java/src/main/native/src/ColumnViewJni.cpp index 6a59ae3ddd5..be121e8b247 100644 --- a/java/src/main/native/src/ColumnViewJni.cpp +++ b/java/src/main/native/src/ColumnViewJni.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -1503,7 +1503,7 @@ JNIEXPORT jlong JNICALL Java_ai_rapids_cudf_ColumnView_transform( cudf::jni::native_jstring n_j_udf(env, j_udf); std::string n_udf(n_j_udf.get()); return release_as_jlong( - cudf::transform(*column, n_udf, cudf::data_type(cudf::type_id::INT32), j_is_ptx)); + cudf::transform({*column}, n_udf, cudf::data_type(cudf::type_id::INT32), j_is_ptx)); } CATCH_STD(env, 0); } diff --git a/python/cudf/cudf/core/column/numerical.py b/python/cudf/cudf/core/column/numerical.py index bb336a9192e..1abd55b110d 100644 --- a/python/cudf/cudf/core/column/numerical.py +++ b/python/cudf/cudf/core/column/numerical.py @@ -182,7 +182,7 @@ def __setitem__(self, key: Any, value: Any): @acquire_spill_lock() def transform(self, compiled_op, np_dtype: np.dtype) -> ColumnBase: plc_column = plc.transform.transform( - self.to_pylibcudf(mode="read"), + [self.to_pylibcudf(mode="read")], compiled_op[0], plc.column._datatype_from_dtype_desc(np_dtype.str[1:]), True, diff --git a/python/pylibcudf/pylibcudf/libcudf/transform.pxd b/python/pylibcudf/pylibcudf/libcudf/transform.pxd index 78ee7b4b0e5..9be137a077e 100644 --- a/python/pylibcudf/pylibcudf/libcudf/transform.pxd +++ b/python/pylibcudf/pylibcudf/libcudf/transform.pxd @@ -1,8 +1,9 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from libcpp cimport bool from libcpp.memory cimport unique_ptr from libcpp.pair cimport pair from libcpp.string cimport string +from libcpp.vector cimport vector from pylibcudf.exception_handler cimport libcudf_exception_handler from pylibcudf.libcudf.column.column cimport column from pylibcudf.libcudf.column.column_view cimport column_view @@ -33,8 +34,8 @@ cdef extern from "cudf/transform.hpp" namespace "cudf" nogil: ) except +libcudf_exception_handler cdef unique_ptr[column] transform( - column_view input, - string unary_udf, + const vector[column_view] & inputs, + const string & transform_udf, data_type output_type, bool is_ptx ) except +libcudf_exception_handler diff --git a/python/pylibcudf/pylibcudf/tests/test_transform.py b/python/pylibcudf/pylibcudf/tests/test_transform.py index 49802fe64ac..26d970d477e 100644 --- a/python/pylibcudf/pylibcudf/tests/test_transform.py +++ b/python/pylibcudf/pylibcudf/tests/test_transform.py @@ -1,8 +1,10 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. import math +import numba import pyarrow as pa +from numba import cuda from utils import assert_column_eq import pylibcudf as plc @@ -81,3 +83,33 @@ def test_one_hot_encode(): schema=pa.schema([pa.field("", pa.bool_(), nullable=False)] * 3), ) assert result.equals(expected) + + +def test_transform_udf(): + @cuda.jit(device=True) + def op(a, b, c): + return (a + b) * c + + ptx, _ = cuda.compile_ptx_for_current_device( + op, (numba.float64, numba.float64, numba.float64), device=True + ) + + A = 5.0 + B = 20.0 + C = 0.5 + + a = pa.array([A] * 100) + b = pa.array([B] * 100) + c = pa.array([C]) + expected = pa.array([(A + B) * C] * 100) + result = plc.transform.transform( + [ + plc.interop.from_arrow(a), + plc.interop.from_arrow(b), + plc.interop.from_arrow(c), + ], + transform_udf=ptx, + output_type=plc.DataType(plc.TypeId.FLOAT64), + is_ptx=True, + ) + assert_column_eq(expected, result) diff --git a/python/pylibcudf/pylibcudf/transform.pxd b/python/pylibcudf/pylibcudf/transform.pxd index 4fb623158f0..45f79158055 100644 --- a/python/pylibcudf/pylibcudf/transform.pxd +++ b/python/pylibcudf/pylibcudf/transform.pxd @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from libcpp cimport bool from pylibcudf.libcudf.types cimport bitmask_type, data_type @@ -17,7 +17,10 @@ cpdef tuple[gpumemoryview, int] bools_to_mask(Column input) cpdef Column mask_to_bools(Py_ssize_t bitmask, int begin_bit, int end_bit) -cpdef Column transform(Column input, str unary_udf, DataType output_type, bool is_ptx) +cpdef Column transform(list[Column] inputs, + str transform_udf, + DataType output_type, + bool is_ptx) cpdef tuple[Table, Column] encode(Table input) diff --git a/python/pylibcudf/pylibcudf/transform.pyi b/python/pylibcudf/pylibcudf/transform.pyi index 5cbd2e635f0..ff7c43115bd 100644 --- a/python/pylibcudf/pylibcudf/transform.pyi +++ b/python/pylibcudf/pylibcudf/transform.pyi @@ -10,7 +10,10 @@ def compute_column(input: Table, expr: Expression) -> Column: ... def bools_to_mask(input: Column) -> tuple[gpumemoryview, int]: ... def mask_to_bools(bitmask: int, begin_bit: int, end_bit: int) -> Column: ... def transform( - input: Column, unary_udf: str, output_type: DataType, is_ptx: bool + inputs: list[Column], + transform_udf: str, + output_type: DataType, + is_ptx: bool, ) -> Column: ... def encode(input: Table) -> tuple[Table, Column]: ... def one_hot_encode(input: Column, categories: Column) -> Table: ... diff --git a/python/pylibcudf/pylibcudf/transform.pyx b/python/pylibcudf/pylibcudf/transform.pyx index 9700bcff221..bd841c1382f 100644 --- a/python/pylibcudf/pylibcudf/transform.pyx +++ b/python/pylibcudf/pylibcudf/transform.pyx @@ -1,11 +1,13 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from cython.operator cimport dereference from libcpp.memory cimport unique_ptr from libcpp.string cimport string +from libcpp.vector cimport vector from libcpp.utility cimport move, pair from pylibcudf.libcudf cimport transform as cpp_transform from pylibcudf.libcudf.column.column cimport column +from pylibcudf.libcudf.column.column_view cimport column_view from pylibcudf.libcudf.table.table cimport table from pylibcudf.libcudf.table.table_view cimport table_view from pylibcudf.libcudf.types cimport bitmask_type, size_type @@ -129,16 +131,19 @@ cpdef Column mask_to_bools(Py_ssize_t bitmask, int begin_bit, int end_bit): return Column.from_libcudf(move(c_result)) -cpdef Column transform(Column input, str unary_udf, DataType output_type, bool is_ptx): - """Create a new column by applying a unary function against every - element of an input column. +cpdef Column transform(list[Column] inputs, + str transform_udf, + DataType output_type, + bool is_ptx): + """Create a new column by applying a transform function against + multiple input columns. Parameters ---------- - input : Column - Column to transform. - unary_udf : str - The PTX/CUDA string of the unary function to apply. + inputs : list[Column] + Columns to transform. + transform_udf : str + The PTX/CUDA string of the transform function to apply. output_type : DataType The output type that is compatible with the output type in the unary_udf. is_ptx : bool @@ -150,13 +155,17 @@ cpdef Column transform(Column input, str unary_udf, DataType output_type, bool i Column The transformed column having the UDF applied to each element. """ + cdef vector[column_view] c_inputs cdef unique_ptr[column] c_result - cdef string c_unary_udf = unary_udf.encode() + cdef string c_transform_udf = transform_udf.encode() cdef bool c_is_ptx = is_ptx + for input in inputs: + c_inputs.push_back((input).view()) + with nogil: c_result = cpp_transform.transform( - input.view(), c_unary_udf, output_type.c_obj, c_is_ptx + c_inputs, c_transform_udf, output_type.c_obj, c_is_ptx ) return Column.from_libcudf(move(c_result)) From c86ff6eae046c56af8be483263e1faf5f66ce714 Mon Sep 17 00:00:00 2001 From: Jake Awe Date: Thu, 13 Feb 2025 09:41:01 -0600 Subject: [PATCH 031/129] Update Changelog [skip ci] --- CHANGELOG.md | 319 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 319 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a75b2a95a4..b1c6a94a17f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,322 @@ +# cudf 25.02.00 (13 Feb 2025) + +## 🚨 Breaking Changes + +- Expose stream-ordering in scalar and avro APIs ([#17766](https://github.com/rapidsai/cudf/pull/17766)) [@shrshi](https://github.com/shrshi) +- Add seed parameter to hash_character_ngrams ([#17643](https://github.com/rapidsai/cudf/pull/17643)) [@davidwendt](https://github.com/davidwendt) +- Performance improvements and simplifications for fixed size row-based rolling windows ([#17623](https://github.com/rapidsai/cudf/pull/17623)) [@wence-](https://github.com/wence-) +- Refactor distinct hash join to handle multiple probes with the same build table ([#17609](https://github.com/rapidsai/cudf/pull/17609)) [@PointKernel](https://github.com/PointKernel) +- Deprecate cudf::grouped_time_range_rolling_window ([#17589](https://github.com/rapidsai/cudf/pull/17589)) [@wence-](https://github.com/wence-) +- Remove "legacy" Dask DataFrame support from Dask cuDF ([#17558](https://github.com/rapidsai/cudf/pull/17558)) [@rjzamora](https://github.com/rjzamora) +- Return empty result for segmented_reduce if input and offsets are both empty ([#17437](https://github.com/rapidsai/cudf/pull/17437)) [@davidwendt](https://github.com/davidwendt) +- Rework minhash APIs for deprecation cycle ([#17421](https://github.com/rapidsai/cudf/pull/17421)) [@davidwendt](https://github.com/davidwendt) +- Change indices for dictionary column to signed integer type ([#17390](https://github.com/rapidsai/cudf/pull/17390)) [@davidwendt](https://github.com/davidwendt) + +## 🐛 Bug Fixes + +- Fix race check failures in shared memory groupby ([#17985](https://github.com/rapidsai/cudf/pull/17985)) [@PointKernel](https://github.com/PointKernel) +- Pin `ibis` version in the cudf.pandas integration tests <10.0.0 ([#17975](https://github.com/rapidsai/cudf/pull/17975)) [@Matt711](https://github.com/Matt711) +- Fix the index type in the indexing operator of the span types ([#17971](https://github.com/rapidsai/cudf/pull/17971)) [@vuule](https://github.com/vuule) +- Add missing pin ([#17915](https://github.com/rapidsai/cudf/pull/17915)) [@vyasr](https://github.com/vyasr) +- Fix third-party `cudf.pandas` tests ([#17900](https://github.com/rapidsai/cudf/pull/17900)) [@galipremsagar](https://github.com/galipremsagar) +- Fix `numpy` data access by making attribute private ([#17890](https://github.com/rapidsai/cudf/pull/17890)) [@galipremsagar](https://github.com/galipremsagar) +- Remove extra local var declaration from cudf.pandas 3rd-party integration shell script ([#17886](https://github.com/rapidsai/cudf/pull/17886)) [@Matt711](https://github.com/Matt711) +- Move `isinstance_cudf_pandas` to `fast_slow_proxy` ([#17875](https://github.com/rapidsai/cudf/pull/17875)) [@galipremsagar](https://github.com/galipremsagar) +- Make `_Series_dtype` method a property ([#17854](https://github.com/rapidsai/cudf/pull/17854)) [@Matt711](https://github.com/Matt711) +- Fix the bug in determining the heuristics for shared memory groupby ([#17851](https://github.com/rapidsai/cudf/pull/17851)) [@PointKernel](https://github.com/PointKernel) +- Fix possible OOB mem access in Parquet decoder ([#17841](https://github.com/rapidsai/cudf/pull/17841)) [@mhaseeb123](https://github.com/mhaseeb123) +- Require batches to be non-empty in multi-batch JSON reader ([#17837](https://github.com/rapidsai/cudf/pull/17837)) [@shrshi](https://github.com/shrshi) +- Fix rolling(min_periods=) with int and null data with mode.pandas_compat ([#17822](https://github.com/rapidsai/cudf/pull/17822)) [@mroeschke](https://github.com/mroeschke) +- Resolve race-condition in `disable_module_accelerator` ([#17811](https://github.com/rapidsai/cudf/pull/17811)) [@galipremsagar](https://github.com/galipremsagar) +- Make Series(dtype=object) raise in mode.pandas_compat with non string data ([#17804](https://github.com/rapidsai/cudf/pull/17804)) [@mroeschke](https://github.com/mroeschke) +- Disable intended disabled ORC tests ([#17790](https://github.com/rapidsai/cudf/pull/17790)) [@davidwendt](https://github.com/davidwendt) +- Fix empty DataFrame construction not returning RangeIndex columns ([#17784](https://github.com/rapidsai/cudf/pull/17784)) [@mroeschke](https://github.com/mroeschke) +- Fix various `.str` methods for pandas compatability ([#17782](https://github.com/rapidsai/cudf/pull/17782)) [@mroeschke](https://github.com/mroeschke) +- Fix `count` API issue about ignoring nan values ([#17779](https://github.com/rapidsai/cudf/pull/17779)) [@galipremsagar](https://github.com/galipremsagar) +- Add `numba` pinning to `cudf` repo ([#17777](https://github.com/rapidsai/cudf/pull/17777)) [@galipremsagar](https://github.com/galipremsagar) +- Allow .sort_values(na_position=) to include NaNs in mode.pandas_compatible ([#17776](https://github.com/rapidsai/cudf/pull/17776)) [@mroeschke](https://github.com/mroeschke) +- allow deselecting nvcomp wheels ([#17774](https://github.com/rapidsai/cudf/pull/17774)) [@jameslamb](https://github.com/jameslamb) +- Use the `aligned_resource_adaptor` to allocate bloom filter device buffers ([#17758](https://github.com/rapidsai/cudf/pull/17758)) [@mhaseeb123](https://github.com/mhaseeb123) +- Avoid instantiating bloom filter query function for nested and bool types ([#17753](https://github.com/rapidsai/cudf/pull/17753)) [@mhaseeb123](https://github.com/mhaseeb123) +- Fix DataFrame.merge(Series, how="left"/"right") on column and index not resulting in a RangeIndex ([#17739](https://github.com/rapidsai/cudf/pull/17739)) [@mroeschke](https://github.com/mroeschke) +- [BUG] xfail Polars excel test ([#17731](https://github.com/rapidsai/cudf/pull/17731)) [@Matt711](https://github.com/Matt711) +- Require to implement `AutoCloseable` for the classes derived from `HostUDFWrapper` ([#17727](https://github.com/rapidsai/cudf/pull/17727)) [@ttnghia](https://github.com/ttnghia) +- Remove jlowe as a java committer since he retired ([#17725](https://github.com/rapidsai/cudf/pull/17725)) [@tgravescs](https://github.com/tgravescs) +- Prevent use of invalid grid sizes in ORC reader and writer ([#17709](https://github.com/rapidsai/cudf/pull/17709)) [@vuule](https://github.com/vuule) +- Enforce schema for partial tables in multi-source multi-batch JSON reader ([#17708](https://github.com/rapidsai/cudf/pull/17708)) [@shrshi](https://github.com/shrshi) +- Compute and use the initial string offset when building `nested` large string cols with chunked parquet reader ([#17702](https://github.com/rapidsai/cudf/pull/17702)) [@mhaseeb123](https://github.com/mhaseeb123) +- Fix writing of compressed ORC files with large stripe footers ([#17700](https://github.com/rapidsai/cudf/pull/17700)) [@vuule](https://github.com/vuule) +- Fix cudf.polars sum of empty not equalling zero ([#17685](https://github.com/rapidsai/cudf/pull/17685)) [@mroeschke](https://github.com/mroeschke) +- Fix formatting in logging ([#17680](https://github.com/rapidsai/cudf/pull/17680)) [@vuule](https://github.com/vuule) +- convert all nulls to nans in a specific scenario ([#17677](https://github.com/rapidsai/cudf/pull/17677)) [@galipremsagar](https://github.com/galipremsagar) +- Define cudf repr methods on the Column ([#17675](https://github.com/rapidsai/cudf/pull/17675)) [@mroeschke](https://github.com/mroeschke) +- Fix groupby.len with null values in cudf.polars ([#17671](https://github.com/rapidsai/cudf/pull/17671)) [@mroeschke](https://github.com/mroeschke) +- Fix: DataFrameGroupBy.get_group was raising with length>1 tuples ([#17653](https://github.com/rapidsai/cudf/pull/17653)) [@MarcoGorelli](https://github.com/MarcoGorelli) +- Fix possible int overflow in compute_mixed_join_output_size ([#17633](https://github.com/rapidsai/cudf/pull/17633)) [@davidwendt](https://github.com/davidwendt) +- Fix a minor potential i32 overflow in `thrust::transform_exclusive_scan` in PQ reader preprocessing ([#17617](https://github.com/rapidsai/cudf/pull/17617)) [@mhaseeb123](https://github.com/mhaseeb123) +- Fix failing xgboost test in the cudf.pandas third-party integration tests ([#17616](https://github.com/rapidsai/cudf/pull/17616)) [@Matt711](https://github.com/Matt711) +- Fix ``dask_cudf.read_csv`` ([#17612](https://github.com/rapidsai/cudf/pull/17612)) [@rjzamora](https://github.com/rjzamora) +- Fix memcheck error in ReplaceTest.NormalizeNansAndZerosMutable gtest ([#17610](https://github.com/rapidsai/cudf/pull/17610)) [@davidwendt](https://github.com/davidwendt) +- Correctly accept a `pandas.CategoricalDtype(pandas.IntervalDtype(...), ...)` type ([#17604](https://github.com/rapidsai/cudf/pull/17604)) [@mroeschke](https://github.com/mroeschke) +- Add ability to modify and propagate `names` of `columns` object ([#17597](https://github.com/rapidsai/cudf/pull/17597)) [@galipremsagar](https://github.com/galipremsagar) +- Ignore NaN correctly in .quantile ([#17593](https://github.com/rapidsai/cudf/pull/17593)) [@mroeschke](https://github.com/mroeschke) +- Fix groupby argmin/max gather of sorted-order indices ([#17591](https://github.com/rapidsai/cudf/pull/17591)) [@davidwendt](https://github.com/davidwendt) +- Fix ctest fail running libcudf tests in a Debug build ([#17576](https://github.com/rapidsai/cudf/pull/17576)) [@davidwendt](https://github.com/davidwendt) +- Specify a version for rapids_logger dependency ([#17573](https://github.com/rapidsai/cudf/pull/17573)) [@jlowe](https://github.com/jlowe) +- Fix the ORC decoding bug for the timestamp data ([#17570](https://github.com/rapidsai/cudf/pull/17570)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- [JNI] remove rmm argument to set rw access for fabric handles ([#17553](https://github.com/rapidsai/cudf/pull/17553)) [@abellina](https://github.com/abellina) +- Document undefined behavior in div_rounding_up_safe ([#17542](https://github.com/rapidsai/cudf/pull/17542)) [@davidwendt](https://github.com/davidwendt) +- Fix nvcc-imposed UB in `constexpr` functions ([#17534](https://github.com/rapidsai/cudf/pull/17534)) [@vuule](https://github.com/vuule) +- Add anonymous namespace to libcudf test source ([#17529](https://github.com/rapidsai/cudf/pull/17529)) [@davidwendt](https://github.com/davidwendt) +- Propagate failures in pandas integration tests and Skip failing tests ([#17521](https://github.com/rapidsai/cudf/pull/17521)) [@Matt711](https://github.com/Matt711) +- Fix libcudf compile error when logging is disabled ([#17512](https://github.com/rapidsai/cudf/pull/17512)) [@davidwendt](https://github.com/davidwendt) +- Fix Dask-cuDF `clip` APIs ([#17509](https://github.com/rapidsai/cudf/pull/17509)) [@rjzamora](https://github.com/rjzamora) +- Fix pylibcudf to_arrow with multiple nested data types ([#17504](https://github.com/rapidsai/cudf/pull/17504)) [@mroeschke](https://github.com/mroeschke) +- Fix groupby(as_index=False).size not reseting index ([#17499](https://github.com/rapidsai/cudf/pull/17499)) [@mroeschke](https://github.com/mroeschke) +- Revert "Temporarily skip tests due to dask/distributed#8953" ([#17492](https://github.com/rapidsai/cudf/pull/17492)) [@Matt711](https://github.com/Matt711) +- Workaround for a misaligned access in `read_csv` on some CUDA versions ([#17477](https://github.com/rapidsai/cudf/pull/17477)) [@vuule](https://github.com/vuule) +- Fix some possible thread-id overflow calculations ([#17473](https://github.com/rapidsai/cudf/pull/17473)) [@davidwendt](https://github.com/davidwendt) +- Temporarily skip tests due to dask/distributed#8953 ([#17472](https://github.com/rapidsai/cudf/pull/17472)) [@wence-](https://github.com/wence-) +- Detect mismatches in begin and end tokens returned by JSON tokenizer FST ([#17471](https://github.com/rapidsai/cudf/pull/17471)) [@shrshi](https://github.com/shrshi) +- Support dask>=2024.11.2 in Dask cuDF ([#17439](https://github.com/rapidsai/cudf/pull/17439)) [@rjzamora](https://github.com/rjzamora) +- Fix write_json failure for zero columns in table/struct ([#17414](https://github.com/rapidsai/cudf/pull/17414)) [@karthikeyann](https://github.com/karthikeyann) +- Fix Debug-mode failing Arrow test ([#17405](https://github.com/rapidsai/cudf/pull/17405)) [@zeroshade](https://github.com/zeroshade) +- Fix all null list column with missing child column in JSON reader ([#17348](https://github.com/rapidsai/cudf/pull/17348)) [@karthikeyann](https://github.com/karthikeyann) + +## 📖 Documentation + +- Fix incorrect example in pylibcudf docs ([#17912](https://github.com/rapidsai/cudf/pull/17912)) [@Matt711](https://github.com/Matt711) +- Explicitly call out that the GPU open beta runs on a single GPU ([#17872](https://github.com/rapidsai/cudf/pull/17872)) [@taureandyernv](https://github.com/taureandyernv) +- Update cudf.pandas colab link in docs ([#17846](https://github.com/rapidsai/cudf/pull/17846)) [@taureandyernv](https://github.com/taureandyernv) +- [DOC] Make pylibcudf docs more visible ([#17803](https://github.com/rapidsai/cudf/pull/17803)) [@Matt711](https://github.com/Matt711) +- Cross-link cudf.pandas profiler documentation. ([#17668](https://github.com/rapidsai/cudf/pull/17668)) [@bdice](https://github.com/bdice) +- Document interpreter install command for cudf.pandas ([#17358](https://github.com/rapidsai/cudf/pull/17358)) [@bdice](https://github.com/bdice) +- add comment to Series.tolist method ([#17350](https://github.com/rapidsai/cudf/pull/17350)) [@tequilayu](https://github.com/tequilayu) + +## 🚀 New Features + +- Bump polars version to <1.22 ([#17771](https://github.com/rapidsai/cudf/pull/17771)) [@Matt711](https://github.com/Matt711) +- Make more constexpr available on device for cuIO ([#17746](https://github.com/rapidsai/cudf/pull/17746)) [@PointKernel](https://github.com/PointKernel) +- Add public interop functions between pylibcudf and cudf classic ([#17730](https://github.com/rapidsai/cudf/pull/17730)) [@Matt711](https://github.com/Matt711) +- Support `dask_expr` migration into `dask.dataframe` ([#17704](https://github.com/rapidsai/cudf/pull/17704)) [@rjzamora](https://github.com/rjzamora) +- Make tests build without relaxed constexpr ([#17691](https://github.com/rapidsai/cudf/pull/17691)) [@PointKernel](https://github.com/PointKernel) +- Set default logger level to warn ([#17684](https://github.com/rapidsai/cudf/pull/17684)) [@vyasr](https://github.com/vyasr) +- Support multithreaded reading of compressed buffers in JSON reader ([#17670](https://github.com/rapidsai/cudf/pull/17670)) [@shrshi](https://github.com/shrshi) +- Control pinned memory use with environment variables ([#17657](https://github.com/rapidsai/cudf/pull/17657)) [@vuule](https://github.com/vuule) +- Host compression ([#17656](https://github.com/rapidsai/cudf/pull/17656)) [@vuule](https://github.com/vuule) +- Enable text build without relying on relaxed constexpr ([#17647](https://github.com/rapidsai/cudf/pull/17647)) [@PointKernel](https://github.com/PointKernel) +- Implement `HOST_UDF` aggregation for reduction and segmented reduction ([#17645](https://github.com/rapidsai/cudf/pull/17645)) [@ttnghia](https://github.com/ttnghia) +- Add JSON reader options structs to pylibcudf ([#17614](https://github.com/rapidsai/cudf/pull/17614)) [@Matt711](https://github.com/Matt711) +- Refactor distinct hash join to handle multiple probes with the same build table ([#17609](https://github.com/rapidsai/cudf/pull/17609)) [@PointKernel](https://github.com/PointKernel) +- Add JSON Writer options classes to pylibcudf ([#17606](https://github.com/rapidsai/cudf/pull/17606)) [@Matt711](https://github.com/Matt711) +- Add ORC reader options structs to pylibcudf ([#17601](https://github.com/rapidsai/cudf/pull/17601)) [@Matt711](https://github.com/Matt711) +- Add Avro Reader options classes to pylibcudf ([#17599](https://github.com/rapidsai/cudf/pull/17599)) [@Matt711](https://github.com/Matt711) +- Enable binaryop build without relying on relaxed constexpr ([#17598](https://github.com/rapidsai/cudf/pull/17598)) [@PointKernel](https://github.com/PointKernel) +- Measure the number of Parquet row groups filtered by predicate pushdown ([#17594](https://github.com/rapidsai/cudf/pull/17594)) [@mhaseeb123](https://github.com/mhaseeb123) +- Implement `HOST_UDF` aggregation for groupby ([#17592](https://github.com/rapidsai/cudf/pull/17592)) [@ttnghia](https://github.com/ttnghia) +- Plumb pylibcudf.io.parquet options classes through cudf python ([#17506](https://github.com/rapidsai/cudf/pull/17506)) [@Matt711](https://github.com/Matt711) +- Add partition-wise `Select` support to cuDF-Polars ([#17495](https://github.com/rapidsai/cudf/pull/17495)) [@rjzamora](https://github.com/rjzamora) +- Add multi-partition `Scan` support to cuDF-Polars ([#17494](https://github.com/rapidsai/cudf/pull/17494)) [@rjzamora](https://github.com/rjzamora) +- Migrate `cudf::io::merge_row_group_metadata` to pylibcudf ([#17491](https://github.com/rapidsai/cudf/pull/17491)) [@Matt711](https://github.com/Matt711) +- Add Parquet Reader options classes to pylibcudf ([#17464](https://github.com/rapidsai/cudf/pull/17464)) [@Matt711](https://github.com/Matt711) +- Add multi-partition `DataFrameScan` support to cuDF-Polars ([#17441](https://github.com/rapidsai/cudf/pull/17441)) [@rjzamora](https://github.com/rjzamora) +- Return empty result for segmented_reduce if input and offsets are both empty ([#17437](https://github.com/rapidsai/cudf/pull/17437)) [@davidwendt](https://github.com/davidwendt) +- Abstract polars function expression nodes to ensure they are serializable ([#17418](https://github.com/rapidsai/cudf/pull/17418)) [@pentschev](https://github.com/pentschev) +- Add CSV Reader options classes to pylibcudf ([#17412](https://github.com/rapidsai/cudf/pull/17412)) [@Matt711](https://github.com/Matt711) +- Add support for `pylibcudf.DataType` serialization ([#17352](https://github.com/rapidsai/cudf/pull/17352)) [@pentschev](https://github.com/pentschev) +- Enable rounding for Decimal32 and Decimal64 in cuDF ([#17332](https://github.com/rapidsai/cudf/pull/17332)) [@a-hirota](https://github.com/a-hirota) +- Remove upper bounds on cuda-python to allow 12.6.2 and 11.8.5 ([#17326](https://github.com/rapidsai/cudf/pull/17326)) [@bdice](https://github.com/bdice) +- Expose stream-ordering to groupby APIs ([#17324](https://github.com/rapidsai/cudf/pull/17324)) [@shrshi](https://github.com/shrshi) +- Migrate ORC Writer to pylibcudf ([#17310](https://github.com/rapidsai/cudf/pull/17310)) [@Matt711](https://github.com/Matt711) +- Support reading bloom filters from Parquet files and filter row groups using them ([#17289](https://github.com/rapidsai/cudf/pull/17289)) [@mhaseeb123](https://github.com/mhaseeb123) + +## 🛠️ Improvements + +- Remove pandas backend from `cudf.pandas` - ibis integration tests ([#17945](https://github.com/rapidsai/cudf/pull/17945)) [@Matt711](https://github.com/Matt711) +- Revert CUDA 12.8 shared workflow branch changes ([#17879](https://github.com/rapidsai/cudf/pull/17879)) [@vyasr](https://github.com/vyasr) +- Remove predicate param from `DataFrameScan` IR ([#17852](https://github.com/rapidsai/cudf/pull/17852)) [@Matt711](https://github.com/Matt711) +- Remove cudf.Scalar from scatter APIs ([#17847](https://github.com/rapidsai/cudf/pull/17847)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf.Scalar from interval_range ([#17844](https://github.com/rapidsai/cudf/pull/17844)) [@mroeschke](https://github.com/mroeschke) +- Add `verify-codeowners` hook ([#17840](https://github.com/rapidsai/cudf/pull/17840)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA) +- Build and test with CUDA 12.8.0 ([#17834](https://github.com/rapidsai/cudf/pull/17834)) [@bdice](https://github.com/bdice) +- Increase timeout for recently added test ([#17829](https://github.com/rapidsai/cudf/pull/17829)) [@galipremsagar](https://github.com/galipremsagar) +- Apply ruff everywhere (notebooks and scripts) ([#17820](https://github.com/rapidsai/cudf/pull/17820)) [@bdice](https://github.com/bdice) +- Fix pre-commit.ci failures ([#17819](https://github.com/rapidsai/cudf/pull/17819)) [@bdice](https://github.com/bdice) +- Remove incorrect calls to set architectures ([#17813](https://github.com/rapidsai/cudf/pull/17813)) [@vyasr](https://github.com/vyasr) +- Fix typo in exception raised when attempting to convert a string column to cupy ([#17800](https://github.com/rapidsai/cudf/pull/17800)) [@dagardner-nv](https://github.com/dagardner-nv) +- Add support for `pyarrow-19` ([#17794](https://github.com/rapidsai/cudf/pull/17794)) [@galipremsagar](https://github.com/galipremsagar) +- increase parallelism in nightly builds ([#17792](https://github.com/rapidsai/cudf/pull/17792)) [@jameslamb](https://github.com/jameslamb) +- Reduce libcudf memcheck tests output ([#17791](https://github.com/rapidsai/cudf/pull/17791)) [@davidwendt](https://github.com/davidwendt) +- Make cudf build with latest CCCL ([#17788](https://github.com/rapidsai/cudf/pull/17788)) [@miscco](https://github.com/miscco) +- Introduce some more rolling window benchmarks ([#17787](https://github.com/rapidsai/cudf/pull/17787)) [@wence-](https://github.com/wence-) +- Add shellcheck to pre-commit and fix warnings ([#17778](https://github.com/rapidsai/cudf/pull/17778)) [@gforsyth](https://github.com/gforsyth) +- Improve parquet reader very-long string performance ([#17773](https://github.com/rapidsai/cudf/pull/17773)) [@pmattione-nvidia](https://github.com/pmattione-nvidia) +- Update how to manage host UDF instance ([#17770](https://github.com/rapidsai/cudf/pull/17770)) [@res-life](https://github.com/res-life) +- Add getInts api for HostMemoryBuffer and UnsafeMemoryAccessor ([#17767](https://github.com/rapidsai/cudf/pull/17767)) [@liurenjie1024](https://github.com/liurenjie1024) +- Expose stream-ordering in scalar and avro APIs ([#17766](https://github.com/rapidsai/cudf/pull/17766)) [@shrshi](https://github.com/shrshi) +- Standarize methods used from `cudf.core._internals` ([#17765](https://github.com/rapidsai/cudf/pull/17765)) [@mroeschke](https://github.com/mroeschke) +- Implement string join in cudf-polars ([#17755](https://github.com/rapidsai/cudf/pull/17755)) [@wence-](https://github.com/wence-) +- Deprecate dataframe protocol ([#17736](https://github.com/rapidsai/cudf/pull/17736)) [@vyasr](https://github.com/vyasr) +- Add parquet reader long row test ([#17735](https://github.com/rapidsai/cudf/pull/17735)) [@pmattione-nvidia](https://github.com/pmattione-nvidia) +- Update kvikio call due to upstream changes ([#17733](https://github.com/rapidsai/cudf/pull/17733)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Delay setting MultiIndex.level/codes until needed ([#17728](https://github.com/rapidsai/cudf/pull/17728)) [@mroeschke](https://github.com/mroeschke) +- Bounding pool size in multi-batch JSON reader ([#17724](https://github.com/rapidsai/cudf/pull/17724)) [@shrshi](https://github.com/shrshi) +- Use GCC 13 in CUDA 12 conda builds. ([#17721](https://github.com/rapidsai/cudf/pull/17721)) [@bdice](https://github.com/bdice) +- Update minimal sphinx theme version so that we can use parallel doc builds ([#17719](https://github.com/rapidsai/cudf/pull/17719)) [@vyasr](https://github.com/vyasr) +- Add more aggregation methods in pylibcudf ([#17717](https://github.com/rapidsai/cudf/pull/17717)) [@mroeschke](https://github.com/mroeschke) +- Make cudf._lib.string_udf work with pylibcudf Columns instead of cudf._lib Columns ([#17715](https://github.com/rapidsai/cudf/pull/17715)) [@mroeschke](https://github.com/mroeschke) +- Add special orc test data: timestamp interspersed with null values ([#17713](https://github.com/rapidsai/cudf/pull/17713)) [@kingcrimsontianyu](https://github.com/kingcrimsontianyu) +- Add pylibcudf.null_mask.null_count ([#17711](https://github.com/rapidsai/cudf/pull/17711)) [@mroeschke](https://github.com/mroeschke) +- Ensure pyarrow.Scalar to pylibcudf.Scalar is cached ([#17707](https://github.com/rapidsai/cudf/pull/17707)) [@mroeschke](https://github.com/mroeschke) +- Adapt cudf numba config for numba 0.61 removal ([#17705](https://github.com/rapidsai/cudf/pull/17705)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.scalar in favor of pylibcudf ([#17701](https://github.com/rapidsai/cudf/pull/17701)) [@mroeschke](https://github.com/mroeschke) +- Fix parquet reader list bug ([#17699](https://github.com/rapidsai/cudf/pull/17699)) [@pmattione-nvidia](https://github.com/pmattione-nvidia) +- Migrated Dynamic AST Expression Trees in Benchmarks and Tests to use AST Tree ([#17697](https://github.com/rapidsai/cudf/pull/17697)) [@lamarrr](https://github.com/lamarrr) +- Skip polars test that can generate timezones that chrono_tz doesn't know ([#17694](https://github.com/rapidsai/cudf/pull/17694)) [@wence-](https://github.com/wence-) +- Use 64-bit offsets only if the current strings column output chunk size exceeds threshold ([#17693](https://github.com/rapidsai/cudf/pull/17693)) [@mhaseeb123](https://github.com/mhaseeb123) +- Use latest ci-conda images ([#17690](https://github.com/rapidsai/cudf/pull/17690)) [@bdice](https://github.com/bdice) +- Add multi-source reading to JSON reader benchmarks ([#17688](https://github.com/rapidsai/cudf/pull/17688)) [@shrshi](https://github.com/shrshi) +- Convert cudf.Scalar usage to pylibcudf and pyarrow usage ([#17686](https://github.com/rapidsai/cudf/pull/17686)) [@mroeschke](https://github.com/mroeschke) +- remove find_package(Python) in libcudf build ([#17683](https://github.com/rapidsai/cudf/pull/17683)) [@jameslamb](https://github.com/jameslamb) +- Fix build metrics report format with long placehold filenames ([#17679](https://github.com/rapidsai/cudf/pull/17679)) [@davidwendt](https://github.com/davidwendt) +- Use rapids-cmake for the logger ([#17674](https://github.com/rapidsai/cudf/pull/17674)) [@vyasr](https://github.com/vyasr) +- Java Parquet reads via multiple host buffers ([#17673](https://github.com/rapidsai/cudf/pull/17673)) [@jlowe](https://github.com/jlowe) +- Remove cudf._libs.types.pyx ([#17665](https://github.com/rapidsai/cudf/pull/17665)) [@mroeschke](https://github.com/mroeschke) +- Add support for `Groupby.cumprod` ([#17661](https://github.com/rapidsai/cudf/pull/17661)) [@galipremsagar](https://github.com/galipremsagar) +- Implement `.dt.total_seconds` ([#17659](https://github.com/rapidsai/cudf/pull/17659)) [@galipremsagar](https://github.com/galipremsagar) +- Avoid shallow copies in groupby methods ([#17646](https://github.com/rapidsai/cudf/pull/17646)) [@mroeschke](https://github.com/mroeschke) +- Avoid double MultiIndex factorization in groupby index result ([#17644](https://github.com/rapidsai/cudf/pull/17644)) [@mroeschke](https://github.com/mroeschke) +- Add seed parameter to hash_character_ngrams ([#17643](https://github.com/rapidsai/cudf/pull/17643)) [@davidwendt](https://github.com/davidwendt) +- Fix possible overflow in WriteCoalescingCallbackWrapper::TearDown ([#17642](https://github.com/rapidsai/cudf/pull/17642)) [@davidwendt](https://github.com/davidwendt) +- Remove pragma GCC diagnostic from source files ([#17637](https://github.com/rapidsai/cudf/pull/17637)) [@davidwendt](https://github.com/davidwendt) +- Move unnecessary utilities from cudf._lib.scalar ([#17636](https://github.com/rapidsai/cudf/pull/17636)) [@mroeschke](https://github.com/mroeschke) +- Support compression= in DataFrame.to_json ([#17634](https://github.com/rapidsai/cudf/pull/17634)) [@mroeschke](https://github.com/mroeschke) +- Bump Polars version to <1.18 ([#17632](https://github.com/rapidsai/cudf/pull/17632)) [@Matt711](https://github.com/Matt711) +- Add public APIs to Access Underlying `cudf` and `pandas` Objects from `cudf.pandas` Proxy Objects ([#17629](https://github.com/rapidsai/cudf/pull/17629)) [@galipremsagar](https://github.com/galipremsagar) +- Use Numba Config to turn on Pynvjitlink Features ([#17628](https://github.com/rapidsai/cudf/pull/17628)) [@isVoid](https://github.com/isVoid) +- Use PyNVML 12 ([#17627](https://github.com/rapidsai/cudf/pull/17627)) [@jakirkham](https://github.com/jakirkham) +- Remove cudf._lib.utils in favor of python APIs ([#17625](https://github.com/rapidsai/cudf/pull/17625)) [@mroeschke](https://github.com/mroeschke) +- Performance improvements and simplifications for fixed size row-based rolling windows ([#17623](https://github.com/rapidsai/cudf/pull/17623)) [@wence-](https://github.com/wence-) +- Fix return types for MurmurHash3_x86_32 template specializations ([#17622](https://github.com/rapidsai/cudf/pull/17622)) [@davidwendt](https://github.com/davidwendt) +- Clean up namespaces and improve compression-related headers ([#17621](https://github.com/rapidsai/cudf/pull/17621)) [@vuule](https://github.com/vuule) +- Use more pylibcudf.types instead of cudf._lib.types ([#17619](https://github.com/rapidsai/cudf/pull/17619)) [@mroeschke](https://github.com/mroeschke) +- Remove patch that is only needed for clang-tidy to run on test files ([#17618](https://github.com/rapidsai/cudf/pull/17618)) [@vyasr](https://github.com/vyasr) +- update telemetry actions to fluent-bit friendly style ([#17615](https://github.com/rapidsai/cudf/pull/17615)) [@msarahan](https://github.com/msarahan) +- Introduce some simple benchmarks for rolling window aggregations ([#17613](https://github.com/rapidsai/cudf/pull/17613)) [@wence-](https://github.com/wence-) +- Bump the oldest `pyarrow` version to `14.0.2` in test matrix ([#17611](https://github.com/rapidsai/cudf/pull/17611)) [@galipremsagar](https://github.com/galipremsagar) +- Use `[[nodiscard]]` attribute before `__device__` ([#17608](https://github.com/rapidsai/cudf/pull/17608)) [@vuule](https://github.com/vuule) +- Use `host_vector` in `flatten_single_pass_aggs` ([#17605](https://github.com/rapidsai/cudf/pull/17605)) [@vuule](https://github.com/vuule) +- Stop memory_resource.hpp from including itself ([#17603](https://github.com/rapidsai/cudf/pull/17603)) [@vyasr](https://github.com/vyasr) +- Replace the outdated cuco window concept with buckets ([#17602](https://github.com/rapidsai/cudf/pull/17602)) [@PointKernel](https://github.com/PointKernel) +- Check if nightlies have succeeded recently enough ([#17596](https://github.com/rapidsai/cudf/pull/17596)) [@vyasr](https://github.com/vyasr) +- Deprecate cudf::grouped_time_range_rolling_window ([#17589](https://github.com/rapidsai/cudf/pull/17589)) [@wence-](https://github.com/wence-) +- A couple of fixes in rapids-logger usage ([#17588](https://github.com/rapidsai/cudf/pull/17588)) [@vyasr](https://github.com/vyasr) +- Simplify expression transformer in Parquet predicate pushdown with `ast::tree` ([#17587](https://github.com/rapidsai/cudf/pull/17587)) [@mhaseeb123](https://github.com/mhaseeb123) +- Remove unused functionality in cudf._lib.utils.pyx ([#17586](https://github.com/rapidsai/cudf/pull/17586)) [@mroeschke](https://github.com/mroeschke) +- Use cuda-python `cuda.bindings` import names. ([#17585](https://github.com/rapidsai/cudf/pull/17585)) [@bdice](https://github.com/bdice) +- Use no-sync copy for fixed-width types in cudf::concatenate ([#17584](https://github.com/rapidsai/cudf/pull/17584)) [@davidwendt](https://github.com/davidwendt) +- Remove cudf._lib.groupby in favor of inlining pylibcudf ([#17582](https://github.com/rapidsai/cudf/pull/17582)) [@mroeschke](https://github.com/mroeschke) +- Remove unused code of json schema in JSON reader ([#17581](https://github.com/rapidsai/cudf/pull/17581)) [@karthikeyann](https://github.com/karthikeyann) +- Expose Scalar's constructor and `Scalar#getScalarHandle()` to public ([#17580](https://github.com/rapidsai/cudf/pull/17580)) [@ttnghia](https://github.com/ttnghia) +- Allow large strings in nvtext benchmarks ([#17579](https://github.com/rapidsai/cudf/pull/17579)) [@davidwendt](https://github.com/davidwendt) +- Remove cudf._lib.reduce in favor of inlining pylibcudf ([#17574](https://github.com/rapidsai/cudf/pull/17574)) [@mroeschke](https://github.com/mroeschke) +- Use batched memcpy when writing ORC statistics ([#17572](https://github.com/rapidsai/cudf/pull/17572)) [@vuule](https://github.com/vuule) +- Allow large strings in nvbench strings benchmarks ([#17571](https://github.com/rapidsai/cudf/pull/17571)) [@davidwendt](https://github.com/davidwendt) +- Update version references in workflow ([#17568](https://github.com/rapidsai/cudf/pull/17568)) [@AyodeAwe](https://github.com/AyodeAwe) +- Enable all json reader options in pylibcudf read_json ([#17563](https://github.com/rapidsai/cudf/pull/17563)) [@karthikeyann](https://github.com/karthikeyann) +- Remove cudf._lib.parquet in favor of inlining pylibcudf ([#17562](https://github.com/rapidsai/cudf/pull/17562)) [@mroeschke](https://github.com/mroeschke) +- Fix CMake format in cudf/_lib/CMakeLists.txt ([#17559](https://github.com/rapidsai/cudf/pull/17559)) [@mroeschke](https://github.com/mroeschke) +- Remove "legacy" Dask DataFrame support from Dask cuDF ([#17558](https://github.com/rapidsai/cudf/pull/17558)) [@rjzamora](https://github.com/rjzamora) +- Replace direct `cudaMemcpyAsync` calls with utility functions (within `/include`) ([#17557](https://github.com/rapidsai/cudf/pull/17557)) [@vuule](https://github.com/vuule) +- Remove cudf._lib.interop in favor of inlining pylibcudf ([#17555](https://github.com/rapidsai/cudf/pull/17555)) [@mroeschke](https://github.com/mroeschke) +- gate telemetry dispatch calls on TELEMETRY_ENABLED env var ([#17551](https://github.com/rapidsai/cudf/pull/17551)) [@msarahan](https://github.com/msarahan) +- Replace direct `cudaMemcpyAsync` calls with utility functions (within `/src`) ([#17550](https://github.com/rapidsai/cudf/pull/17550)) [@vuule](https://github.com/vuule) +- Remove unused `BufferArrayFromVector` ([#17549](https://github.com/rapidsai/cudf/pull/17549)) [@Matt711](https://github.com/Matt711) +- Move cudf._lib.copying to cudf.core._internals ([#17548](https://github.com/rapidsai/cudf/pull/17548)) [@mroeschke](https://github.com/mroeschke) +- Update cuda-python lower bounds to 12.6.2 / 11.8.5 ([#17547](https://github.com/rapidsai/cudf/pull/17547)) [@bdice](https://github.com/bdice) +- Fix typos, rename types, and add null_probability benchmark axis for distinct ([#17546](https://github.com/rapidsai/cudf/pull/17546)) [@PointKernel](https://github.com/PointKernel) +- Mark more constexpr functions as device-available ([#17545](https://github.com/rapidsai/cudf/pull/17545)) [@vyasr](https://github.com/vyasr) +- Use cooperative-groups instead of cub warp-reduce for strings contains ([#17540](https://github.com/rapidsai/cudf/pull/17540)) [@davidwendt](https://github.com/davidwendt) +- Remove cudf._lib.nvtext in favor of inlining pylibcudf ([#17535](https://github.com/rapidsai/cudf/pull/17535)) [@mroeschke](https://github.com/mroeschke) +- Add XXHash_32 hasher ([#17533](https://github.com/rapidsai/cudf/pull/17533)) [@PointKernel](https://github.com/PointKernel) +- Remove unused masked keyword in column_empty ([#17530](https://github.com/rapidsai/cudf/pull/17530)) [@mroeschke](https://github.com/mroeschke) +- Remove Thrust patch in favor of CMake definition for Thrust 32-bit offset types. ([#17527](https://github.com/rapidsai/cudf/pull/17527)) [@bdice](https://github.com/bdice) +- [JNI] Enables fabric handles for CUDA async memory pools ([#17526](https://github.com/rapidsai/cudf/pull/17526)) [@abellina](https://github.com/abellina) +- Force Thrust to use 32-bit offset type. ([#17523](https://github.com/rapidsai/cudf/pull/17523)) [@bdice](https://github.com/bdice) +- Replace cudf::detail::copy_if logic with thrust::copy_if and gather ([#17520](https://github.com/rapidsai/cudf/pull/17520)) [@davidwendt](https://github.com/davidwendt) +- Replaces uses of `cudf._lib.Column.from_unique_ptr` with `pylibcudf.Column.from_libcudf` ([#17517](https://github.com/rapidsai/cudf/pull/17517)) [@Matt711](https://github.com/Matt711) +- Move cudf._lib.aggregation to cudf.core._internals ([#17516](https://github.com/rapidsai/cudf/pull/17516)) [@mroeschke](https://github.com/mroeschke) +- Migrate copy_column and Column.from_scalar to pylibcudf ([#17513](https://github.com/rapidsai/cudf/pull/17513)) [@Matt711](https://github.com/Matt711) +- Remove cudf._lib.transform in favor of inlining pylibcudf ([#17505](https://github.com/rapidsai/cudf/pull/17505)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.string.convert/split in favor of inlining pylibcudf ([#17496](https://github.com/rapidsai/cudf/pull/17496)) [@mroeschke](https://github.com/mroeschke) +- Move cudf._lib.sort to cudf.core._internals ([#17488](https://github.com/rapidsai/cudf/pull/17488)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.csv in favor in inlining pylibcudf ([#17485](https://github.com/rapidsai/cudf/pull/17485)) [@mroeschke](https://github.com/mroeschke) +- Update PyTorch to >=2.4.0 to get fix for CUDA array interface bug, and drop CUDA 11 PyTorch tests. ([#17475](https://github.com/rapidsai/cudf/pull/17475)) [@bdice](https://github.com/bdice) +- Remove cudf._lib.binops in favor of inlining pylibcudf ([#17468](https://github.com/rapidsai/cudf/pull/17468)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.orc in favor of inlining pylibcudf ([#17466](https://github.com/rapidsai/cudf/pull/17466)) [@mroeschke](https://github.com/mroeschke) +- skip most CI on devcontainer-only changes ([#17465](https://github.com/rapidsai/cudf/pull/17465)) [@jameslamb](https://github.com/jameslamb) +- Set build type for all examples ([#17463](https://github.com/rapidsai/cudf/pull/17463)) [@vyasr](https://github.com/vyasr) +- Update the hook versions in pre-commit ([#17462](https://github.com/rapidsai/cudf/pull/17462)) [@wence-](https://github.com/wence-) +- Remove cudf._lib.string_casting in favor of inlining pylibcudf ([#17460](https://github.com/rapidsai/cudf/pull/17460)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.filling in favor of inlining pylibcudf ([#17459](https://github.com/rapidsai/cudf/pull/17459)) [@mroeschke](https://github.com/mroeschke) +- Update MurmurHash3_x64_128 to use the cuco equivalent implementation ([#17457](https://github.com/rapidsai/cudf/pull/17457)) [@PointKernel](https://github.com/PointKernel) +- Move cudf._lib.stream_compaction to cudf.core._internals ([#17456](https://github.com/rapidsai/cudf/pull/17456)) [@mroeschke](https://github.com/mroeschke) +- Clean up xxhash_64 implementations ([#17455](https://github.com/rapidsai/cudf/pull/17455)) [@PointKernel](https://github.com/PointKernel) +- Update Hadoop dependency in Java pom ([#17454](https://github.com/rapidsai/cudf/pull/17454)) [@jlowe](https://github.com/jlowe) +- Adapt to rmm logger changes ([#17451](https://github.com/rapidsai/cudf/pull/17451)) [@vyasr](https://github.com/vyasr) +- Require approval to run CI on draft PRs ([#17450](https://github.com/rapidsai/cudf/pull/17450)) [@bdice](https://github.com/bdice) +- Expose stream-ordering in nvtext API ([#17446](https://github.com/rapidsai/cudf/pull/17446)) [@shrshi](https://github.com/shrshi) +- Use exec_policy_nosync in write_json ([#17445](https://github.com/rapidsai/cudf/pull/17445)) [@karthikeyann](https://github.com/karthikeyann) +- Remove cudf._lib.json in favor of inlining pylibcudf ([#17443](https://github.com/rapidsai/cudf/pull/17443)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.null_mask in favor of inlining pylibcudf ([#17440](https://github.com/rapidsai/cudf/pull/17440)) [@mroeschke](https://github.com/mroeschke) +- Expose stream-ordering in replace API ([#17436](https://github.com/rapidsai/cudf/pull/17436)) [@shrshi](https://github.com/shrshi) +- Expose stream-ordering in copying APIs ([#17435](https://github.com/rapidsai/cudf/pull/17435)) [@shrshi](https://github.com/shrshi) +- Expose stream-ordering in column view APIs ([#17434](https://github.com/rapidsai/cudf/pull/17434)) [@shrshi](https://github.com/shrshi) +- Apply clang-tidy autofixes from new rules ([#17431](https://github.com/rapidsai/cudf/pull/17431)) [@vyasr](https://github.com/vyasr) +- Remove cudf._lib.round in favor of inlining pylibcudf ([#17430](https://github.com/rapidsai/cudf/pull/17430)) [@mroeschke](https://github.com/mroeschke) +- Update MurmurHash3_x86_32 to use the cuco equivalent implementation ([#17429](https://github.com/rapidsai/cudf/pull/17429)) [@PointKernel](https://github.com/PointKernel) +- Remove cudf._lib.replace in favor of inlining pylibcudf ([#17428](https://github.com/rapidsai/cudf/pull/17428)) [@mroeschke](https://github.com/mroeschke) +- Remove nvtx/ranges.hpp include from cuda.cuh ([#17427](https://github.com/rapidsai/cudf/pull/17427)) [@davidwendt](https://github.com/davidwendt) +- Remove the unused detail `int_fastdiv.h` header ([#17426](https://github.com/rapidsai/cudf/pull/17426)) [@PointKernel](https://github.com/PointKernel) +- Remove cudf._lib.lists in favor of inlining pylibcudf ([#17425](https://github.com/rapidsai/cudf/pull/17425)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.quantile ([#17424](https://github.com/rapidsai/cudf/pull/17424)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.rolling in favor of inlining pylibcudf ([#17423](https://github.com/rapidsai/cudf/pull/17423)) [@mroeschke](https://github.com/mroeschke) +- Avoid converting Decimal32/Decimal64 in `to_arrow` and `from_arrow` APIs ([#17422](https://github.com/rapidsai/cudf/pull/17422)) [@zeroshade](https://github.com/zeroshade) +- Rework minhash APIs for deprecation cycle ([#17421](https://github.com/rapidsai/cudf/pull/17421)) [@davidwendt](https://github.com/davidwendt) +- Use thread_index_type in binary-ops jit kernel.cu ([#17420](https://github.com/rapidsai/cudf/pull/17420)) [@davidwendt](https://github.com/davidwendt) +- Change binops for-each kernel to thrust::for_each_n ([#17419](https://github.com/rapidsai/cudf/pull/17419)) [@davidwendt](https://github.com/davidwendt) +- Move cudf._lib.search to cudf.core._internals ([#17411](https://github.com/rapidsai/cudf/pull/17411)) [@mroeschke](https://github.com/mroeschke) +- Use grid_1d utilities in copy_range.cuh ([#17409](https://github.com/rapidsai/cudf/pull/17409)) [@davidwendt](https://github.com/davidwendt) +- Remove cudf._lib.text in favor of inlining pylibcudf ([#17408](https://github.com/rapidsai/cudf/pull/17408)) [@mroeschke](https://github.com/mroeschke) +- Run clang-tidy checks in PR CI ([#17407](https://github.com/rapidsai/cudf/pull/17407)) [@bdice](https://github.com/bdice) +- Update strings/text source to use grid_1d for thread/block/stride calculations ([#17404](https://github.com/rapidsai/cudf/pull/17404)) [@davidwendt](https://github.com/davidwendt) +- Expose stream-ordering to strings attribute APIs ([#17398](https://github.com/rapidsai/cudf/pull/17398)) [@shrshi](https://github.com/shrshi) +- Expose stream-ordering to interop APIs ([#17397](https://github.com/rapidsai/cudf/pull/17397)) [@shrshi](https://github.com/shrshi) +- Remove unused type aliases ([#17396](https://github.com/rapidsai/cudf/pull/17396)) [@PointKernel](https://github.com/PointKernel) +- Remove some cudf._lib.strings files in favor of inlining pylibcudf ([#17394](https://github.com/rapidsai/cudf/pull/17394)) [@mroeschke](https://github.com/mroeschke) +- Update xxhash_64 to utilize the cuco equivalent implementation ([#17393](https://github.com/rapidsai/cudf/pull/17393)) [@PointKernel](https://github.com/PointKernel) +- Change indices for dictionary column to signed integer type ([#17390](https://github.com/rapidsai/cudf/pull/17390)) [@davidwendt](https://github.com/davidwendt) +- Return categorical values in to_numpy/to_cupy ([#17388](https://github.com/rapidsai/cudf/pull/17388)) [@mroeschke](https://github.com/mroeschke) +- Forward-merge branch-24.12 to branch-25.02 ([#17379](https://github.com/rapidsai/cudf/pull/17379)) [@bdice](https://github.com/bdice) +- Remove unused IO utilities from cudf python ([#17374](https://github.com/rapidsai/cudf/pull/17374)) [@Matt711](https://github.com/Matt711) +- Remove cudf._lib.datetime in favor of inlining pylibcudf ([#17372](https://github.com/rapidsai/cudf/pull/17372)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.join in favor of inlining pylibcudf ([#17371](https://github.com/rapidsai/cudf/pull/17371)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.merge in favor of inlining pylibcudf ([#17370](https://github.com/rapidsai/cudf/pull/17370)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.partitioning in favor of inlining pylibcudf ([#17369](https://github.com/rapidsai/cudf/pull/17369)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.reshape in favor of inlining pylibcudf ([#17368](https://github.com/rapidsai/cudf/pull/17368)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.timezone in favor of inlining pylibcudf ([#17366](https://github.com/rapidsai/cudf/pull/17366)) [@mroeschke](https://github.com/mroeschke) +- Remove cudf._lib.transpose in favor of inlining pylibcudf ([#17365](https://github.com/rapidsai/cudf/pull/17365)) [@mroeschke](https://github.com/mroeschke) +- Move make_strings_column benchmark to nvbench ([#17340](https://github.com/rapidsai/cudf/pull/17340)) [@davidwendt](https://github.com/davidwendt) +- Improve strings contains/find performance for smaller strings ([#17330](https://github.com/rapidsai/cudf/pull/17330)) [@davidwendt](https://github.com/davidwendt) +- Use rapids-logger to generate the cudf logger ([#17307](https://github.com/rapidsai/cudf/pull/17307)) [@vyasr](https://github.com/vyasr) +- Mukernels strings ([#17286](https://github.com/rapidsai/cudf/pull/17286)) [@pmattione-nvidia](https://github.com/pmattione-nvidia) +- Add write_parquet to pylibcudf ([#17263](https://github.com/rapidsai/cudf/pull/17263)) [@mroeschke](https://github.com/mroeschke) +- Single-partition Dask executor for cuDF-Polars ([#17262](https://github.com/rapidsai/cudf/pull/17262)) [@rjzamora](https://github.com/rjzamora) +- Add breaking change workflow trigger ([#17248](https://github.com/rapidsai/cudf/pull/17248)) [@AyodeAwe](https://github.com/AyodeAwe) +- Precompute AST arity ([#17234](https://github.com/rapidsai/cudf/pull/17234)) [@bdice](https://github.com/bdice) +- Update to CCCL 2.7.0-rc2. ([#17233](https://github.com/rapidsai/cudf/pull/17233)) [@bdice](https://github.com/bdice) +- Make `column_empty` mask buffer creation consistent with libcudf ([#16715](https://github.com/rapidsai/cudf/pull/16715)) [@mroeschke](https://github.com/mroeschke) + # cudf 24.10.00 (9 Oct 2024) ## 🚨 Breaking Changes From b07faa3890192accc4e1c5914909714c20b8e3d9 Mon Sep 17 00:00:00 2001 From: Lawrence Mitchell Date: Thu, 13 Feb 2025 18:06:24 +0000 Subject: [PATCH 032/129] Remove deprecated rolling window functionality (#17993) Remove deprecated rolling window functionality. Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Basit Ayantunde (https://github.com/lamarrr) URL: https://github.com/rapidsai/cudf/pull/17993 --- cpp/include/cudf/rolling.hpp | 134 ------------------------- cpp/src/rolling/grouped_rolling.cu | 151 ----------------------------- 2 files changed, 285 deletions(-) diff --git a/cpp/include/cudf/rolling.hpp b/cpp/include/cudf/rolling.hpp index 3158445841e..6087c025b94 100644 --- a/cpp/include/cudf/rolling.hpp +++ b/cpp/include/cudf/rolling.hpp @@ -321,140 +321,6 @@ std::unique_ptr grouped_rolling_window( rmm::cuda_stream_view stream = cudf::get_default_stream(), rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); -/** - * @brief Applies a grouping-aware, timestamp-based rolling window function to the values in a - * column. - * - * @deprecated Since 25.02, to be removed in 25.04 - * - * Like `rolling_window()`, this function aggregates values in a window around each - * element of a specified `input` column. It differs from `rolling_window()` in two respects: - * 1. The elements of the `input` column are grouped into distinct groups (e.g. the result of a - * groupby), determined by the corresponding values of the columns under `group_keys`. The - * window-aggregation cannot cross the group boundaries. - * 2. Within a group, the aggregation window is calculated based on a time interval (e.g. number - * of days preceding/following the current row). The timestamps for the input data are - * specified by the `timestamp_column` argument. - * - * Note: This method requires that the rows are presorted by the group keys and timestamp values. - * - * @code{.pseudo} - * Example: Consider a user-sales dataset, where the rows look as follows: - * { "user_id", sales_amt, date } - * - * This method enables windowing queries such as grouping a dataset by `user_id`, sorting by - * increasing `date`, and summing up the `sales_amt` column over a window of 3 days (1 preceding - *day, the current day, and 1 following day). - * - * In this example, - * 1. `group_keys == [ user_id ]` - * 2. `timestamp_column == date` - * 3. `input == sales_amt` - * The data are grouped by `user_id`, and ordered by `date`. The aggregation - * (SUM) is then calculated for a window of 3 days around (and including) each row. - * - * For the following input: - * - * [ // user, sales_amt, YYYYMMDD (date) - * { "user1", 10, 20200101 }, - * { "user2", 20, 20200101 }, - * { "user1", 20, 20200102 }, - * { "user1", 10, 20200103 }, - * { "user2", 30, 20200101 }, - * { "user2", 80, 20200102 }, - * { "user1", 50, 20200107 }, - * { "user1", 60, 20200107 }, - * { "user2", 40, 20200104 } - * ] - * - * Partitioning (grouping) by `user_id`, and ordering by `date` yields the following `sales_amt` - * vector (with 2 groups, one for each distinct `user_id`): - * - * Date :(202001-) [ 01, 02, 03, 07, 07, 01, 01, 02, 04 ] - * Input: [ 10, 20, 10, 50, 60, 20, 30, 80, 40 ] - * <-------user1-------->|<---------user2---------> - * - * The SUM aggregation is applied, with 1 day preceding, and 1 day following, with a minimum of 1 - * period. The aggregation window is thus 3 *days* wide, yielding the following output column: - * - * Results: [ 30, 40, 30, 110, 110, 130, 130, 130, 40 ] - * - * @endcode - * - * Note: The number of rows participating in each window might vary, based on the index within the - * group, datestamp, and `min_periods`. Apropos: - * 1. results[0] considers 2 values, because it is at the beginning of its group, and has no - * preceding values. - * 2. results[5] considers 3 values, despite being at the beginning of its group. It must include 2 - * following values, based on its datestamp. - * - * Each aggregation operation cannot cross group boundaries. - * - * The returned column for `op == COUNT` always has `INT32` type. All other operators return a - * column of the same type as the input. Therefore it is suggested to convert integer column types - * (especially low-precision integers) to `FLOAT32` or `FLOAT64` before doing a rolling `MEAN`. - * - * @param[in] group_keys The (pre-sorted) grouping columns - * @param[in] timestamp_column The (pre-sorted) timestamps for each row - * @param[in] timestamp_order The order (ASCENDING/DESCENDING) in which the timestamps are sorted - * @param[in] input The input column (to be aggregated) - * @param[in] preceding_window_in_days The rolling window time-interval in the backward direction - * @param[in] following_window_in_days The rolling window time-interval in the forward direction - * @param[in] min_periods Minimum number of observations in window required to have a value, - * otherwise element `i` is null. - * @param[in] aggr The rolling window aggregation type (SUM, MAX, MIN, etc.) - * @param[in] stream CUDA stream used for device memory operations and kernel launches - * @param[in] mr Device memory resource used to allocate the returned column's device memory - * - * @returns A nullable output column containing the rolling window results - */ -[[deprecated("Use cudf::grouped_range_rolling_window instead")]] std::unique_ptr -grouped_time_range_rolling_window( - table_view const& group_keys, - column_view const& timestamp_column, - cudf::order const& timestamp_order, - column_view const& input, - size_type preceding_window_in_days, - size_type following_window_in_days, - size_type min_periods, - rolling_aggregation const& aggr, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Applies a grouping-aware, timestamp-based rolling window function to the values in a - * column,. - * - * @deprecated Since 25.02, to be removed in 25.04 - * - * @details @copydetails grouped_time_range_rolling_window( - * table_view const& group_keys, - * column_view const& timestamp_column, - * cudf::order const& timestamp_order, - * column_view const& input, - * size_type preceding_window_in_days, - * size_type following_window_in_days, - * size_type min_periods, - * rolling_aggregation const& aggr, - * rmm::cuda_stream_view stream, - * rmm::device_async_resource_ref mr) - * - * The `preceding_window_in_days` and `following_window_in_days` are specified as a `window_bounds` - * and supports "unbounded" windows, if set to `window_bounds::unbounded()`. - */ -[[deprecated("Use cudf::grouped_range_rolling_window instead")]] std::unique_ptr -grouped_time_range_rolling_window( - table_view const& group_keys, - column_view const& timestamp_column, - cudf::order const& timestamp_order, - column_view const& input, - window_bounds preceding_window_in_days, - window_bounds following_window_in_days, - size_type min_periods, - rolling_aggregation const& aggr, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - /** * @brief Applies a grouping-aware, value range-based rolling window function to the values in a * column. diff --git a/cpp/src/rolling/grouped_rolling.cu b/cpp/src/rolling/grouped_rolling.cu index 18c793029b6..8ab2ce65124 100644 --- a/cpp/src/rolling/grouped_rolling.cu +++ b/cpp/src/rolling/grouped_rolling.cu @@ -942,77 +942,6 @@ struct dispatch_grouped_range_rolling_window { } }; -/** - * @brief Functor to convert from size_type (number of days) to appropriate duration type. - */ -struct to_duration_bounds { - template (), void>* = nullptr> - range_window_bounds operator()(size_type num_days, rmm::cuda_stream_view stream) const - { - using DurationT = typename OrderBy::duration; - return range_window_bounds::get(duration_scalar{duration_D{num_days}, true, stream}, - stream); - } - - template (), void>* = nullptr> - range_window_bounds operator()(size_type, rmm::cuda_stream_view) const - { - CUDF_FAIL("Expected timestamp orderby column."); - } -}; - -/** - * @brief Get duration type corresponding to specified timestamp type. - */ -data_type get_duration_type_for(cudf::data_type timestamp_type) -{ - switch (timestamp_type.id()) { - case type_id::TIMESTAMP_DAYS: return data_type{type_id::DURATION_DAYS}; - case type_id::TIMESTAMP_SECONDS: return data_type{type_id::DURATION_SECONDS}; - case type_id::TIMESTAMP_MILLISECONDS: return data_type{type_id::DURATION_MILLISECONDS}; - case type_id::TIMESTAMP_MICROSECONDS: return data_type{type_id::DURATION_MICROSECONDS}; - case type_id::TIMESTAMP_NANOSECONDS: return data_type{type_id::DURATION_NANOSECONDS}; - default: CUDF_FAIL("Expected timestamp orderby column."); - } -} - -/** - * @brief Bridge function to convert from size_type (number of days) to appropriate duration type. - * - * This helps adapt the old `grouped_time_range_rolling_window()` functions that took a "number of - * days" to the new `range_window_bounds` interface. - * - * @param num_days Window bounds specified in number of days in `size_type` - * @param timestamp_type Data-type of the orderby column to which the `num_days` is to be adapted. - * @return range_window_bounds A `range_window_bounds` to be used with the new API. - */ -range_window_bounds to_range_bounds(cudf::size_type num_days, - cudf::data_type timestamp_type, - rmm::cuda_stream_view stream) -{ - return cudf::type_dispatcher(timestamp_type, to_duration_bounds{}, num_days, stream); -} - -/** - * @brief Bridge function to convert from `window_bounds` (in days) to appropriate duration type. - * - * This helps adapt the old `grouped_time_range_rolling_window()` functions that took a - * `window_bounds` to the new `range_window_bounds` interface. - * - * @param days_bounds The static window-width `window_bounds` object - * @param timestamp_type Data-type of the orderby column to which the `num_days` is to be adapted. - * @return range_window_bounds A `range_window_bounds` to be used with the new API. - */ -range_window_bounds to_range_bounds(cudf::window_bounds const& days_bounds, - cudf::data_type timestamp_type, - rmm::cuda_stream_view stream) -{ - return days_bounds.is_unbounded() - ? range_window_bounds::unbounded(get_duration_type_for(timestamp_type), stream) - : cudf::type_dispatcher( - timestamp_type, to_duration_bounds{}, days_bounds.value(), stream); -} - } // namespace namespace detail { @@ -1084,86 +1013,6 @@ std::unique_ptr grouped_range_rolling_window(table_view const& group_key } // namespace detail -/** - * @copydoc std::unique_ptr grouped_time_range_rolling_window( - * table_view const& group_keys, - * column_view const& timestamp_column, - * cudf::order const& timestamp_order, - * column_view const& input, - * size_type preceding_window_in_days, - * size_type following_window_in_days, - * size_type min_periods, - * rolling_aggregation const& aggr, - * rmm::device_async_resource_ref mr); - */ -std::unique_ptr grouped_time_range_rolling_window(table_view const& group_keys, - column_view const& timestamp_column, - cudf::order const& timestamp_order, - column_view const& input, - size_type preceding_window_in_days, - size_type following_window_in_days, - size_type min_periods, - rolling_aggregation const& aggr, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - auto preceding = to_range_bounds(preceding_window_in_days, timestamp_column.type(), stream); - auto following = to_range_bounds(following_window_in_days, timestamp_column.type(), stream); - - return detail::grouped_range_rolling_window(group_keys, - timestamp_column, - timestamp_order, - input, - preceding, - following, - min_periods, - aggr, - stream, - mr); -} - -/** - * @copydoc grouped_time_range_rolling_window( - * table_view const& group_keys, - * column_view const& timestamp_column, - * cudf::order const& timestamp_order, - * column_view const& input, - * window_bounds preceding_window_in_days, - * window_bounds following_window_in_days, - * size_type min_periods, - * rolling_aggregation const& aggr, - * rmm::device_async_resource_ref mr); - */ -std::unique_ptr grouped_time_range_rolling_window(table_view const& group_keys, - column_view const& timestamp_column, - cudf::order const& timestamp_order, - column_view const& input, - window_bounds preceding_window_in_days, - window_bounds following_window_in_days, - size_type min_periods, - rolling_aggregation const& aggr, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - range_window_bounds preceding = - to_range_bounds(preceding_window_in_days, timestamp_column.type(), stream); - range_window_bounds following = - to_range_bounds(following_window_in_days, timestamp_column.type(), stream); - - return detail::grouped_range_rolling_window(group_keys, - timestamp_column, - timestamp_order, - input, - preceding, - following, - min_periods, - aggr, - stream, - mr); -} - /** * @copydoc grouped_range_rolling_window( * table_view const& group_keys, From 53eee38b00768463901627723899f4cf937d68fb Mon Sep 17 00:00:00 2001 From: Kyle Edwards Date: Thu, 13 Feb 2025 13:14:54 -0500 Subject: [PATCH 033/129] Create Conda CI test env in one step (#17995) Issue: https://github.com/rapidsai/build-planning/issues/22 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17995 --- ci/release/update-version.sh | 1 + ci/test_python_common.sh | 8 ++++---- ci/test_python_other.sh | 12 +----------- dependencies.yaml | 25 +++++++++++++++++++++++++ 4 files changed, 31 insertions(+), 15 deletions(-) diff --git a/ci/release/update-version.sh b/ci/release/update-version.sh index f4f31dfbb6f..d014d3b08ff 100755 --- a/ci/release/update-version.sh +++ b/ci/release/update-version.sh @@ -43,6 +43,7 @@ sed_runner "s/branch-.*/branch-${NEXT_SHORT_TAG}/g" ci/test_wheel_dask_cudf.sh DEPENDENCIES=( cudf cudf_kafka + cudf-polars cugraph cuml custreamz diff --git a/ci/test_python_common.sh b/ci/test_python_common.sh index 65d3125552a..63f7317c19f 100755 --- a/ci/test_python_common.sh +++ b/ci/test_python_common.sh @@ -9,6 +9,10 @@ set -euo pipefail RAPIDS_VERSION="$(rapids-version)" +rapids-logger "Downloading artifacts from previous jobs" +CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) +PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python) + rapids-logger "Generate Python testing dependencies" ENV_YAML_DIR="$(mktemp -d)" @@ -26,10 +30,6 @@ set +u conda activate test set -u -rapids-logger "Downloading artifacts from previous jobs" -CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) -PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python) - RESULTS_DIR=${RAPIDS_TESTS_DIR:-"$(mktemp -d)"} RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${RESULTS_DIR}/test-results"}/ RAPIDS_COVERAGE_DIR=${RAPIDS_COVERAGE_DIR:-"${RESULTS_DIR}/coverage-results"} diff --git a/ci/test_python_other.sh b/ci/test_python_other.sh index 3c6dba72164..b0a03ba69cc 100755 --- a/ci/test_python_other.sh +++ b/ci/test_python_other.sh @@ -7,19 +7,9 @@ cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../ # Common setup steps shared by Python test jobs source ./ci/test_python_common.sh test_python_other -RAPIDS_VERSION="$(rapids-version)" - -rapids-mamba-retry install \ - --channel "${CPP_CHANNEL}" \ - --channel "${PYTHON_CHANNEL}" \ - "dask-cudf=${RAPIDS_VERSION}" \ - "cudf_kafka=${RAPIDS_VERSION}" \ - "custreamz=${RAPIDS_VERSION}" \ - "cudf-polars=${RAPIDS_VERSION}" - rapids-logger "Check GPU usage" nvidia-smi - +rapids-print-env EXITCODE=0 trap "EXITCODE=1" ERR set +e diff --git a/dependencies.yaml b/dependencies.yaml index db3ce1e535d..2cb876e075a 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -73,6 +73,9 @@ files: - test_python_common - test_python_cudf_common - test_python_cudf + - depends_on_cudf + - depends_on_pylibcudf + - depends_on_libcudf test_python_other: output: none includes: @@ -81,6 +84,13 @@ files: - test_python_common - test_python_cudf_common - test_python_dask_cudf + - depends_on_cudf + - depends_on_pylibcudf + - depends_on_libcudf + - depends_on_dask_cudf + - depends_on_cudf_kafka + - depends_on_custreamz + - depends_on_cudf_polars test_java: output: none includes: @@ -1174,3 +1184,18 @@ dependencies: - nbconvert - nbformat - openpyxl + depends_on_dask_cudf: + common: + - output_types: conda + packages: + - dask-cudf==25.4.*,>=0.0.0a0 + depends_on_custreamz: + common: + - output_types: conda + packages: + - custreamz==25.4.*,>=0.0.0a0 + depends_on_cudf_polars: + common: + - output_types: conda + packages: + - cudf-polars==25.4.*,>=0.0.0a0 From 3fa56d06afe57bfa54a84ab1a1e9faf99f5436fd Mon Sep 17 00:00:00 2001 From: "Richard (Rick) Zamora" Date: Thu, 13 Feb 2025 12:59:31 -0600 Subject: [PATCH 034/129] Add `Column.serialize` to cudf-polars (#17990) It will be useful to serialize individual columns during multi-GPU cudf-polars execution. For example, the `Expr`-decomposition approach proposed in https://github.com/rapidsai/cudf/pull/17941 may "require" `Column` serialization (or an ugly workaround). Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Matthew Murray (https://github.com/Matt711) - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/17990 --- .../cudf_polars/containers/column.py | 61 +++++++++++++++++++ .../cudf_polars/containers/dataframe.py | 23 ++++--- .../experimental/dask_serialize.py | 26 ++++++-- .../cudf_polars/typing/__init__.py | 33 +++++++++- .../tests/experimental/test_dask_serialize.py | 11 +++- 5 files changed, 133 insertions(+), 21 deletions(-) diff --git a/python/cudf_polars/cudf_polars/containers/column.py b/python/cudf_polars/cudf_polars/containers/column.py index 2c83e05fe9c..f296b2dc828 100644 --- a/python/cudf_polars/cudf_polars/containers/column.py +++ b/python/cudf_polars/cudf_polars/containers/column.py @@ -26,6 +26,8 @@ import polars as pl + from cudf_polars.typing import ColumnHeader, ColumnOptions + __all__: list[str] = ["Column"] @@ -55,6 +57,65 @@ def __init__( self.name = name self.set_sorted(is_sorted=is_sorted, order=order, null_order=null_order) + @classmethod + def deserialize( + cls, header: ColumnHeader, frames: tuple[memoryview, plc.gpumemoryview] + ) -> Self: + """ + Create a Column from a serialized representation returned by `.serialize()`. + + Parameters + ---------- + header + The (unpickled) metadata required to reconstruct the object. + frames + Two-tuple of frames (a memoryview and a gpumemoryview). + + Returns + ------- + Column + The deserialized Column. + """ + packed_metadata, packed_gpu_data = frames + (plc_column,) = plc.contiguous_split.unpack_from_memoryviews( + packed_metadata, packed_gpu_data + ).columns() + return cls(plc_column, **header["column_kwargs"]) + + def serialize( + self, + ) -> tuple[ColumnHeader, tuple[memoryview, plc.gpumemoryview]]: + """ + Serialize the Column into header and frames. + + Follows the Dask serialization scheme with a picklable header (dict) and + a tuple of frames (in this case a contiguous host and device buffer). + + To enable dask support, dask serializers must be registered + + >>> from cudf_polars.experimental.dask_serialize import register + >>> register() + + Returns + ------- + header + A dict containing any picklable metadata required to reconstruct the object. + frames + Two-tuple of frames suitable for passing to `plc.contiguous_split.unpack_from_memoryviews` + """ + packed = plc.contiguous_split.pack(plc.Table([self.obj])) + column_kwargs: ColumnOptions = { + "is_sorted": self.is_sorted, + "order": self.order, + "null_order": self.null_order, + "name": self.name, + } + header: ColumnHeader = { + "column_kwargs": column_kwargs, + "frame_count": 2, + } + return header, packed.release() + @functools.cached_property def obj_scalar(self) -> plc.Scalar: """ diff --git a/python/cudf_polars/cudf_polars/containers/dataframe.py b/python/cudf_polars/cudf_polars/containers/dataframe.py index 36e0fbe370e..a605b476197 100644 --- a/python/cudf_polars/cudf_polars/containers/dataframe.py +++ b/python/cudf_polars/cudf_polars/containers/dataframe.py @@ -1,13 +1,12 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 """A dataframe, with some properties.""" from __future__ import annotations -import pickle from functools import cached_property -from typing import TYPE_CHECKING, Any, cast +from typing import TYPE_CHECKING, cast import pyarrow as pa @@ -23,6 +22,8 @@ from typing_extensions import Self + from cudf_polars.typing import ColumnOptions, DataFrameHeader + __all__: list[str] = ["DataFrame"] @@ -150,7 +151,7 @@ def from_table(cls, table: plc.Table, names: Sequence[str]) -> Self: @classmethod def deserialize( - cls, header: Mapping[str, Any], frames: tuple[memoryview, plc.gpumemoryview] + cls, header: DataFrameHeader, frames: tuple[memoryview, plc.gpumemoryview] ) -> Self: """ Create a DataFrame from a serialized representation returned by `.serialize()`. @@ -178,7 +179,7 @@ def deserialize( def serialize( self, - ) -> tuple[Mapping[str, Any], tuple[memoryview, plc.gpumemoryview]]: + ) -> tuple[DataFrameHeader, tuple[memoryview, plc.gpumemoryview]]: """ Serialize the table into header and frames. @@ -187,20 +188,20 @@ def serialize( To enable dask support, dask serializers must be registered - >>> from cudf_polars.experimental.dask_serialize import register - >>> register() + >>> from cudf_polars.experimental.dask_serialize import register + >>> register() Returns ------- header A dict containing any picklable metadata required to reconstruct the object. frames - Two-tuple of frames suitable for passing to `unpack_from_memoryviews` + Two-tuple of frames suitable for passing to `plc.contiguous_split.unpack_from_memoryviews` """ packed = plc.contiguous_split.pack(self.table) # Keyword arguments for `Column.__init__`. - columns_kwargs = [ + columns_kwargs: list[ColumnOptions] = [ { "is_sorted": col.is_sorted, "order": col.order, @@ -209,10 +210,8 @@ def serialize( } for col in self.columns ] - header = { + header: DataFrameHeader = { "columns_kwargs": columns_kwargs, - # Dask Distributed uses "type-serialized" to dispatch deserialization - "type-serialized": pickle.dumps(type(self)), "frame_count": 2, } return header, packed.release() diff --git a/python/cudf_polars/cudf_polars/experimental/dask_serialize.py b/python/cudf_polars/cudf_polars/experimental/dask_serialize.py index aae78e07690..09a9556bb31 100644 --- a/python/cudf_polars/cudf_polars/experimental/dask_serialize.py +++ b/python/cudf_polars/cudf_polars/experimental/dask_serialize.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 """Dask serialization.""" @@ -12,7 +12,7 @@ import pylibcudf as plc import rmm -from cudf_polars.containers import DataFrame +from cudf_polars.containers import Column, DataFrame __all__ = ["register"] @@ -20,8 +20,8 @@ def register() -> None: """Register dask serialization routines for DataFrames.""" - @cuda_serialize.register(DataFrame) - def _(x: DataFrame): + @cuda_serialize.register((Column, DataFrame)) + def _(x: DataFrame | Column): with log_errors(): header, frames = x.serialize() return header, list(frames) # Dask expect a list of frames @@ -32,8 +32,14 @@ def _(header, frames): assert len(frames) == 2 return DataFrame.deserialize(header, tuple(frames)) - @dask_serialize.register(DataFrame) - def _(x: DataFrame): + @cuda_deserialize.register(Column) + def _(header, frames): + with log_errors(): + assert len(frames) == 2 + return Column.deserialize(header, tuple(frames)) + + @dask_serialize.register((Column, DataFrame)) + def _(x: DataFrame | Column): with log_errors(): header, (metadata, gpudata) = x.serialize() @@ -57,3 +63,11 @@ def _(header, frames) -> DataFrame: # Copy the second frame (the gpudata in host memory) back to the gpu frames = frames[0], plc.gpumemoryview(rmm.DeviceBuffer.to_device(frames[1])) return DataFrame.deserialize(header, frames) + + @dask_deserialize.register(Column) + def _(header, frames) -> Column: + with log_errors(): + assert len(frames) == 2 + # Copy the second frame (the gpudata in host memory) back to the gpu + frames = frames[0], plc.gpumemoryview(rmm.DeviceBuffer.to_device(frames[1])) + return Column.deserialize(header, frames) diff --git a/python/cudf_polars/cudf_polars/typing/__init__.py b/python/cudf_polars/cudf_polars/typing/__init__.py index 52be130ab90..7a5795867ca 100644 --- a/python/cudf_polars/cudf_polars/typing/__init__.py +++ b/python/cudf_polars/cudf_polars/typing/__init__.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 """Typing utilities for cudf_polars.""" @@ -6,7 +6,7 @@ from __future__ import annotations from collections.abc import Hashable, Mapping -from typing import TYPE_CHECKING, Any, Literal, Protocol, TypeVar, Union +from typing import TYPE_CHECKING, Any, Literal, Protocol, TypeVar, TypedDict, Union from polars.polars import _expr_nodes as pl_expr, _ir_nodes as pl_ir @@ -145,3 +145,32 @@ def state(self) -> Mapping[str, Any]: IRTransformer: TypeAlias = GenericTransformer["ir.IR", "ir.IR"] """Protocol for transformation of IR nodes.""" + + +class ColumnOptions(TypedDict): + """ + Column constructor options. + + Notes + ----- + Used to serialize Column and DataFrame containers. + """ + + is_sorted: plc.types.Sorted + order: plc.types.Order + null_order: plc.types.NullOrder + name: str | None + + +class ColumnHeader(TypedDict): + """Column serialization header.""" + + column_kwargs: ColumnOptions + frame_count: int + + +class DataFrameHeader(TypedDict): + """DataFrame serialization header.""" + + columns_kwargs: list[ColumnOptions] + frame_count: int diff --git a/python/cudf_polars/tests/experimental/test_dask_serialize.py b/python/cudf_polars/tests/experimental/test_dask_serialize.py index e556b7e4445..e0da2e834fc 100644 --- a/python/cudf_polars/tests/experimental/test_dask_serialize.py +++ b/python/cudf_polars/tests/experimental/test_dask_serialize.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 from __future__ import annotations @@ -38,3 +38,12 @@ def test_dask_serialization_roundtrip(arrow_tbl, protocol): res = deserialize(header, frames, deserializers=[protocol]) assert_frame_equal(df.to_polars(), res.to_polars()) + + # Check that we can serialize individual columns + for column in df.columns: + expect = DataFrame([column]) + + header, frames = serialize(column, on_error="raise", serializers=[protocol]) + res = deserialize(header, frames, deserializers=[protocol]) + + assert_frame_equal(expect.to_polars(), DataFrame([res]).to_polars()) From a035ccc3342ac187943cd3d4d3d338fbbe862ed9 Mon Sep 17 00:00:00 2001 From: "Richard (Rick) Zamora" Date: Thu, 13 Feb 2025 13:00:05 -0600 Subject: [PATCH 035/129] [Bug] Fix Parquet-metadata sampling in cudf-polars (#17991) The experimental multi-GPU executor for cudf-polars always attempts to sample the metadata from three files when reading from a Parquet dataset. This is obviously a problem when there are fewer than three files to sample from. This PR fixes the trivial bug and adds test coverage. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) - GALI PREM SAGAR (https://github.com/galipremsagar) - Matthew Murray (https://github.com/Matt711) Approvers: - Matthew Murray (https://github.com/Matt711) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/17991 --- python/cudf_polars/cudf_polars/experimental/io.py | 2 +- python/cudf_polars/tests/experimental/test_scan.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/python/cudf_polars/cudf_polars/experimental/io.py b/python/cudf_polars/cudf_polars/experimental/io.py index d24ae5772c0..ba4432ecdea 100644 --- a/python/cudf_polars/cudf_polars/experimental/io.py +++ b/python/cudf_polars/cudf_polars/experimental/io.py @@ -243,7 +243,7 @@ def _sample_pq_statistics(ir: Scan) -> dict[str, float]: # Use average total_uncompressed_size of three files # TODO: Use plc.io.parquet_metadata.read_parquet_metadata - n_sample = 3 + n_sample = min(3, len(ir.paths)) column_sizes = {} ds = pa_ds.dataset(random.sample(ir.paths, n_sample), format="parquet") for i, frag in enumerate(ds.get_fragments()): diff --git a/python/cudf_polars/tests/experimental/test_scan.py b/python/cudf_polars/tests/experimental/test_scan.py index a26d751dc86..306a0daf091 100644 --- a/python/cudf_polars/tests/experimental/test_scan.py +++ b/python/cudf_polars/tests/experimental/test_scan.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 from __future__ import annotations @@ -59,8 +59,8 @@ def test_parallel_scan(tmp_path, df, fmt, scan_fn): @pytest.mark.parametrize("blocksize", [1_000, 10_000, 1_000_000]) -def test_parquet_blocksize(tmp_path, df, blocksize): - n_files = 3 +@pytest.mark.parametrize("n_files", [2, 3]) +def test_parquet_blocksize(tmp_path, df, blocksize, n_files): make_source(df, tmp_path, "parquet", n_files) q = pl.scan_parquet(tmp_path) engine = pl.GPUEngine( From 9ead47bb8161f227abd792088dabfa96ad34b8fd Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Thu, 13 Feb 2025 14:29:36 -0500 Subject: [PATCH 036/129] Make `cudf.pandas` proxy array picklable (#17929) Apart of #17490. We employ custom pickling logic for our cudf.pandas wrapped types. The logic lets us serialize and de-serialize wrapped types by serializing and de-serializing the underlying wrapped types (ie. the type of `_fsproxy_wrapped`). This pickling logic is defined in `_FinalProxy`, which is the base class of all of our "final" proxy types. The failures in the integration tests occurred because this pickling logic wasn't used for the proxy numpy array type. This is because the "final" proxy array type inherits from an additional base class: `ProxyNDarrayBase` (which contains logic to inherit from `np.ndarray` ). And it comes before `_FinalProxy` in the classes MRO, so the custom pickling is not used. Additionally, the custom pickling logic used for other proxy types is incompatible with our proxy array. So this PR defines a custom function for handling proxy array serialization. Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/17929 --- python/cudf/cudf/pandas/_wrappers/numpy.py | 18 ++++++++++++++++++ .../cudf/cudf_pandas_tests/test_cudf_pandas.py | 15 +++++++++++++++ .../tests/test_holoviews.py | 5 +---- .../tests/test_matplotlib.py | 8 +------- .../tests/test_numpy.py | 5 +---- .../tests/test_seaborn.py | 5 +---- 6 files changed, 37 insertions(+), 19 deletions(-) diff --git a/python/cudf/cudf/pandas/_wrappers/numpy.py b/python/cudf/cudf/pandas/_wrappers/numpy.py index 1fc53bbbaae..68ebe620013 100644 --- a/python/cudf/cudf/pandas/_wrappers/numpy.py +++ b/python/cudf/cudf/pandas/_wrappers/numpy.py @@ -126,6 +126,23 @@ def ndarray__array_ufunc__(self, ufunc, method, *inputs, **kwargs): return result +def ndarray__reduce__(self): + # As it stands the custom pickling logic used for all other + # proxy types is incompatible with our proxy ndarray. The pickle + # constructor we use to deserialize the other proxy types calls + # object.__new__(type) which you cannot call on subclasses of + # numpy arrays because the new array won't be created with numpy's + # specific memory management logic. Therefore, we have to handle + # serialization separately for proxy arrays. + return ( + ndarray.__new__, + ( + ndarray, + self._fsproxy_wrapped, + ), + ) + + ndarray = make_final_proxy_type( "ndarray", cupy.ndarray, @@ -140,6 +157,7 @@ def ndarray__array_ufunc__(self, ufunc, method, *inputs, **kwargs): "__cuda_array_interface__": cuda_array_interface, "__array_interface__": array_interface, "__array_ufunc__": ndarray__array_ufunc__, + "__reduce__": ndarray__reduce__, # ndarrays are unhashable "__hash__": None, # iter(cupy-array) produces an iterable of zero-dim device diff --git a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py index 800702a6544..648931f212a 100644 --- a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py +++ b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py @@ -1979,3 +1979,18 @@ def test_numpy_data_access(): actual = xs.values.data assert type(expected) is type(actual) + + +def test_pickle_round_trip_proxy_numpy_array(array): + arr, proxy_arr = array + pickled_arr = BytesIO() + pickled_proxy_arr = BytesIO() + pickle.dump(arr, pickled_arr) + pickle.dump(proxy_arr, pickled_proxy_arr) + + pickled_arr.seek(0) + pickled_proxy_arr.seek(0) + + np.testing.assert_equal( + pickle.load(pickled_proxy_arr), pickle.load(pickled_arr) + ) diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_holoviews.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_holoviews.py index 8be48953974..b42c70aa4e1 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_holoviews.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_holoviews.py @@ -1,4 +1,4 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. import holoviews as hv import numpy as np import pandas as pd @@ -71,9 +71,6 @@ def test_holoviews_heatmap(df): ) -@pytest.mark.skip( - reason="AttributeError: 'ndarray' object has no attribute '_fsproxy_wrapped'" -) def test_holoviews_histogram(df): return get_plot_info(hv.Histogram(df.values)) diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_matplotlib.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_matplotlib.py index c91808021e8..6a33666790d 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_matplotlib.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_matplotlib.py @@ -1,4 +1,4 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. import matplotlib.pyplot as plt import numpy as np import pandas as pd @@ -33,9 +33,6 @@ def assert_plots_equal(expect, got): pytestmark = pytest.mark.assert_eq(fn=assert_plots_equal) -@pytest.mark.skip( - reason="AttributeError: 'ndarray' object has no attribute '_fsproxy_wrapped'" -) def test_line(): df = pd.DataFrame({"x": [1, 2, 3, 4, 5], "y": [2, 4, 6, 8, 10]}) (data,) = plt.plot(df["x"], df["y"], marker="o", linestyle="-") @@ -43,9 +40,6 @@ def test_line(): return plt.gca() -@pytest.mark.skip( - reason="AttributeError: 'ndarray' object has no attribute '_fsproxy_wrapped'" -) def test_bar(): data = pd.Series([1, 2, 3, 4, 5], index=["a", "b", "c", "d", "e"]) ax = data.plot(kind="bar") diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_numpy.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_numpy.py index 4d35d9e8946..d090dc44092 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_numpy.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_numpy.py @@ -1,4 +1,4 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. import numpy as np import pandas as pd @@ -37,9 +37,6 @@ def test_numpy_dot(df): return np.dot(df, df.T) -@pytest.mark.skip( - reason="AttributeError: 'ndarray' object has no attribute '_fsproxy_wrapped'" -) def test_numpy_fft(sr): fft = np.fft.fft(sr) return fft diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_seaborn.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_seaborn.py index f6a8a96ae3c..02b2b1b9997 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_seaborn.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_seaborn.py @@ -1,4 +1,4 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. import pandas as pd import pytest import seaborn as sns @@ -54,9 +54,6 @@ def test_scatter(df): return ax -@pytest.mark.skip( - reason="AttributeError: 'ndarray' object has no attribute '_fsproxy_wrapped'" -) def test_lineplot_with_sns_data(): df = sns.load_dataset("flights") ax = sns.lineplot(data=df, x="month", y="passengers") From 6c281fded5590e9896a901b5e12ce3a05d510be7 Mon Sep 17 00:00:00 2001 From: Muhammad Haseeb <14217455+mhaseeb123@users.noreply.github.com> Date: Thu, 13 Feb 2025 14:10:04 -0800 Subject: [PATCH 037/129] Refactor predicate pushdown to reuse row group pruning in experimental PQ reader (#17946) Related to #17896 This PR refactors Parquet reader's predicate pushdown to separate out row group pruning with stats, reading bloom filters, and row group pruning with bloom filters. This allows reusing corresponding functionalities in the experimental PQ reader for highly selective queries (Hybrid scan) as needed. Note that no code has been added or removed in this PR. Only moved around. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Karthikeyan (https://github.com/karthikeyann) - Vukasin Milovanovic (https://github.com/vuule) URL: https://github.com/rapidsai/cudf/pull/17946 --- cpp/src/io/parquet/bloom_filter_reader.cu | 252 ++++++++------------- cpp/src/io/parquet/predicate_pushdown.cpp | 117 +++++++--- cpp/src/io/parquet/reader_impl_helpers.hpp | 122 ++++++++-- 3 files changed, 285 insertions(+), 206 deletions(-) diff --git a/cpp/src/io/parquet/bloom_filter_reader.cu b/cpp/src/io/parquet/bloom_filter_reader.cu index a883981a467..87024719d87 100644 --- a/cpp/src/io/parquet/bloom_filter_reader.cu +++ b/cpp/src/io/parquet/bloom_filter_reader.cu @@ -32,7 +32,6 @@ #include #include #include -#include #include #include @@ -163,108 +162,6 @@ struct bloom_filter_caster { } }; -/** - * @brief Collects lists of equality predicate literals in the AST expression, one list per input - * table column. This is used in row group filtering based on bloom filters. - */ -class equality_literals_collector : public ast::detail::expression_transformer { - public: - equality_literals_collector() = default; - - equality_literals_collector(ast::expression const& expr, cudf::size_type num_input_columns) - : _num_input_columns{num_input_columns} - { - _equality_literals.resize(_num_input_columns); - expr.accept(*this); - } - - /** - * @copydoc ast::detail::expression_transformer::visit(ast::literal const& ) - */ - std::reference_wrapper visit(ast::literal const& expr) override - { - return expr; - } - - /** - * @copydoc ast::detail::expression_transformer::visit(ast::column_reference const& ) - */ - std::reference_wrapper visit(ast::column_reference const& expr) override - { - CUDF_EXPECTS(expr.get_table_source() == ast::table_reference::LEFT, - "BloomfilterAST supports only left table"); - CUDF_EXPECTS(expr.get_column_index() < _num_input_columns, - "Column index cannot be more than number of columns in the table"); - return expr; - } - - /** - * @copydoc ast::detail::expression_transformer::visit(ast::column_name_reference const& ) - */ - std::reference_wrapper visit( - ast::column_name_reference const& expr) override - { - CUDF_FAIL("Column name reference is not supported in BloomfilterAST"); - } - - /** - * @copydoc ast::detail::expression_transformer::visit(ast::operation const& ) - */ - std::reference_wrapper visit(ast::operation const& expr) override - { - using cudf::ast::ast_operator; - auto const operands = expr.get_operands(); - auto const op = expr.get_operator(); - - if (auto* v = dynamic_cast(&operands[0].get())) { - // First operand should be column reference, second should be literal. - CUDF_EXPECTS(cudf::ast::detail::ast_operator_arity(op) == 2, - "Only binary operations are supported on column reference"); - auto const literal_ptr = dynamic_cast(&operands[1].get()); - CUDF_EXPECTS(literal_ptr != nullptr, - "Second operand of binary operation with column reference must be a literal"); - v->accept(*this); - - // Push to the corresponding column's literals list iff equality predicate is seen - if (op == ast_operator::EQUAL) { - auto const col_idx = v->get_column_index(); - _equality_literals[col_idx].emplace_back(const_cast(literal_ptr)); - } - } else { - // Just visit the operands and ignore any output - std::ignore = visit_operands(operands); - } - - return expr; - } - - /** - * @brief Vectors of equality literals in the AST expression, one per input table column - * - * @return Vectors of equality literals, one per input table column - */ - [[nodiscard]] std::vector> get_equality_literals() && - { - return std::move(_equality_literals); - } - - private: - std::vector> _equality_literals; - - protected: - std::vector> visit_operands( - cudf::host_span const> operands) - { - std::vector> transformed_operands; - for (auto const& operand : operands) { - auto const new_operand = operand.get().accept(*this); - transformed_operands.push_back(new_operand); - } - return transformed_operands; - } - size_type _num_input_columns; -}; - /** * @brief Converts AST expression to bloom filter membership (BloomfilterAST) expression. * This is used in row group filtering based on equality predicate. @@ -502,6 +399,17 @@ void read_bloom_filter_data(host_span const> sources } // namespace +size_t aggregate_reader_metadata::get_bloom_filter_alignment() const +{ + // Required alignment: + // https://github.com/NVIDIA/cuCollections/blob/deab5799f3e4226cb8a49acf2199c03b14941ee4/include/cuco/detail/bloom_filter/bloom_filter_impl.cuh#L55-L67 + using policy_type = cuco::arrow_filter_policy; + return alignof(cuco::bloom_filter_ref, + cuco::thread_scope_thread, + policy_type>::filter_block_type); +} + std::vector aggregate_reader_metadata::read_bloom_filters( host_span const> sources, host_span const> row_group_indices, @@ -599,55 +507,19 @@ std::vector aggregate_reader_metadata::get_parquet_types( return parquet_types; } -std::pair>>, bool> -aggregate_reader_metadata::apply_bloom_filters( - host_span const> sources, +std::optional>> aggregate_reader_metadata::apply_bloom_filters( + std::vector& bloom_filter_data, host_span const> input_row_group_indices, + host_span const> literals, size_type total_row_groups, host_span output_dtypes, - host_span output_column_schemas, + host_span equality_col_schemas, std::reference_wrapper filter, rmm::cuda_stream_view stream) const { // Number of input table columns auto const num_input_columns = static_cast(output_dtypes.size()); - // Collect equality literals for each input table column - auto const equality_literals = - equality_literals_collector{filter.get(), num_input_columns}.get_equality_literals(); - - // Collect schema indices of columns with equality predicate(s) - std::vector equality_col_schemas; - thrust::copy_if(thrust::host, - output_column_schemas.begin(), - output_column_schemas.end(), - equality_literals.begin(), - std::back_inserter(equality_col_schemas), - [](auto& eq_literals) { return not eq_literals.empty(); }); - - // Return early if no column with equality predicate(s) - if (equality_col_schemas.empty()) { return {std::nullopt, false}; } - - // Required alignment: - // https://github.com/NVIDIA/cuCollections/blob/deab5799f3e4226cb8a49acf2199c03b14941ee4/include/cuco/detail/bloom_filter/bloom_filter_impl.cuh#L55-L67 - using policy_type = cuco::arrow_filter_policy; - auto constexpr alignment = alignof(cuco::bloom_filter_ref, - cuco::thread_scope_thread, - policy_type>::filter_block_type); - - // Aligned resource adaptor to allocate bloom filter buffers with - auto aligned_mr = - rmm::mr::aligned_resource_adaptor(cudf::get_current_device_resource(), alignment); - - // Read a vector of bloom filter bitset device buffers for all columns with equality - // predicate(s) across all row groups - auto bloom_filter_data = read_bloom_filters( - sources, input_row_group_indices, equality_col_schemas, total_row_groups, stream, aligned_mr); - - // No bloom filter buffers, return early - if (bloom_filter_data.empty()) { return {std::nullopt, false}; } - // Get parquet types for the predicate columns auto const parquet_types = get_parquet_types(input_row_group_indices, equality_col_schemas); @@ -684,13 +556,13 @@ aggregate_reader_metadata::apply_bloom_filters( auto const& dtype = output_dtypes[input_col_idx]; // Skip if no equality literals for this column - if (equality_literals[input_col_idx].empty()) { return; } + if (literals[input_col_idx].empty()) { return; } // Skip if non-comparable (compound) type except string if (cudf::is_compound(dtype) and dtype.id() != cudf::type_id::STRING) { return; } // Add a column for all literals associated with an equality column - for (auto const& literal : equality_literals[input_col_idx]) { + for (auto const& literal : literals[input_col_idx]) { bloom_filter_membership_columns.emplace_back(cudf::type_dispatcher( dtype, bloom_filter_col, equality_col_idx, dtype, literal, stream)); } @@ -702,16 +574,92 @@ aggregate_reader_metadata::apply_bloom_filters( // Convert AST to BloomfilterAST expression with reference to bloom filter membership // in above `bloom_filter_membership_table` - bloom_filter_expression_converter bloom_filter_expr{ - filter.get(), num_input_columns, {equality_literals}}; + bloom_filter_expression_converter bloom_filter_expr{filter.get(), num_input_columns, {literals}}; // Filter bloom filter membership table with the BloomfilterAST expression and collect // filtered row group indices - return {collect_filtered_row_group_indices(bloom_filter_membership_table, - bloom_filter_expr.get_bloom_filter_expr(), - input_row_group_indices, - stream), - true}; + return collect_filtered_row_group_indices(bloom_filter_membership_table, + bloom_filter_expr.get_bloom_filter_expr(), + input_row_group_indices, + stream); +} + +equality_literals_collector::equality_literals_collector() = default; + +equality_literals_collector::equality_literals_collector(ast::expression const& expr, + cudf::size_type num_input_columns) + : _num_input_columns{num_input_columns} +{ + _literals.resize(_num_input_columns); + expr.accept(*this); +} + +std::reference_wrapper equality_literals_collector::visit( + ast::literal const& expr) +{ + return expr; +} + +std::reference_wrapper equality_literals_collector::visit( + ast::column_reference const& expr) +{ + CUDF_EXPECTS(expr.get_table_source() == ast::table_reference::LEFT, + "BloomfilterAST supports only left table"); + CUDF_EXPECTS(expr.get_column_index() < _num_input_columns, + "Column index cannot be more than number of columns in the table"); + return expr; +} + +std::reference_wrapper equality_literals_collector::visit( + ast::column_name_reference const& expr) +{ + CUDF_FAIL("Column name reference is not supported in BloomfilterAST"); +} + +std::reference_wrapper equality_literals_collector::visit( + ast::operation const& expr) +{ + using cudf::ast::ast_operator; + auto const operands = expr.get_operands(); + auto const op = expr.get_operator(); + + if (auto* v = dynamic_cast(&operands[0].get())) { + // First operand should be column reference, second should be literal. + CUDF_EXPECTS(cudf::ast::detail::ast_operator_arity(op) == 2, + "Only binary operations are supported on column reference"); + auto const literal_ptr = dynamic_cast(&operands[1].get()); + CUDF_EXPECTS(literal_ptr != nullptr, + "Second operand of binary operation with column reference must be a literal"); + v->accept(*this); + + // Push to the corresponding column's literals list iff equality predicate is seen + if (op == ast_operator::EQUAL) { + auto const col_idx = v->get_column_index(); + _literals[col_idx].emplace_back(const_cast(literal_ptr)); + } + } else { + // Just visit the operands and ignore any output + std::ignore = visit_operands(operands); + } + + return expr; +} + +std::vector> equality_literals_collector::get_literals() && +{ + return std::move(_literals); +} + +std::vector> +equality_literals_collector::visit_operands( + cudf::host_span const> operands) +{ + std::vector> transformed_operands; + for (auto const& operand : operands) { + auto const new_operand = operand.get().accept(*this); + transformed_operands.push_back(new_operand); + } + return transformed_operands; } } // namespace cudf::io::parquet::detail diff --git a/cpp/src/io/parquet/predicate_pushdown.cpp b/cpp/src/io/parquet/predicate_pushdown.cpp index 1508b7eef8b..e1d7dbb03b3 100644 --- a/cpp/src/io/parquet/predicate_pushdown.cpp +++ b/cpp/src/io/parquet/predicate_pushdown.cpp @@ -29,6 +29,8 @@ #include #include +#include + #include #include @@ -388,9 +390,7 @@ class stats_expression_converter : public ast::detail::expression_transformer { }; } // namespace -std::pair>>, surviving_row_group_metrics> -aggregate_reader_metadata::filter_row_groups( - host_span const> sources, +std::optional>> aggregate_reader_metadata::apply_stats_filters( host_span const> input_row_group_indices, size_type total_row_groups, host_span output_dtypes, @@ -430,14 +430,33 @@ aggregate_reader_metadata::filter_row_groups( static_cast(output_dtypes.size())}; // Filter stats table with StatsAST expression and collect filtered row group indices - auto const filtered_row_group_indices = collect_filtered_row_group_indices( + return collect_filtered_row_group_indices( stats_table, stats_expr.get_stats_expr(), input_row_group_indices, stream); +} + +std::pair>>, surviving_row_group_metrics> +aggregate_reader_metadata::filter_row_groups( + host_span const> sources, + host_span const> input_row_group_indices, + size_type total_row_groups, + host_span output_dtypes, + host_span output_column_schemas, + std::reference_wrapper filter, + rmm::cuda_stream_view stream) const +{ + // Apply stats filtering on input row groups + auto const stats_filtered_row_groups = apply_stats_filters(input_row_group_indices, + total_row_groups, + output_dtypes, + output_column_schemas, + filter, + stream); // Number of surviving row groups after applying stats filter auto const num_stats_filtered_row_groups = - filtered_row_group_indices.has_value() - ? std::accumulate(filtered_row_group_indices.value().cbegin(), - filtered_row_group_indices.value().cend(), + stats_filtered_row_groups.has_value() + ? std::accumulate(stats_filtered_row_groups.value().cbegin(), + stats_filtered_row_groups.value().cend(), size_type{0}, [](auto& sum, auto const& per_file_row_groups) { return sum + per_file_row_groups.size(); @@ -446,37 +465,75 @@ aggregate_reader_metadata::filter_row_groups( // Span of row groups to apply bloom filtering on. auto const bloom_filter_input_row_groups = - filtered_row_group_indices.has_value() - ? host_span const>(filtered_row_group_indices.value()) + stats_filtered_row_groups.has_value() + ? host_span const>(stats_filtered_row_groups.value()) : input_row_group_indices; - // Apply bloom filtering on the bloom filter input row groups - auto const [bloom_filtered_row_groups, bloom_filters_exist] = - apply_bloom_filters(sources, - bloom_filter_input_row_groups, - num_stats_filtered_row_groups, - output_dtypes, - output_column_schemas, - filter, - stream); + // Collect equality literals for each input table column for bloom filtering + auto const equality_literals = + equality_literals_collector{filter.get(), static_cast(output_dtypes.size())} + .get_literals(); + + // Collect schema indices of columns with equality predicate(s) + std::vector equality_col_schemas; + thrust::copy_if(thrust::host, + output_column_schemas.begin(), + output_column_schemas.end(), + equality_literals.begin(), + std::back_inserter(equality_col_schemas), + [](auto& eq_literals) { return not eq_literals.empty(); }); + + // Return early if no column with equality predicate(s) + if (equality_col_schemas.empty()) { + return {stats_filtered_row_groups, + {std::make_optional(num_stats_filtered_row_groups), std::nullopt}}; + } + + // Aligned resource adaptor to allocate bloom filter buffers with + auto aligned_mr = rmm::mr::aligned_resource_adaptor(cudf::get_current_device_resource(), + get_bloom_filter_alignment()); + + // Read a vector of bloom filter bitset device buffers for all columns with equality + // predicate(s) across all row groups + auto bloom_filter_data = read_bloom_filters(sources, + bloom_filter_input_row_groups, + equality_col_schemas, + num_stats_filtered_row_groups, + stream, + aligned_mr); + + // No bloom filter buffers, return early + if (bloom_filter_data.empty()) { + return {stats_filtered_row_groups, + {std::make_optional(num_stats_filtered_row_groups), std::nullopt}}; + } + + // Apply bloom filtering on the output row groups from stats filter + auto const bloom_filtered_row_groups = apply_bloom_filters(bloom_filter_data, + bloom_filter_input_row_groups, + equality_literals, + num_stats_filtered_row_groups, + output_dtypes, + equality_col_schemas, + filter, + stream); // Number of surviving row groups after applying bloom filter auto const num_bloom_filtered_row_groups = - bloom_filters_exist - ? (bloom_filtered_row_groups.has_value() - ? std::make_optional(std::accumulate(bloom_filtered_row_groups.value().cbegin(), - bloom_filtered_row_groups.value().cend(), - size_type{0}, - [](auto& sum, auto const& per_file_row_groups) { - return sum + per_file_row_groups.size(); - })) - : std::make_optional(num_stats_filtered_row_groups)) - : std::nullopt; + bloom_filtered_row_groups.has_value() + ? std::accumulate(bloom_filtered_row_groups.value().cbegin(), + bloom_filtered_row_groups.value().cend(), + size_type{0}, + [](auto& sum, auto const& per_file_row_groups) { + return sum + per_file_row_groups.size(); + }) + : num_stats_filtered_row_groups; // Return bloom filtered row group indices iff collected return { - bloom_filtered_row_groups.has_value() ? bloom_filtered_row_groups : filtered_row_group_indices, - {std::make_optional(num_stats_filtered_row_groups), num_bloom_filtered_row_groups}}; + bloom_filtered_row_groups.has_value() ? bloom_filtered_row_groups : stats_filtered_row_groups, + {std::make_optional(num_stats_filtered_row_groups), + std::make_optional(num_bloom_filtered_row_groups)}}; } // convert column named expression to column index reference expression diff --git a/cpp/src/io/parquet/reader_impl_helpers.hpp b/cpp/src/io/parquet/reader_impl_helpers.hpp index c4372b2c1ff..f08ba5f8b85 100644 --- a/cpp/src/io/parquet/reader_impl_helpers.hpp +++ b/cpp/src/io/parquet/reader_impl_helpers.hpp @@ -203,6 +203,11 @@ class aggregate_reader_metadata { */ void column_info_for_row_group(row_group_info& rg_info, size_type chunk_start_row) const; + /** + * @brief Returns the required alignment for bloom filter buffers + */ + [[nodiscard]] size_t get_bloom_filter_alignment() const; + /** * @brief Reads bloom filter bitsets for the specified columns from the given lists of row * groups. @@ -237,6 +242,50 @@ class aggregate_reader_metadata { host_span const> row_group_indices, host_span column_schemas) const; + /** + * @brief Filters the row groups using stats filter + * + * @param input_row_group_indices Lists of input row groups, one per source + * @param total_row_groups Total number of row groups in `input_row_group_indices` + * @param output_dtypes Datatypes of output columns + * @param output_column_schemas schema indices of output columns + * @param filter AST expression to filter row groups based on bloom filter membership + * @param stream CUDA stream used for device memory operations and kernel launches + * + * @return Filtered row group indices if any is filtered + */ + [[nodiscard]] std::optional>> apply_stats_filters( + host_span const> input_row_group_indices, + size_type total_row_groups, + host_span output_dtypes, + host_span output_column_schemas, + std::reference_wrapper filter, + rmm::cuda_stream_view stream) const; + + /** + * @brief Filters the row groups using bloom filters + * + * @param bloom_filter_data Bloom filter data device buffers for each input row group + * @param input_row_group_indices Lists of input row groups, one per source + * @param literals Lists of equality literals, one per each input row group + * @param total_row_groups Total number of row groups in `input_row_group_indices` + * @param output_dtypes Datatypes of output columns + * @param equality_col_schemas schema indices of equality columns only + * @param filter AST expression to filter row groups based on bloom filter membership + * @param stream CUDA stream used for device memory operations and kernel launches + * + * @return Filtered row group indices if any is filtered + */ + [[nodiscard]] std::optional>> apply_bloom_filters( + std::vector& bloom_filter_data, + host_span const> input_row_group_indices, + host_span const> literals, + size_type total_row_groups, + host_span output_dtypes, + host_span equality_col_schemas, + std::reference_wrapper filter, + rmm::cuda_stream_view stream) const; + public: aggregate_reader_metadata(host_span const> sources, bool use_arrow_schema, @@ -363,7 +412,7 @@ class aggregate_reader_metadata { [[nodiscard]] std::vector get_pandas_index_names() const; /** - * @brief Filters the row groups based on predicate filter + * @brief Filters the row groups using stats and bloom filters based on predicate filter * * @param sources Lists of input datasources * @param input_row_group_indices Lists of input row groups, one per source @@ -385,29 +434,6 @@ class aggregate_reader_metadata { std::reference_wrapper filter, rmm::cuda_stream_view stream) const; - /** - * @brief Filters the row groups using bloom filters - * - * @param sources Dataset sources - * @param input_row_group_indices Lists of input row groups, one per source - * @param total_row_groups Total number of row groups in `input_row_group_indices` - * @param output_dtypes Datatypes of output columns - * @param output_column_schemas schema indices of output columns - * @param filter AST expression to filter row groups based on bloom filter membership - * @param stream CUDA stream used for device memory operations and kernel launches - * - * @return A pair of filtered row group indices if any is filtered, and a boolean indicating if - * bloom filtering was applied - */ - [[nodiscard]] std::pair>>, bool> - apply_bloom_filters(host_span const> sources, - host_span const> input_row_group_indices, - size_type total_row_groups, - host_span output_dtypes, - host_span output_column_schemas, - std::reference_wrapper filter, - rmm::cuda_stream_view stream) const; - /** * @brief Filters and reduces down to a selection of row groups * @@ -513,6 +539,54 @@ class named_to_reference_converter : public ast::detail::expression_transformer std::list _operators; }; +/** + * @brief Collects lists of equality predicate literals in the AST expression, one list per input + * table column. This is used in row group filtering based on bloom filters. + */ +class equality_literals_collector : public ast::detail::expression_transformer { + public: + equality_literals_collector(); + + equality_literals_collector(ast::expression const& expr, cudf::size_type num_input_columns); + + /** + * @copydoc ast::detail::expression_transformer::visit(ast::literal const& ) + */ + std::reference_wrapper visit(ast::literal const& expr) override; + + /** + * @copydoc ast::detail::expression_transformer::visit(ast::column_reference const& ) + */ + std::reference_wrapper visit(ast::column_reference const& expr) override; + + /** + * @copydoc ast::detail::expression_transformer::visit(ast::column_name_reference const& ) + */ + std::reference_wrapper visit( + ast::column_name_reference const& expr) override; + + /** + * @copydoc ast::detail::expression_transformer::visit(ast::operation const& ) + */ + std::reference_wrapper visit(ast::operation const& expr) override; + + /** + * @brief Vectors of equality literals in the AST expression, one per input table column + * + * @return Vectors of equality literals, one per input table column + */ + [[nodiscard]] std::vector> get_literals() &&; + + protected: + std::vector> visit_operands( + cudf::host_span const> operands); + + size_type _num_input_columns; + + private: + std::vector> _literals; +}; + /** * @brief Get the column names in expression object * From e4209d3cb6ee922866e9a6c2a507ca48fc072531 Mon Sep 17 00:00:00 2001 From: "Richard (Rick) Zamora" Date: Fri, 14 Feb 2025 09:14:13 -0600 Subject: [PATCH 038/129] Make most cudf-polars `Node` objects pickleable (#17998) Adds some necessary groundwork for distributed execution of cudf-polars on a Dask-CUDA cluster. The general approach is to make all `Expr` (`Node`) objects pickleable as long as the constructor arguments for the class are also pickleable. Since `_non_child_args` may also contain non-`Expr` objects that are not serializable, this PR also introduces a few "wrapper" classes (e.g. `AggInfos` and `Predicate`). These wrapper classes follow the same approach as `Expr` nodes for serialization. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Matthew Murray (https://github.com/Matt711) - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/17998 --- .../cudf_polars/dsl/expressions/literal.py | 12 +-- python/cudf_polars/cudf_polars/dsl/ir.py | 76 ++++++++++++++----- .../cudf_polars/cudf_polars/dsl/nodebase.py | 9 ++- .../cudf_polars/cudf_polars/dsl/translate.py | 5 +- .../tests/experimental/test_parallel.py | 52 ++++++++++++- 5 files changed, 125 insertions(+), 29 deletions(-) diff --git a/python/cudf_polars/cudf_polars/dsl/expressions/literal.py b/python/cudf_polars/cudf_polars/dsl/expressions/literal.py index 8528e66c69c..b2007bcc6f0 100644 --- a/python/cudf_polars/cudf_polars/dsl/expressions/literal.py +++ b/python/cudf_polars/cudf_polars/dsl/expressions/literal.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 # TODO: remove need for this # ruff: noqa: D101 @@ -8,21 +8,16 @@ from typing import TYPE_CHECKING, Any -import pyarrow as pa - import pylibcudf as plc from cudf_polars.containers import Column from cudf_polars.dsl.expressions.base import AggInfo, ExecutionContext, Expr -from cudf_polars.utils import dtypes if TYPE_CHECKING: from collections.abc import Hashable, Mapping import pyarrow as pa - import polars as pl - from cudf_polars.containers import DataFrame __all__ = ["Literal", "LiteralColumn"] @@ -61,10 +56,9 @@ class LiteralColumn(Expr): _non_child = ("dtype", "value") value: pa.Array[Any] - def __init__(self, dtype: plc.DataType, value: pl.Series) -> None: + def __init__(self, dtype: plc.DataType, value: pa.Array) -> None: self.dtype = dtype - data = value.to_arrow() - self.value = data.cast(dtypes.downcast_arrow_lists(data.type)) + self.value = value self.children = () self.is_pointwise = True diff --git a/python/cudf_polars/cudf_polars/dsl/ir.py b/python/cudf_polars/cudf_polars/dsl/ir.py index 78bf10fdac7..8f12a4a7570 100644 --- a/python/cudf_polars/cudf_polars/dsl/ir.py +++ b/python/cudf_polars/cudf_polars/dsl/ir.py @@ -716,7 +716,11 @@ def __init__( self.df = df self.projection = tuple(projection) if projection is not None else None self.config_options = config_options - self._non_child_args = (schema, df, self.projection) + self._non_child_args = ( + schema, + pl.DataFrame._from_pydf(df), + self.projection, + ) self.children = () def get_hashable(self) -> Hashable: @@ -743,10 +747,9 @@ def do_evaluate( projection: tuple[str, ...] | None, ) -> DataFrame: """Evaluate and return a dataframe.""" - pdf = pl.DataFrame._from_pydf(df) if projection is not None: - pdf = pdf.select(projection) - df = DataFrame.from_polars(pdf) + df = df.select(projection) + df = DataFrame.from_polars(df) assert all( c.obj.type() == dtype for c, dtype in zip(df.columns, schema.values(), strict=True) @@ -827,6 +830,28 @@ def do_evaluate( class GroupBy(IR): """Perform a groupby.""" + class AggInfos: + """Serializable wrapper for GroupBy aggregation info.""" + + agg_requests: Sequence[expr.NamedExpr] + agg_infos: Sequence[expr.AggInfo] + + def __init__(self, agg_requests: Sequence[expr.NamedExpr]): + self.agg_requests = tuple(agg_requests) + self.agg_infos = [req.collect_agg(depth=0) for req in self.agg_requests] + + def __reduce__(self): + """Pickle an AggInfos object.""" + return (type(self), (self.agg_requests,)) + + class GroupbyOptions: + """Serializable wrapper for polars GroupbyOptions.""" + + def __init__(self, polars_groupby_options: Any): + self.dynamic = polars_groupby_options.dynamic + self.rolling = polars_groupby_options.rolling + self.slice = polars_groupby_options.slice + __slots__ = ( "agg_infos", "agg_requests", @@ -841,7 +866,7 @@ class GroupBy(IR): """Aggregation expressions.""" maintain_order: bool """Preserve order in groupby.""" - options: Any + options: GroupbyOptions """Arbitrary options.""" def __init__( @@ -857,7 +882,7 @@ def __init__( self.keys = tuple(keys) self.agg_requests = tuple(agg_requests) self.maintain_order = maintain_order - self.options = options + self.options = self.GroupbyOptions(options) self.children = (df,) if self.options.rolling: raise NotImplementedError( @@ -867,13 +892,12 @@ def __init__( raise NotImplementedError("dynamic group by") if any(GroupBy.check_agg(a.value) > 1 for a in self.agg_requests): raise NotImplementedError("Nested aggregations in groupby") - self.agg_infos = [req.collect_agg(depth=0) for req in self.agg_requests] self._non_child_args = ( self.keys, self.agg_requests, maintain_order, - options, - self.agg_infos, + self.options, + self.AggInfos(self.agg_requests), ) @staticmethod @@ -910,8 +934,8 @@ def do_evaluate( keys_in: Sequence[expr.NamedExpr], agg_requests: Sequence[expr.NamedExpr], maintain_order: bool, # noqa: FBT001 - options: Any, - agg_infos: Sequence[expr.AggInfo], + options: GroupbyOptions, + agg_info_wrapper: AggInfos, df: DataFrame, ): """Evaluate and return a dataframe.""" @@ -931,7 +955,7 @@ def do_evaluate( # TODO: uniquify requests = [] replacements: list[expr.Expr] = [] - for info in agg_infos: + for info in agg_info_wrapper.agg_infos: for pre_eval, req, rep in info.requests: if pre_eval is None: # A count aggregation, doesn't touch the column, @@ -1002,6 +1026,20 @@ def do_evaluate( class ConditionalJoin(IR): """A conditional inner join of two dataframes on a predicate.""" + class Predicate: + """Serializable wrapper for a predicate expression.""" + + predicate: expr.Expr + ast: plc.expressions.Expression + + def __init__(self, predicate: expr.Expr): + self.predicate = predicate + self.ast = to_ast(predicate) + + def __reduce__(self): + """Pickle a Predicate object.""" + return (type(self), (self.predicate,)) + __slots__ = ("ast_predicate", "options", "predicate") _non_child = ("schema", "predicate", "options") predicate: expr.Expr @@ -1034,22 +1072,22 @@ def __init__( self.predicate = predicate self.options = options self.children = (left, right) - self.ast_predicate = to_ast(predicate) + predicate_wrapper = self.Predicate(predicate) _, join_nulls, zlice, suffix, coalesce, maintain_order = self.options # Preconditions from polars assert not join_nulls assert not coalesce assert maintain_order == "none" - if self.ast_predicate is None: + if predicate_wrapper.ast is None: raise NotImplementedError( f"Conditional join with predicate {predicate}" ) # pragma: no cover; polars never delivers expressions we can't handle - self._non_child_args = (self.ast_predicate, zlice, suffix, maintain_order) + self._non_child_args = (predicate_wrapper, zlice, suffix, maintain_order) @classmethod def do_evaluate( cls, - predicate: plc.expressions.Expression, + predicate_wrapper: Predicate, zlice: tuple[int, int] | None, suffix: str, maintain_order: Literal["none", "left", "right", "left_right", "right_left"], @@ -1057,7 +1095,11 @@ def do_evaluate( right: DataFrame, ) -> DataFrame: """Evaluate and return a dataframe.""" - lg, rg = plc.join.conditional_inner_join(left.table, right.table, predicate) + lg, rg = plc.join.conditional_inner_join( + left.table, + right.table, + predicate_wrapper.ast, + ) left = DataFrame.from_table( plc.copying.gather( left.table, lg, plc.copying.OutOfBoundsPolicy.DONT_CHECK diff --git a/python/cudf_polars/cudf_polars/dsl/nodebase.py b/python/cudf_polars/cudf_polars/dsl/nodebase.py index dd5c40a00be..4f2ccb77d91 100644 --- a/python/cudf_polars/cudf_polars/dsl/nodebase.py +++ b/python/cudf_polars/cudf_polars/dsl/nodebase.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 """Base class for IR nodes, and utilities.""" @@ -58,6 +58,13 @@ def reconstruct(self, children: Sequence[T]) -> Self: """ return type(self)(*self._ctor_arguments(children)) + def __reduce__(self): + """Pickle a Node object.""" + return ( + type(self), + self._ctor_arguments(self.children), + ) + def get_hashable(self) -> Hashable: """ Return a hashable object for the node. diff --git a/python/cudf_polars/cudf_polars/dsl/translate.py b/python/cudf_polars/cudf_polars/dsl/translate.py index 966c7fd7be7..4ed36e463f3 100644 --- a/python/cudf_polars/cudf_polars/dsl/translate.py +++ b/python/cudf_polars/cudf_polars/dsl/translate.py @@ -651,7 +651,10 @@ def _(node: pl_expr.Window, translator: Translator, dtype: plc.DataType) -> expr @_translate_expr.register def _(node: pl_expr.Literal, translator: Translator, dtype: plc.DataType) -> expr.Expr: if isinstance(node.value, plrs.PySeries): - return expr.LiteralColumn(dtype, pl.Series._from_pyseries(node.value)) + data = pl.Series._from_pyseries(node.value).to_arrow() + return expr.LiteralColumn( + dtype, data.cast(dtypes.downcast_arrow_lists(data.type)) + ) value = pa.scalar(node.value, type=plc.interop.to_arrow(dtype)) return expr.Literal(dtype, value) diff --git a/python/cudf_polars/tests/experimental/test_parallel.py b/python/cudf_polars/tests/experimental/test_parallel.py index d46ab88eebf..3145549e1bd 100644 --- a/python/cudf_polars/tests/experimental/test_parallel.py +++ b/python/cudf_polars/tests/experimental/test_parallel.py @@ -1,12 +1,19 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 from __future__ import annotations +import pickle + +import pytest + import polars as pl from polars import GPUEngine from polars.testing import assert_frame_equal +from cudf_polars import Translator +from cudf_polars.dsl.traversal import traversal + def test_evaluate_dask(): df = pl.LazyFrame({"a": [1, 2, 3], "b": [3, 4, 5], "c": [5, 6, 7], "d": [7, 9, 8]}) @@ -19,3 +26,46 @@ def test_evaluate_dask(): ) assert_frame_equal(expected, got_gpu) assert_frame_equal(expected, got_dask) + + +@pytest.mark.parametrize( + "agg", + [ + pl.col("int").max(), + # Check LiteralColumn serialization + pl.Series("value", [[4, 5, 6]], dtype=pl.List(pl.Int32)), + ], +) +def test_pickle_groupby_args(agg): + df = pl.LazyFrame( + { + "key": [1, 1, 1, 2, 3, 1, 4, 6, 7], + "int": [1, 2, 3, 4, 5, 6, 7, 8, 9], + "float": [7.0, 1, 2, 3, 4, 5, 6, 7, 8], + } + ) + q = df.group_by(pl.col("key")).agg(agg) + ir = Translator(q._ldf.visit(), GPUEngine()).translate_ir() + for node in traversal([ir]): + pickle.loads(pickle.dumps(node._non_child_args)) + + +def test_pickle_conditional_join_args(): + left = pl.LazyFrame( + { + "a": [1, 2, 3, 1, None], + "b": [1, 2, 3, 4, 5], + "c": [2, 3, 4, 5, 6], + } + ) + right = pl.LazyFrame( + { + "a": [1, 4, 3, 7, None, None, 1], + "c": [2, 3, 4, 5, 6, 7, 8], + "d": [6, None, 7, 8, -1, 2, 4], + } + ) + q = left.join_where(right, pl.col("a") < pl.col("a_right")) + ir = Translator(q._ldf.visit(), GPUEngine()).translate_ir() + for node in traversal([ir]): + pickle.loads(pickle.dumps(node._non_child_args)) From c3d6b4c6623ea3236212276ac481a065ac2435e8 Mon Sep 17 00:00:00 2001 From: GALI PREM SAGAR Date: Fri, 14 Feb 2025 12:18:18 -0600 Subject: [PATCH 039/129] Fix pickle and unpickling for all objects (#17980) Fixes: #15459 This PR fixes hangs in pickle and unpickling code by registering custom pickling and unpickling methods for all proxied types. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17980 --- python/cudf/cudf/pandas/_wrappers/pandas.py | 124 ++++++++++++++---- python/cudf/cudf/pandas/fast_slow_proxy.py | 2 +- .../cudf/pandas/scripts/run-pandas-tests.sh | 5 +- .../cudf_pandas_tests/test_cudf_pandas.py | 22 ++-- 4 files changed, 113 insertions(+), 40 deletions(-) diff --git a/python/cudf/cudf/pandas/_wrappers/pandas.py b/python/cudf/cudf/pandas/_wrappers/pandas.py index c65e058cd62..d539f8038b8 100644 --- a/python/cudf/cudf/pandas/_wrappers/pandas.py +++ b/python/cudf/cudf/pandas/_wrappers/pandas.py @@ -1712,30 +1712,6 @@ def holiday_calendar_factory_wrapper(*args, **kwargs): ) -# timestamps and timedeltas are not proxied, but non-proxied -# pandas types are currently not picklable. Thus, we define -# custom reducer/unpicker functions for these types: -def _reduce_obj(obj): - from cudf.pandas.module_accelerator import disable_module_accelerator - - with disable_module_accelerator(): - # args can contain objects that are unpicklable - # when the module accelerator is disabled - # (freq is of a proxy type): - pickled_args = pickle.dumps(obj.__reduce__()) - - return _unpickle_obj, (pickled_args,) - - -def _unpickle_obj(pickled_args): - from cudf.pandas.module_accelerator import disable_module_accelerator - - with disable_module_accelerator(): - unpickler, args = pickle.loads(pickled_args) - obj = unpickler(*args) - return obj - - # Save the original __init__ methods _original_Series_init = cudf.Series.__init__ _original_DataFrame_init = cudf.DataFrame.__init__ @@ -1893,6 +1869,106 @@ def initial_setup(): cudf.set_option("mode.pandas_compatible", True) +def _reduce_obj(obj): + from cudf.pandas.module_accelerator import disable_module_accelerator + + with disable_module_accelerator(): + pickled_args = pickle.dumps(obj.__reduce__()) + + return _unpickle_obj, (pickled_args,) + + +def _unpickle_obj(pickled_args): + from cudf.pandas.module_accelerator import disable_module_accelerator + + with disable_module_accelerator(): + unpickler, args = pickle.loads(pickled_args) + obj = unpickler(*args) + return obj + + +def _generic_reduce_obj(obj, unpickle_func): + from cudf.pandas.module_accelerator import disable_module_accelerator + + with disable_module_accelerator(): + pickled_args = pickle.dumps(obj.__reduce__()) + + return unpickle_func, (pickled_args,) + + +def _frame_unpickle_obj(pickled_args): + from cudf.pandas.module_accelerator import disable_module_accelerator + + with disable_module_accelerator(): + unpickled_intermediate = pickle.loads(pickled_args) + reconstructor_func = unpickled_intermediate[0] + obj = reconstructor_func(*unpickled_intermediate[1]) + obj.__setstate__(unpickled_intermediate[2]) + return obj + + +def _index_unpickle_obj(pickled_args): + from cudf.pandas.module_accelerator import disable_module_accelerator + + with disable_module_accelerator(): + unpickled_intermediate = pickle.loads(pickled_args) + reconstructor_func = unpickled_intermediate[0] + obj = reconstructor_func(*unpickled_intermediate[1]) + + return obj + + +def _reduce_offset_obj(obj): + from cudf.pandas.module_accelerator import disable_module_accelerator + + with disable_module_accelerator(): + pickled_args = pickle.dumps(obj.__getstate__()) + + return _unpickle_offset_obj, (pickled_args,) + + +def _unpickle_offset_obj(pickled_args): + from cudf.pandas.module_accelerator import disable_module_accelerator + + with disable_module_accelerator(): + data = pickle.loads(pickled_args) + data.pop("_offset") + data.pop("_use_relativedelta") + obj = pd._libs.tslibs.offsets.DateOffset(**data) + return obj + + copyreg.dispatch_table[pd.Timestamp] = _reduce_obj # same reducer/unpickler can be used for Timedelta: copyreg.dispatch_table[pd.Timedelta] = _reduce_obj + +# TODO: Need to find a way to unpickle cross-version(old) pickled objects. +# Register custom reducer/unpickler functions for pandas objects +# so that they can be pickled/unpickled correctly: +copyreg.dispatch_table[pd.Series] = lambda obj: _generic_reduce_obj( + obj, _frame_unpickle_obj +) +copyreg.dispatch_table[pd.DataFrame] = lambda obj: _generic_reduce_obj( + obj, _frame_unpickle_obj +) + +copyreg.dispatch_table[pd.Index] = lambda obj: _generic_reduce_obj( + obj, _index_unpickle_obj +) +copyreg.dispatch_table[pd.RangeIndex] = lambda obj: _generic_reduce_obj( + obj, _index_unpickle_obj +) +copyreg.dispatch_table[pd.DatetimeIndex] = lambda obj: _generic_reduce_obj( + obj, _index_unpickle_obj +) +copyreg.dispatch_table[pd.TimedeltaIndex] = lambda obj: _generic_reduce_obj( + obj, _index_unpickle_obj +) +copyreg.dispatch_table[pd.CategoricalIndex] = lambda obj: _generic_reduce_obj( + obj, _index_unpickle_obj +) +copyreg.dispatch_table[pd.MultiIndex] = lambda obj: _generic_reduce_obj( + obj, _index_unpickle_obj +) + +copyreg.dispatch_table[pd._libs.tslibs.offsets.DateOffset] = _reduce_offset_obj diff --git a/python/cudf/cudf/pandas/fast_slow_proxy.py b/python/cudf/cudf/pandas/fast_slow_proxy.py index 46df2b047a4..45944452c17 100644 --- a/python/cudf/cudf/pandas/fast_slow_proxy.py +++ b/python/cudf/cudf/pandas/fast_slow_proxy.py @@ -112,7 +112,7 @@ def __init__(self, type_): self._type = type_ def __call__(self): - return object.__new__(self._type) + return object.__new__(get_final_type_map().get(self._type, self._type)) _DELETE = object() diff --git a/python/cudf/cudf/pandas/scripts/run-pandas-tests.sh b/python/cudf/cudf/pandas/scripts/run-pandas-tests.sh index fe8a0ef24f3..9ee89787cb1 100755 --- a/python/cudf/cudf/pandas/scripts/run-pandas-tests.sh +++ b/python/cudf/cudf/pandas/scripts/run-pandas-tests.sh @@ -24,8 +24,7 @@ PANDAS_VERSION=$(python -c "import pandas; print(pandas.__version__)") # tests/io/test_clipboard.py::TestClipboard crashes pytest workers (possibly due to fixture patching clipboard functionality) PYTEST_IGNORES="--ignore=tests/io/parser/common/test_read_errors.py \ ---ignore=tests/io/test_clipboard.py \ ---ignore=tests/io/test_pickle.py" +--ignore=tests/io/test_clipboard.py" mkdir -p pandas-testing cd pandas-testing @@ -138,7 +137,7 @@ and not test_array_tz" # TODO: Remove "not db" once a postgres & mysql container is set up on the CI PANDAS_CI="1" timeout 90m python -m pytest -p cudf.pandas \ -v -m "not single_cpu and not db" \ - -k "$TEST_THAT_NEED_MOTO_SERVER and $TEST_THAT_CRASH_PYTEST_WORKERS and not test_groupby_raises_category_on_category and not test_constructor_no_pandas_array and not test_is_monotonic_na and not test_index_contains and not test_index_contains and not test_frame_op_subclass_nonclass_constructor and not test_round_trip_current" \ + -k "$TEST_THAT_NEED_MOTO_SERVER and $TEST_THAT_CRASH_PYTEST_WORKERS and not test_groupby_raises_category_on_category and not test_constructor_no_pandas_array and not test_is_monotonic_na and not test_index_contains and not test_index_contains and not test_frame_op_subclass_nonclass_constructor and not test_round_trip_current and not test_pickle_frame_v124_unpickle_130" \ --import-mode=importlib \ ${PYTEST_IGNORES} \ "$@" || [ $? = 1 ] # Exit success if exit code was 1 (permit test failures but not other errors) diff --git a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py index 648931f212a..47de8fb1435 100644 --- a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py +++ b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py @@ -1095,6 +1095,7 @@ def test_np_array_of_timestamps(): xpd.Series([1, 2, 3]), # Index (doesn't support nullary construction) xpd.Index([1, 2, 3]), + xpd.RangeIndex(0, 10), xpd.Index(["a", "b", "c"]), # Complex index xpd.to_datetime( @@ -1104,6 +1105,8 @@ def test_np_array_of_timestamps(): datetime.datetime(2018, 1, 1), ] ), + xpd.TimedeltaIndex([100, 200, 300], dtype="timedelta64[ns]"), + xpd.MultiIndex.from_tuples([(1, 2), (3, 4)]), # Objects where the underlying store is the slow type. xpd.Series(["a", 2, 3]), xpd.Index(["a", 2, 3]), @@ -1115,18 +1118,13 @@ def test_np_array_of_timestamps(): xpd.Timedelta(1, "D"), ], ) -def test_pickle(obj): +@pytest.mark.parametrize("pickle_func", [pickle.dump, xpd.to_pickle]) +@pytest.mark.parametrize("read_pickle_func", [pickle.load, xpd.read_pickle]) +def test_pickle(obj, pickle_func, read_pickle_func): with tempfile.TemporaryFile() as f: - pickle.dump(obj, f) + pickle_func(obj, f) f.seek(0) - copy = pickle.load(f) - - tm.assert_equal(obj, copy) - - with tempfile.TemporaryFile() as f: - xpd.to_pickle(obj, f) - f.seek(0) - copy = xpd.read_pickle(f) + copy = read_pickle_func(f) tm.assert_equal(obj, copy) @@ -1552,8 +1550,8 @@ def mock_mean_none(self, *args, **kwargs): monkeypatch.setattr(xpd.Series.mean, "_fsproxy_slow", pd_mean) -def test_excelwriter_pathlike(): - assert isinstance(pd.ExcelWriter("foo.xlsx"), os.PathLike) +def test_excelwriter_pathlike(tmpdir): + assert isinstance(pd.ExcelWriter(tmpdir.join("foo.xlsx")), os.PathLike) def test_is_proxy_object(): From f592fe01b4f5e620f6cc34bfaf1690214b93f457 Mon Sep 17 00:00:00 2001 From: Shruti Shivakumar Date: Fri, 14 Feb 2025 14:46:37 -0800 Subject: [PATCH 040/129] Expose JSON reader options to builder in pylibcudf (#17866) This PR adds all JsonReaderOptionsBuilder to pylibcudf. Authors: - Shruti Shivakumar (https://github.com/shrshi) - Lawrence Mitchell (https://github.com/wence-) - Matthew Murray (https://github.com/Matt711) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - Lawrence Mitchell (https://github.com/wence-) - Karthikeyan (https://github.com/karthikeyann) URL: https://github.com/rapidsai/cudf/pull/17866 --- python/cudf/cudf/utils/ioutils.py | 16 ++ python/pylibcudf/pylibcudf/io/json.pxd | 23 +- python/pylibcudf/pylibcudf/io/json.pyi | 18 +- python/pylibcudf/pylibcudf/io/json.pyx | 269 ++++++++++++++++-- .../pylibcudf/pylibcudf/libcudf/io/json.pxd | 8 +- 5 files changed, 308 insertions(+), 26 deletions(-) diff --git a/python/cudf/cudf/utils/ioutils.py b/python/cudf/cudf/utils/ioutils.py index e2e60ea1bf0..26d5aee8896 100644 --- a/python/cudf/cudf/utils/ioutils.py +++ b/python/cudf/cudf/utils/ioutils.py @@ -771,6 +771,22 @@ - ``'error'``, raise an Exception when a bad line is encountered. - ``'recover'``, fills the row with when a bad line is encountered. +**kwargs : Additional parameters to be passed to the JSON reader. These are experimental features subject to change. + - ``'normalize_single_quotes'``, normalize single quotes to double quotes in the input buffer + - ``'normalize_whitespace'``, normalize unquoted whitespace in input buffer + - ``'delimiter'``, delimiter separating records in JSONL inputs + - ``'experimental'``, whether to enable experimental features. + When set to true, experimental features, such as the new column tree + construction, utf-8 matching of field names will be enabled. + - ``'na_values'``, sets additional values to recognize as null values. + - ``'nonnumeric_numbers'``, set whether unquoted number values should be allowed NaN, +INF, -INF, +Infinity, + Infinity, and -Infinity. Strict validation must be enabled for this to work. + - ``'nonnumeric_numbers'``, set whether leading zeros are allowed in numeric values. Strict validation + must be enabled for this to work. + - ``'strict_validation'``, set whether strict validation is enabled or not + - ``'unquoted_control_chars'``, set whether in a quoted string should characters greater than or equal to 0 + and less than 32 be allowed without some form of escaping. Strict validation + must be enabled for this to work. Returns ------- result : Series or DataFrame, depending on the value of `typ`. diff --git a/python/pylibcudf/pylibcudf/io/json.pxd b/python/pylibcudf/pylibcudf/io/json.pxd index 7ce3cb859a5..d05a778ed82 100644 --- a/python/pylibcudf/pylibcudf/io/json.pxd +++ b/python/pylibcudf/pylibcudf/io/json.pxd @@ -1,5 +1,7 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from libcpp cimport bool +from libcpp.map cimport map +from libcpp.vector cimport vector from pylibcudf.io.types cimport ( SinkInfo, SourceInfo, @@ -43,14 +45,27 @@ cdef class JsonReaderOptions: cdef class JsonReaderOptionsBuilder: cdef json_reader_options_builder c_obj cdef SourceInfo source - cpdef JsonReaderOptionsBuilder compression(self, compression_type compression) - cpdef JsonReaderOptionsBuilder lines(self, bool val) - cpdef JsonReaderOptionsBuilder keep_quotes(self, bool val) cpdef JsonReaderOptionsBuilder byte_range_offset(self, size_t byte_range_offset) cpdef JsonReaderOptionsBuilder byte_range_size(self, size_t byte_range_size) + cpdef JsonReaderOptionsBuilder compression(self, compression_type compression) + cpdef JsonReaderOptionsBuilder dayfirst(self, bool val) + cpdef JsonReaderOptionsBuilder delimiter(self, str delimiter) + cpdef JsonReaderOptionsBuilder dtypes(self, list types) + cpdef JsonReaderOptionsBuilder experimental(self, bool val) + cpdef JsonReaderOptionsBuilder keep_quotes(self, bool val) + cpdef JsonReaderOptionsBuilder lines(self, bool val) + cpdef JsonReaderOptionsBuilder mixed_types_as_string(self, bool val) + cpdef JsonReaderOptionsBuilder na_values(self, list vals) + cpdef JsonReaderOptionsBuilder nonnumeric_numbers(self, bool val) + cpdef JsonReaderOptionsBuilder normalize_single_quotes(self, bool val) + cpdef JsonReaderOptionsBuilder normalize_whitespace(self, bool val) + cpdef JsonReaderOptionsBuilder numeric_leading_zeros(self, bool val) + cpdef JsonReaderOptionsBuilder prune_columns(self, bool val) cpdef JsonReaderOptionsBuilder recovery_mode( self, json_recovery_mode_t recovery_mode ) + cpdef JsonReaderOptionsBuilder strict_validation(self, bool val) + cpdef JsonReaderOptionsBuilder unquoted_control_chars(self, bool val) cpdef build(self) cpdef TableWithMetadata read_json(JsonReaderOptions options) diff --git a/python/pylibcudf/pylibcudf/io/json.pyi b/python/pylibcudf/pylibcudf/io/json.pyi index db4546f138d..bdd15931858 100644 --- a/python/pylibcudf/pylibcudf/io/json.pyi +++ b/python/pylibcudf/pylibcudf/io/json.pyi @@ -45,11 +45,25 @@ class JsonReaderOptions: def builder(source: SourceInfo) -> JsonReaderOptionsBuilder: ... class JsonReaderOptionsBuilder: - def compression(self, compression: CompressionType) -> Self: ... - def lines(self, lines: bool) -> Self: ... def byte_range_offset(self, byte_range_offset: int) -> Self: ... def byte_range_size(self, byte_range_size: int) -> Self: ... + def compression(self, compression_type: CompressionType) -> Self: ... + def dayfirst(self, val: bool) -> Self: ... + def delimiter(self, delimiter: str) -> Self: ... + def dtypes(self, types: list) -> Self: ... + def experimental(self, val: bool) -> Self: ... + def keep_quotes(self, val: bool) -> Self: ... + def lines(self, val: bool) -> Self: ... + def mixed_types_as_string(self, val: bool) -> Self: ... + def na_values(self, vals: list) -> Self: ... + def nonnumeric_numbers(self, val: bool) -> Self: ... + def normalize_single_quotes(self, val: bool) -> Self: ... + def normalize_whitespace(self, val: bool) -> Self: ... + def numeric_leading_zeros(self, val: bool) -> Self: ... + def prune_columns(self, val: bool) -> Self: ... def recovery_mode(self, recovery_mode: JSONRecoveryMode) -> Self: ... + def strict_validation(self, val: bool) -> Self: ... + def unquoted_control_chars(self, val: bool) -> Self: ... def build(self) -> JsonReaderOptions: ... def read_json(options: JsonReaderOptions) -> TableWithMetadata: ... diff --git a/python/pylibcudf/pylibcudf/io/json.pyx b/python/pylibcudf/pylibcudf/io/json.pyx index cf286378902..fae9244e1f6 100644 --- a/python/pylibcudf/pylibcudf/io/json.pyx +++ b/python/pylibcudf/pylibcudf/io/json.pyx @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from libcpp cimport bool from libcpp.map cimport map from libcpp.string cimport string @@ -307,6 +307,38 @@ cdef class JsonReaderOptions: cdef class JsonReaderOptionsBuilder: + cpdef JsonReaderOptionsBuilder byte_range_offset(self, size_t byte_range_offset): + """ + Set number of bytes to skip from source start. + + Parameters + ---------- + byte_range_offset : size_t + Number of bytes of offset + + Returns + ------- + Self + """ + self.c_obj.byte_range_offset(byte_range_offset) + return self + + cpdef JsonReaderOptionsBuilder byte_range_size(self, size_t byte_range_size): + """ + Set number of bytes to read. + + Parameters + ---------- + byte_range_size : size_t + Number of bytes to read + + Returns + ------- + Self + """ + self.c_obj.byte_range_size(byte_range_size) + return self + cpdef JsonReaderOptionsBuilder compression(self, compression_type compression): """ Sets compression type. @@ -323,21 +355,81 @@ cdef class JsonReaderOptionsBuilder: self.c_obj.compression(compression) return self - cpdef JsonReaderOptionsBuilder lines(self, bool val): + cpdef JsonReaderOptionsBuilder dayfirst(self, bool val): """ - Set whether to read the file as a json object per line. + Set whether the reader should parse dates as DD/MM versus MM/DD. Parameters ---------- val : bool - Boolean value to enable/disable the option - to read each line as a json object + Boolean value to indicate whether the + reader should enable/disable DD/MM parsing Returns ------- Self """ - self.c_obj.lines(val) + self.c_obj.dayfirst(val) + return self + + cpdef JsonReaderOptionsBuilder delimiter(self, str delimiter): + """ + Set delimiter character separating records in JSON lines inputs + + Parameters + ---------- + delimiter : str + Character to be used as delimiter separating records + + Returns + ------- + Self + """ + self.c_obj.delimiter(delimiter) + return self + + cpdef JsonReaderOptionsBuilder dtypes(self, list types): + """ + Set data type for columns to be read + + Parameters + ---------- + types : list + List of dtypes or a list of tuples of + column names, dtypes, and list of tuples + (to support nested column hierarchy) + + Returns + ------- + Self + """ + cdef vector[data_type] types_vec + if isinstance(types[0], tuple): + self.c_obj.dtypes(_generate_schema_map(types)) + return self + else: + types_vec.reserve(len(types)) + for dtype in types: + types_vec.push_back((dtype).c_obj) + self.c_obj.dtypes(types_vec) + return self + + cpdef JsonReaderOptionsBuilder experimental(self, bool val): + """ + Set whether to enable experimental features. + When set to true, experimental features, such as the new column tree + construction, utf-8 matching of field names will be enabled. + + Parameters + ---------- + val : bool + Boolean value to enable/disable experimental features + + Returns + ------- + Self + """ + self.c_obj.experimental(val) return self cpdef JsonReaderOptionsBuilder keep_quotes(self, bool val): @@ -357,36 +449,147 @@ cdef class JsonReaderOptionsBuilder: self.c_obj.keep_quotes(val) return self - cpdef JsonReaderOptionsBuilder byte_range_offset(self, size_t byte_range_offset): + cpdef JsonReaderOptionsBuilder lines(self, bool val): """ - Set number of bytes to skip from source start. + Set whether to read the file as a json object per line. Parameters ---------- - byte_range_offset : size_t - Number of bytes of offset + val : bool + Boolean value to enable/disable the option + to read each line as a json object Returns ------- Self """ - self.c_obj.byte_range_offset(byte_range_offset) + self.c_obj.lines(val) return self - cpdef JsonReaderOptionsBuilder byte_range_size(self, size_t byte_range_size): + cpdef JsonReaderOptionsBuilder mixed_types_as_string(self, bool val): """ - Set number of bytes to read. + Set whether to parse mixed types as a string column. + Also enables forcing to read a struct as string column using schema. Parameters ---------- - byte_range_size : size_t - Number of bytes to read + val : bool + Boolean value to enable/disable parsing mixed types as a string column Returns ------- Self """ - self.c_obj.byte_range_size(byte_range_size) + self.c_obj.mixed_types_as_string(val) + return self + + cpdef JsonReaderOptionsBuilder na_values(self, list vals): + """ + Sets additional values to recognize as null values. + + Parameters + ---------- + vals : list + Vector of values to be considered to be null + + Returns + ------- + Self + """ + cdef vector[string] vec + for val in vals: + if isinstance(val, str): + vec.push_back(val.encode()) + self.c_obj.na_values(vec) + return self + + cpdef JsonReaderOptionsBuilder nonnumeric_numbers(self, bool val): + """ + Set whether unquoted number values should be allowed NaN, +INF, -INF, +Infinity, + Infinity, and -Infinity. Strict validation must be enabled for this to work. + + Parameters + ---------- + val : bool + Boolean value to indicate whether leading zeros are allowed in numeric + values + + Returns + ------- + Self + """ + self.c_obj.nonnumeric_numbers(val) + return self + + cpdef JsonReaderOptionsBuilder normalize_single_quotes(self, bool val): + """ + Sets whether to normalize single quotes around strings. + + Parameters + ---------- + val : bool + Boolean value to enable/disable the option to normalize single quotes + around strings + + Returns + ------- + Self + """ + self.c_obj.normalize_single_quotes(val) + return self + + cpdef JsonReaderOptionsBuilder normalize_whitespace(self, bool val): + """ + Sets whether to normalize unquoted whitespace characters + + Parameters + ---------- + val : bool + Boolean value to enable/disable the option to normalize unquoted + whitespace characters + + Returns + ------- + Self + """ + self.c_obj.normalize_whitespace(val) + return self + + cpdef JsonReaderOptionsBuilder numeric_leading_zeros(self, bool val): + """ + Set whether leading zeros are allowed in numeric values. Strict validation + must be enabled for this to work. + + Parameters + ---------- + val : bool + Boolean value to indicate whether leading zeros are allowed in numeric + values + + Returns + ------- + Self + """ + self.c_obj.numeric_leading_zeros(val) + return self + + cpdef JsonReaderOptionsBuilder prune_columns(self, bool val): + """ + Set whether to prune columns on read, selected based on the @ref dtypes option. + When set as true, if the reader options include @ref dtypes, then + the reader will only return those columns which are mentioned in @ref dtypes. + If false, then all columns are returned, independent of the @ref dtypes setting. + + Parameters + ---------- + val : bool + Boolean value to enable/disable column pruning + + Returns + ------- + Self + """ + self.c_obj.prune_columns(val) return self cpdef JsonReaderOptionsBuilder recovery_mode( @@ -409,6 +612,40 @@ cdef class JsonReaderOptionsBuilder: self.c_obj.recovery_mode(recovery_mode) return self + cpdef JsonReaderOptionsBuilder strict_validation(self, bool val): + """ + Set whether strict validation is enabled or not + + Parameters + ---------- + val : bool + Boolean value to indicate whether strict validation is to be enabled + + Returns + ------- + Self + """ + self.c_obj.strict_validation(val) + return self + + cpdef JsonReaderOptionsBuilder unquoted_control_chars(self, bool val): + """ + Set whether in a quoted string should characters greater than or equal to 0 + and less than 32 be allowed without some form of escaping. Strict validation + must be enabled for this to work. + + Parameters + ---------- + val : bool + Boolean value to indicate whether unquoted control chars are allowed + + Returns + ------- + Self + """ + self.c_obj.unquoted_control_chars(val) + return self + cpdef build(self): """Create a JsonReaderOptions object""" cdef JsonReaderOptions json_options = JsonReaderOptions.__new__( diff --git a/python/pylibcudf/pylibcudf/libcudf/io/json.pxd b/python/pylibcudf/pylibcudf/libcudf/io/json.pxd index d23dd0685d1..da7742f8bc2 100644 --- a/python/pylibcudf/pylibcudf/libcudf/io/json.pxd +++ b/python/pylibcudf/pylibcudf/libcudf/io/json.pxd @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. cimport pylibcudf.libcudf.io.types as cudf_io_types cimport pylibcudf.libcudf.table.table_view as cudf_table_view from libc.stdint cimport int32_t, uint8_t @@ -88,15 +88,15 @@ cdef extern from "cudf/io/json.hpp" \ json_reader_options_builder( cudf_io_types.source_info src ) except +libcudf_exception_handler - json_reader_options_builder& dtypes( - vector[string] types - ) except +libcudf_exception_handler json_reader_options_builder& dtypes( vector[data_type] types ) except +libcudf_exception_handler json_reader_options_builder& dtypes( map[string, schema_element] types ) except +libcudf_exception_handler + json_reader_options_builder& dtypes( + map[string, data_type] types + ) except +libcudf_exception_handler json_reader_options_builder& dtypes( schema_element types ) except +libcudf_exception_handler From 3e40b30298475d811a0116ad728d84f27bddf9f7 Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Fri, 14 Feb 2025 17:52:14 -0500 Subject: [PATCH 041/129] Move cudf::lists::detail::make_empty_lists_column to public API (#17996) This copies the existing `make_empty_lists_column` in the `detail` namespace to the public `cudf` namespace. The `cudf::make_empty_column` does not support lists or structs types. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Yunsong Wang (https://github.com/PointKernel) - Vyas Ramasubramani (https://github.com/vyasr) - Basit Ayantunde (https://github.com/lamarrr) URL: https://github.com/rapidsai/cudf/pull/17996 --- cpp/include/cudf/column/column_factories.hpp | 17 ++++++++++++++++- cpp/src/lists/lists_column_factories.cu | 11 ++++++++++- cpp/src/lists/sequences.cu | 2 +- 3 files changed, 27 insertions(+), 3 deletions(-) diff --git a/cpp/include/cudf/column/column_factories.hpp b/cpp/include/cudf/column/column_factories.hpp index e72661ce49a..2c645942ba6 100644 --- a/cpp/include/cudf/column/column_factories.hpp +++ b/cpp/include/cudf/column/column_factories.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -516,6 +516,21 @@ std::unique_ptr make_lists_column( rmm::cuda_stream_view stream = cudf::get_default_stream(), rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); +/** + * @brief Create an empty LIST column + * + * A list column requires a child type and so cannot be created with `make_empty_column`. + * + * @param child_type The type used for the empty child column + * @param stream CUDA stream used for device memory operations and kernel launches + * @param mr Device memory resource used to allocate the returned column's device memory + * @return New empty lists column + */ +std::unique_ptr make_empty_lists_column( + data_type child_type, + rmm::cuda_stream_view stream = cudf::get_default_stream(), + rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); + /** * @brief Construct a STRUCT column using specified child columns as members. * diff --git a/cpp/src/lists/lists_column_factories.cu b/cpp/src/lists/lists_column_factories.cu index dea38947a54..5d85938608d 100644 --- a/cpp/src/lists/lists_column_factories.cu +++ b/cpp/src/lists/lists_column_factories.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -112,6 +112,13 @@ std::unique_ptr make_all_nulls_lists_column(size_type size, } // namespace detail } // namespace lists +std::unique_ptr make_empty_lists_column(data_type child_type, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) +{ + return lists::detail::make_empty_lists_column(child_type, stream, mr); +} + /** * @copydoc cudf::make_lists_column */ @@ -144,6 +151,8 @@ std::unique_ptr make_lists_column(size_type num_rows, null_count, std::move(children)); + if (num_rows == 0) { return output; } + // We need to enforce all null lists to be empty. // `has_nonempty_nulls` is less expensive than `purge_nonempty_nulls` and can save some // run time if we don't have any non-empty nulls. diff --git a/cpp/src/lists/sequences.cu b/cpp/src/lists/sequences.cu index a98f3021da5..21730e7d233 100644 --- a/cpp/src/lists/sequences.cu +++ b/cpp/src/lists/sequences.cu @@ -156,7 +156,7 @@ std::unique_ptr sequences(column_view const& starts, } auto const n_lists = starts.size(); - if (n_lists == 0) { return make_empty_lists_column(starts.type(), stream, mr); } + if (n_lists == 0) { return cudf::make_empty_lists_column(starts.type(), stream, mr); } // Generate list offsets for the output. auto list_offsets = make_numeric_column( From b82d0e73551e4b759c716dd4489465db7808c8b1 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Fri, 14 Feb 2025 22:18:25 -0500 Subject: [PATCH 042/129] Skip failing polars tests (#18015) These tests re failing in CI, but I'm not able to reproduce the error locally. This PR is intended to unblock CI, while I investigate. Authors: - Matthew Murray (https://github.com/Matt711) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18015 --- ci/test_cudf_polars_polars_tests.sh | 2 ++ python/cudf_polars/cudf_polars/testing/plugin.py | 3 +++ 2 files changed, 5 insertions(+) diff --git a/ci/test_cudf_polars_polars_tests.sh b/ci/test_cudf_polars_polars_tests.sh index 3466edacfc5..909abbe9d1e 100755 --- a/ci/test_cudf_polars_polars_tests.sh +++ b/ci/test_cudf_polars_polars_tests.sh @@ -27,6 +27,8 @@ git clone https://github.com/pola-rs/polars.git --branch "${TAG}" --depth 1 # Install requirements for running polars tests rapids-logger "Install polars test requirements" rapids-pip-retry install -r polars/py-polars/requirements-dev.txt -r polars/py-polars/requirements-ci.txt +# TODO: Workaround until https://github.com/pola-rs/polars/issues/21274 is fixed. +rapids-pip-retry install connectorx==0.4.1 # shellcheck disable=SC2317 function set_exitcode() diff --git a/python/cudf_polars/cudf_polars/testing/plugin.py b/python/cudf_polars/cudf_polars/testing/plugin.py index 0b52cf1c61c..48629af920d 100644 --- a/python/cudf_polars/cudf_polars/testing/plugin.py +++ b/python/cudf_polars/cudf_polars/testing/plugin.py @@ -214,6 +214,9 @@ def pytest_configure(config: pytest.Config) -> None: "tests/unit/streaming/test_streaming_group_by.py::test_streaming_group_by_literal[1]": "May segfault w/the legacy streaming engine", # Fails in CI, but passes locally "tests/unit/streaming/test_streaming.py::test_streaming_streamable_functions": "RuntimeError: polars_python::sql::PySQLContext is unsendable, but is being dropped on another thread", + "tests/unit/io/database/test_read.py::test_read_database[uri: connectorx]": "ValueError: arrow2", + "tests/unit/io/database/test_read.py::test_read_database_cx_credentials[fakedb:/123:456@account/database/schema?warehouse=warehouse&role=role]": "ValueError: arrow2", + "tests/unit/io/database/test_read.py::test_read_database_cx_credentials[fakedb:/my#%us3r:p433w0rd@not_a_real_host:9999/database]": "ValueError: arrow2", } From 35b18f408d75e30b40e9ff83d72df1904f46f18f Mon Sep 17 00:00:00 2001 From: Vyas Ramasubramani Date: Fri, 14 Feb 2025 21:37:16 -0800 Subject: [PATCH 043/129] Ensure disabling the module accelerator is thread-safe (#17955) Disabling must be handled on a per-thread basis because each thread could be enabling or disabling the accelerator independently. This is a follow-up to https://github.com/rapidsai/cudf/pull/17811 to address https://github.com/rapidsai/cudf/pull/17811#discussion_r1930660804. Authors: - Vyas Ramasubramani (https://github.com/vyasr) - Matthew Murray (https://github.com/Matt711) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Bradley Dice (https://github.com/bdice) - Matthew Murray (https://github.com/Matt711) - Lawrence Mitchell (https://github.com/wence-) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/17955 --- python/cudf/cudf/pandas/module_accelerator.py | 58 +++++++++++-------- .../test_disable_per_thread_safety.py | 44 ++++++++++++++ 2 files changed, 78 insertions(+), 24 deletions(-) create mode 100644 python/cudf/cudf_pandas_tests/test_disable_per_thread_safety.py diff --git a/python/cudf/cudf/pandas/module_accelerator.py b/python/cudf/cudf/pandas/module_accelerator.py index c4020887907..a33ec5e289b 100644 --- a/python/cudf/cudf/pandas/module_accelerator.py +++ b/python/cudf/cudf/pandas/module_accelerator.py @@ -15,6 +15,7 @@ import threading import warnings from abc import abstractmethod +from collections import defaultdict from importlib._bootstrap import _ImportLockContext as ImportLock from types import ModuleType from typing import Any, ContextManager, NamedTuple # noqa: UP035 @@ -378,8 +379,7 @@ class ModuleAccelerator(ModuleAcceleratorBase): """ _denylist: tuple[str] - _use_fast_lib: bool - _use_fast_lib_lock: threading.RLock + _disable_count: defaultdict[int, int] _module_cache_prefix: str = "_slow_lib_" # TODO: Add possibility for either an explicit allow-list of @@ -409,9 +409,9 @@ def __new__( del sys.modules[mod] self._denylist = (*slow_module.__path__, *fast_module.__path__) - # Lock to manage temporarily disabling delivering wrapped attributes - self._use_fast_lib_lock = threading.RLock() - self._use_fast_lib = True + # This initialization does not need to be protected since a given instance is + # always being created on a given thread. + self._disable_count = defaultdict(int) return self def _populate_module(self, mod: ModuleType): @@ -503,20 +503,11 @@ def disabled(self): ------- Context manager for disabling things """ - with self._use_fast_lib_lock: - # Have to hold the lock to modify this variable since - # another thread might be reading it. - # Modification has to happen with the lock held for the - # duration, so if someone else has modified things, then - # we block trying to acquire the lock (hence it is safe to - # release the lock after modifying this value) - saved = self._use_fast_lib - self._use_fast_lib = False + self._disable_count[threading.get_ident()] += 1 try: yield finally: - with self._use_fast_lib_lock: - self._use_fast_lib = saved + self._disable_count[threading.get_ident()] -= 1 @staticmethod def getattr_real_or_wrapped( @@ -545,14 +536,20 @@ def getattr_real_or_wrapped( ------- The requested attribute (either real or wrapped) """ - with loader._use_fast_lib_lock: - # Have to hold the lock to read this variable since - # another thread might modify it. - # Modification has to happen with the lock held for the - # duration, so if someone else has modified things, then - # we block trying to acquire the lock (hence it is safe to - # release the lock after reading this value) - use_real = not loader._use_fast_lib + use_real = ( + loader._disable_count[threading.get_ident()] > 0 + # If acceleration was disabled on the main thread, we should respect that. + # This only works because we currently have no way to re-enable other than + # exiting the disable context, so disabling on the parent thread means that + # the inner threads will also typically be disabled. This logic breaks if + # the parent thread queues work on a thread and only then disables + # acceleration because in that case there is a potential race condition by + # which the child thread may wind up disabled even though the parent was not + # disabled when the child was launched. That is a fairly rare pattern though + # and we can document the limitations. + # The main thread is always started, so the ident is always an int + or loader._disable_count[threading.main_thread().ident] > 0 # type: ignore + ) if not use_real: # Only need to check the denylist if we're not turned off. frame = sys._getframe() @@ -616,6 +613,19 @@ def install( def disable_module_accelerator() -> contextlib.ExitStack: """ Temporarily disable any module acceleration. + + This function only offers limited guarantees of thread safety. + Cases that will work: + - multiple threads are launched and each independently turns off acceleration + - a single thread turns off acceleration and then launches multiple threads + inside the context manager + + Cases that trigger race conditions: + - a single thread launches multiple threads and then enters the context manager + while those threads are still running + - nested thread launching and acceleration disabling, i.e. if a thread launches + a thread that disables acceleration and then launches another thread, the + innermost thread will not have the accelerator disabled. """ with ImportLock(), contextlib.ExitStack() as stack: for finder in sys.meta_path: diff --git a/python/cudf/cudf_pandas_tests/test_disable_per_thread_safety.py b/python/cudf/cudf_pandas_tests/test_disable_per_thread_safety.py new file mode 100644 index 00000000000..25f3a1dd60b --- /dev/null +++ b/python/cudf/cudf_pandas_tests/test_disable_per_thread_safety.py @@ -0,0 +1,44 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. + +from concurrent.futures import ThreadPoolExecutor +from time import sleep + +import pandas as pd + +from cudf.pandas.fast_slow_proxy import _FastSlowProxyMeta +from cudf.pandas.module_accelerator import disable_module_accelerator + + +def is_enabled(df: pd.DataFrame): + return type(type(df)) is _FastSlowProxyMeta + + +def per_thread_work(_): + assert is_enabled(pd.DataFrame()) + + with disable_module_accelerator(): + assert not is_enabled(pd.DataFrame()) + + # Do some fake work to allow other threads to potentially modify this one + for _ in range(1000): + sleep(1e-6) + + assert not is_enabled(pd.DataFrame()) + + # Ensure that nesting the context manager works too + with disable_module_accelerator(): + assert not is_enabled(pd.DataFrame()) + for _ in range(1000): + sleep(1e-6) + + assert not is_enabled(pd.DataFrame()) + assert not is_enabled(pd.DataFrame()) + + assert is_enabled(pd.DataFrame()) + + +def test_disable_pandas_accelerator_multi_threaded(): + num_threads = 20 + with ThreadPoolExecutor(max_workers=num_threads) as executor: + for _ in executor.map(per_thread_work, range(num_threads * 10)): + pass From 4e3780662c3da2cef27211dfd01bedd094b061f5 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Sat, 15 Feb 2025 02:55:14 -0500 Subject: [PATCH 044/129] Install duckdb the default backend for ibis in the cudf.pandas integration tests (#17972) Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - James Lamb (https://github.com/jameslamb) URL: https://github.com/rapidsai/cudf/pull/17972 --- .../dependencies.yaml | 2 +- .../tests/test_ibis.py | 14 +++++++++----- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml index 059a4ff3c98..2ce9fa45f5e 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml @@ -277,7 +277,7 @@ dependencies: packages: - pip - pip: - - ibis-framework[pandas]<10.0.0 + - ibis-framework[duckdb] test_hvplot: common: - output_types: conda diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py index 70f20b2810e..f43493814ba 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py @@ -5,8 +5,6 @@ import pandas as pd import pytest -ibis.set_backend("pandas") - ibis.options.interactive = False @@ -59,7 +57,7 @@ def ibis_table_num(): rng.integers(0, 100, (N, K)), columns=[f"val{x}" for x in np.arange(K)] ) df["key"] = rng.choice(np.arange(10), N) - table = ibis.memtable(df, name="t") + table = ibis.memtable(df, name="u") return table @@ -72,12 +70,15 @@ def test_column_reductions(ibis_table_num_str, op): @pytest.mark.parametrize("op", ["mean", "sum", "min", "max"]) def test_groupby_reductions(ibis_table_num_str, op): t = ibis_table_num_str - return getattr(t.group_by("key").col1, op)().to_pandas() + return getattr(t.group_by("key").col1, "min")().order_by("key").to_pandas() @pytest.mark.parametrize("op", ELEMENTWISE_UFUNCS) def test_mutate_ufunc(ibis_table_num_str, op): t = ibis_table_num_str + if op == "log": + # avoid duckdb log of 0 error + t = t.mutate(col1=t.col1 + 1) expr = getattr(t.col1, op)() return t.mutate(col1_sin=expr).to_pandas() @@ -116,7 +117,10 @@ def test_notin(ibis_table_num_str): def test_window(ibis_table_num_str): t = ibis_table_num_str return ( - t.group_by("key").mutate(demeaned=t.col1 - t.col1.mean()).to_pandas() + t.group_by("key") + .mutate(demeaned=t.col1 - t.col1.mean()) + .order_by("key") + .to_pandas() ) From e497a1ede2580a9964d07974839df74f6df253a2 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Sat, 15 Feb 2025 07:41:45 -0600 Subject: [PATCH 045/129] consolidate more conda solves in CI (#18014) Contributes to https://github.com/rapidsai/build-planning/issues/22 Follow-up to #17995 * ensures all of the conda solves in CI are consolidated * ensures the locally-downloaded files are considered in conda solves # Authors: - James Lamb (https://github.com/jameslamb) - Matthew Murray (https://github.com/Matt711) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: https://github.com/rapidsai/cudf/pull/18014 --- ci/build_docs.sh | 18 ++++++------------ ci/release/update-version.sh | 3 +++ ci/test_cpp_common.sh | 15 ++++----------- ci/test_java.sh | 13 ++++--------- ci/test_notebooks.sh | 16 +++++----------- ci/test_python_common.sh | 11 ++--------- dependencies.yaml | 18 ++++++++++++++++++ 7 files changed, 42 insertions(+), 52 deletions(-) diff --git a/ci/build_docs.sh b/ci/build_docs.sh index c24a58b0232..3f584c004ba 100755 --- a/ci/build_docs.sh +++ b/ci/build_docs.sh @@ -13,9 +13,15 @@ rapids-logger "Create test conda environment" ENV_YAML_DIR="$(mktemp -d)" +rapids-logger "Downloading artifacts from previous jobs" +CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) +PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python) + rapids-dependency-file-generator \ --output conda \ --file-key docs \ + --prepend-channel "${CPP_CHANNEL}" \ + --prepend-channel "${PYTHON_CHANNEL}" \ --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee "${ENV_YAML_DIR}/env.yaml" rapids-mamba-retry env create --yes -f "${ENV_YAML_DIR}/env.yaml" -n docs @@ -23,18 +29,6 @@ conda activate docs rapids-print-env -rapids-logger "Downloading artifacts from previous jobs" -CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) -PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python) - -rapids-mamba-retry install \ - --channel "${CPP_CHANNEL}" \ - --channel "${PYTHON_CHANNEL}" \ - "libcudf=${RAPIDS_VERSION}" \ - "pylibcudf=${RAPIDS_VERSION}" \ - "cudf=${RAPIDS_VERSION}" \ - "dask-cudf=${RAPIDS_VERSION}" - RAPIDS_DOCS_DIR="$(mktemp -d)" export RAPIDS_DOCS_DIR diff --git a/ci/release/update-version.sh b/ci/release/update-version.sh index d014d3b08ff..80426a8071a 100755 --- a/ci/release/update-version.sh +++ b/ci/release/update-version.sh @@ -51,6 +51,9 @@ DEPENDENCIES=( dask-cudf kvikio libcudf + libcudf-example + libcudf_kafka + libcudf-tests libkvikio librmm pylibcudf diff --git a/ci/test_cpp_common.sh b/ci/test_cpp_common.sh index 8cd78eb11c2..bc33e85a6a5 100755 --- a/ci/test_cpp_common.sh +++ b/ci/test_cpp_common.sh @@ -1,11 +1,12 @@ #!/bin/bash -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. set -euo pipefail . /opt/conda/etc/profile.d/conda.sh -RAPIDS_VERSION="$(rapids-version)" +rapids-logger "Downloading artifacts from previous jobs" +CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) rapids-logger "Generate C++ testing dependencies" @@ -14,6 +15,7 @@ ENV_YAML_DIR="$(mktemp -d)" rapids-dependency-file-generator \ --output conda \ --file-key test_cpp \ + --prepend-channel "${CPP_CHANNEL}" \ --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee "${ENV_YAML_DIR}/env.yaml" rapids-mamba-retry env create --yes -f "${ENV_YAML_DIR}/env.yaml" -n test @@ -23,20 +25,11 @@ set +u conda activate test set -u -CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) - RESULTS_DIR=${RAPIDS_TESTS_DIR:-"$(mktemp -d)"} RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${RESULTS_DIR}/test-results"}/ mkdir -p "${RAPIDS_TESTS_DIR}" rapids-print-env -rapids-mamba-retry install \ - --channel "${CPP_CHANNEL}" \ - "libcudf=${RAPIDS_VERSION}" \ - "libcudf_kafka=${RAPIDS_VERSION}" \ - "libcudf-tests=${RAPIDS_VERSION}" \ - "libcudf-example=${RAPIDS_VERSION}" - rapids-logger "Check GPU usage" nvidia-smi diff --git a/ci/test_java.sh b/ci/test_java.sh index 7f1aa633afc..05020ae3b04 100755 --- a/ci/test_java.sh +++ b/ci/test_java.sh @@ -1,11 +1,12 @@ #!/bin/bash -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. set -euo pipefail . /opt/conda/etc/profile.d/conda.sh -RAPIDS_VERSION="$(rapids-version)" +rapids-logger "Downloading artifacts from previous jobs" +CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) rapids-logger "Generate Java testing dependencies" @@ -14,6 +15,7 @@ ENV_YAML_DIR="$(mktemp -d)" rapids-dependency-file-generator \ --output conda \ --file-key test_java \ + --prepend-channel "${CPP_CHANNEL}" \ --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee "${ENV_YAML_DIR}/env.yaml" rapids-mamba-retry env create --yes -f "${ENV_YAML_DIR}/env.yaml" -n test @@ -27,13 +29,6 @@ set -u rapids-print-env -rapids-logger "Downloading artifacts from previous jobs" -CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) - -rapids-mamba-retry install \ - --channel "${CPP_CHANNEL}" \ - "libcudf=${RAPIDS_VERSION}" - rapids-logger "Check GPU usage" nvidia-smi diff --git a/ci/test_notebooks.sh b/ci/test_notebooks.sh index 329246ef9d7..1c2f152b084 100755 --- a/ci/test_notebooks.sh +++ b/ci/test_notebooks.sh @@ -5,7 +5,9 @@ set -euo pipefail . /opt/conda/etc/profile.d/conda.sh -RAPIDS_VERSION="$(rapids-version)" +rapids-logger "Downloading artifacts from previous jobs" +CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) +PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python) rapids-logger "Generate notebook testing dependencies" @@ -14,6 +16,8 @@ ENV_YAML_DIR="$(mktemp -d)" rapids-dependency-file-generator \ --output conda \ --file-key test_notebooks \ + --prepend-channel "${CPP_CHANNEL}" \ + --prepend-channel "${PYTHON_CHANNEL}" \ --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee "${ENV_YAML_DIR}/env.yaml" rapids-mamba-retry env create --yes -f "${ENV_YAML_DIR}/env.yaml" -n test @@ -25,16 +29,6 @@ set -u rapids-print-env -rapids-logger "Downloading artifacts from previous jobs" -CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) -PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python) - -rapids-mamba-retry install \ - --channel "${CPP_CHANNEL}" \ - --channel "${PYTHON_CHANNEL}" \ - "cudf=${RAPIDS_VERSION}" \ - "libcudf=${RAPIDS_VERSION}" - NBTEST="$(realpath "$(dirname "$0")/utils/nbtest.sh")" pushd notebooks diff --git a/ci/test_python_common.sh b/ci/test_python_common.sh index 63f7317c19f..604121ac5dd 100755 --- a/ci/test_python_common.sh +++ b/ci/test_python_common.sh @@ -7,8 +7,6 @@ set -euo pipefail . /opt/conda/etc/profile.d/conda.sh -RAPIDS_VERSION="$(rapids-version)" - rapids-logger "Downloading artifacts from previous jobs" CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python) @@ -20,6 +18,8 @@ FILE_KEY=$1 rapids-dependency-file-generator \ --output conda \ --file-key "${FILE_KEY}" \ + --prepend-channel "${CPP_CHANNEL}" \ + --prepend-channel "${PYTHON_CHANNEL}" \ --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};dependencies=${RAPIDS_DEPENDENCIES}" \ | tee "${ENV_YAML_DIR}/env.yaml" @@ -36,10 +36,3 @@ RAPIDS_COVERAGE_DIR=${RAPIDS_COVERAGE_DIR:-"${RESULTS_DIR}/coverage-results"} mkdir -p "${RAPIDS_TESTS_DIR}" "${RAPIDS_COVERAGE_DIR}" rapids-print-env - -rapids-mamba-retry install \ - --channel "${CPP_CHANNEL}" \ - --channel "${PYTHON_CHANNEL}" \ - "cudf=${RAPIDS_VERSION}" \ - "pylibcudf=${RAPIDS_VERSION}" \ - "libcudf=${RAPIDS_VERSION}" diff --git a/dependencies.yaml b/dependencies.yaml index 2cb876e075a..7188e10b058 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -55,7 +55,9 @@ files: output: none includes: - cuda_version + - depends_on_libcudf - test_cpp + - test_cpp_cudf test_python_cudf_pandas: output: none includes: @@ -98,11 +100,14 @@ files: - build_all - cuda - cuda_version + - depends_on_libcudf - test_java test_notebooks: output: none includes: - cuda_version + - depends_on_cudf + - depends_on_libcudf - notebooks - py_version checks: @@ -125,6 +130,10 @@ files: includes: - cuda - cuda_version + - depends_on_cudf + - depends_on_dask_cudf + - depends_on_pylibcudf + - depends_on_libcudf - docs - py_version py_build_cudf: @@ -835,6 +844,15 @@ dependencies: - cuda-sanitizer-api=11.8.86 - matrix: # Fallback for CUDA 11 or no matrix packages: + # packages we want in the 'test_cpp' group in 'files', for CI, but which + # shouldn't be added to 'all' for building a development environment + test_cpp_cudf: + common: + - output_types: conda + packages: + - libcudf-example==25.4.*,>=0.0.0a0 + - libcudf_kafka==25.4.*,>=0.0.0a0 + - libcudf-tests==25.4.*,>=0.0.0a0 test_java: common: - output_types: conda From b6b9e8df26867d9a16209767544bc8686fc633a4 Mon Sep 17 00:00:00 2001 From: GALI PREM SAGAR Date: Sat, 15 Feb 2025 11:26:06 -0600 Subject: [PATCH 046/129] Fix `to_arrow` to return consistent pandas-metadata (#18009) This PR makes change to `to_arrow` API by making the metadata being generated for types like that of `list` type to have uniform type across IO readers and Arrow table. This will help fix the failure occurring in the added pytest in this PR. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - Praateek Mahajan (https://github.com/praateekmahajan) URL: https://github.com/rapidsai/cudf/pull/18009 --- python/cudf/cudf/core/dataframe.py | 6 +++++- python/cudf/cudf/tests/test_list.py | 24 ++++++++++++++++++++++++ python/cudf/cudf/utils/ioutils.py | 12 ++++++++---- 3 files changed, 37 insertions(+), 5 deletions(-) diff --git a/python/cudf/cudf/core/dataframe.py b/python/cudf/cudf/core/dataframe.py index 5225a4b97ec..a52fd49b001 100644 --- a/python/cudf/cudf/core/dataframe.py +++ b/python/cudf/cudf/core/dataframe.py @@ -5,6 +5,7 @@ import functools import inspect import itertools +import json import numbers import os import re @@ -5749,8 +5750,11 @@ def to_arrow(self, preserve_index=None) -> pa.Table: preserve_index=preserve_index, types=out.schema.types, ) + md_dict = json.loads(metadata[b"pandas"]) - return out.replace_schema_metadata(metadata) + cudf.utils.ioutils._update_pandas_metadata_types_inplace(self, md_dict) + + return out.replace_schema_metadata({b"pandas": json.dumps(md_dict)}) @_performance_tracking def to_records(self, index=True, column_dtypes=None, index_dtypes=None): diff --git a/python/cudf/cudf/tests/test_list.py b/python/cudf/cudf/tests/test_list.py index 359660e76a7..3ffbd5ff2a8 100644 --- a/python/cudf/cudf/tests/test_list.py +++ b/python/cudf/cudf/tests/test_list.py @@ -954,3 +954,27 @@ def test_empty_nested_list_uninitialized_offsets_memory_usage(): ) ser = cudf.Series._from_column(col_empty_offset) assert ser.memory_usage() == 8 + + +def test_dataframe_list_round_trip(): + data = [{"text": "hello", "list_col": np.asarray([1, 2], dtype="uint32")}] + cudf_arrow = cudf.DataFrame(data).to_arrow() + pdf_arrow = pa.Table.from_pandas(pd.DataFrame(data)) + + for metadata in [ + None, + pdf_arrow.schema.metadata, + cudf_arrow.schema.metadata, + ]: + schema = pa.schema( + [ + pa.field("text", pa.string()), + pa.field("list_col", pa.list_(pa.uint32())), + ], + metadata=metadata, + ) + + data = {"text": ["asd", "pqr"], "list_col": [[1, 2, 3], [4, 5]]} + + table = pa.Table.from_pydict(data, schema=schema) + assert_eq(table.to_pandas(), pd.DataFrame(data)) diff --git a/python/cudf/cudf/utils/ioutils.py b/python/cudf/cudf/utils/ioutils.py index 26d5aee8896..9fb06faa66c 100644 --- a/python/cudf/cudf/utils/ioutils.py +++ b/python/cudf/cudf/utils/ioutils.py @@ -1639,12 +1639,18 @@ def generate_pandas_metadata(table: cudf.DataFrame, index: bool | None) -> str: ) md_dict = json.loads(metadata[b"pandas"]) + _update_pandas_metadata_types_inplace(table, md_dict) + return json.dumps(md_dict) + +def _update_pandas_metadata_types_inplace( + df: cudf.DataFrame, md_dict: dict +) -> None: # correct metadata for list and struct and nullable numeric types for col_meta in md_dict["columns"]: if ( - col_meta["name"] in table._column_names - and table._data[col_meta["name"]].nullable + col_meta["name"] in df._column_names + and df._data[col_meta["name"]].nullable and col_meta["numpy_type"] in PARQUET_META_TYPE_MAP and col_meta["pandas_type"] != "decimal" ): @@ -1654,8 +1660,6 @@ def generate_pandas_metadata(table: cudf.DataFrame, index: bool | None) -> str: if col_meta["numpy_type"] in ("list", "struct"): col_meta["numpy_type"] = "object" - return json.dumps(md_dict) - def is_url(url): """Check if a string is a valid URL to a network location. From 4530e1de99ade670fa8a6c9da3b44f0766801c12 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Sat, 15 Feb 2025 21:41:13 -0500 Subject: [PATCH 047/129] Fix failing ibis test (#18022) Follows up #17972 to fix one more failing ibis test. The PR ensures the ibis test is deterministic by returning the aggregation in sorted order. Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/18022 --- .../tests/test_ibis.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py index f43493814ba..ff24af52b4b 100644 --- a/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py +++ b/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_ibis.py @@ -166,9 +166,13 @@ def test_order_by(ibis_table_num_str): def test_aggregate_having(ibis_table_num_str): t = ibis_table_num_str - return t.aggregate( - by=["key"], - sum_c0=t.col0.sum(), - avg_c0=t.col0.mean(), - having=t.col1.mean() > 50, - ).to_pandas() + return ( + t.aggregate( + by=["key"], + sum_c0=t.col0.sum(), + avg_c0=t.col0.mean(), + having=t.col1.mean() > 50, + ) + .order_by("key") + .to_pandas() + ) From 65a2fbc618995dc10c872b6a7946b20ff34f4116 Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Mon, 17 Feb 2025 11:02:44 -0500 Subject: [PATCH 048/129] Add seed parameter to cudf hash_character_ngrams (#17994) The seed parameter was added to `nvtext::hash_character_ngrams` in #17643 but was not added to the Python interface. This PR adds the parameter to the pylibcudf function and Python interface methods. Authors: - David Wendt (https://github.com/davidwendt) - Matthew Murray (https://github.com/Matt711) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - Matthew Murray (https://github.com/Matt711) URL: https://github.com/rapidsai/cudf/pull/17994 --- python/cudf/cudf/core/column/string.py | 14 +++++++++----- python/cudf/cudf/tests/text/test_text_methods.py | 6 +++--- .../pylibcudf/libcudf/nvtext/generate_ngrams.pxd | 6 ++++-- .../pylibcudf/pylibcudf/nvtext/generate_ngrams.pxd | 5 +++-- .../pylibcudf/pylibcudf/nvtext/generate_ngrams.pyi | 4 ++-- .../pylibcudf/pylibcudf/nvtext/generate_ngrams.pyx | 9 +++++++-- .../pylibcudf/tests/test_nvtext_generate_ngrams.py | 8 ++++---- 7 files changed, 32 insertions(+), 20 deletions(-) diff --git a/python/cudf/cudf/core/column/string.py b/python/cudf/cudf/core/column/string.py index 074da57c470..5d8fa6a90a4 100644 --- a/python/cudf/cudf/core/column/string.py +++ b/python/cudf/cudf/core/column/string.py @@ -4986,7 +4986,7 @@ def character_ngrams( return result def hash_character_ngrams( - self, n: int = 5, as_list: bool = False + self, n: int = 5, as_list: bool = False, seed: np.uint32 = 0 ) -> SeriesOrIndex: """ Generate hashes of n-grams from characters in a column of strings. @@ -5000,12 +5000,14 @@ def hash_character_ngrams( as_list : bool Set to True to return the hashes in a list column where each list element is the hashes for each string. + seed: uint32 + The seed value for the hash algorithm. Examples -------- >>> import cudf >>> str_series = cudf.Series(['abcdefg','stuvwxyz']) - >>> str_series.str.hash_character_ngrams(5, True) + >>> str_series.str.hash_character_ngrams(n=5, as_list=True) 0 [3902511862, 570445242, 4202475763] 1 [556054766, 3166857694, 3760633458, 192452857] dtype: list @@ -5021,7 +5023,7 @@ def hash_character_ngrams( """ result = self._return_or_inplace( - self._column.hash_character_ngrams(n), + self._column.hash_character_ngrams(n, seed), retain_index=True, ) if isinstance(result, cudf.Series) and not as_list: @@ -6176,9 +6178,11 @@ def generate_character_ngrams(self, ngrams: int) -> ListColumn: return type(self).from_pylibcudf(result) # type: ignore[return-value] @acquire_spill_lock() - def hash_character_ngrams(self, ngrams: int) -> ListColumn: + def hash_character_ngrams( + self, ngrams: int, seed: np.uint32 + ) -> ListColumn: result = plc.nvtext.generate_ngrams.hash_character_ngrams( - self.to_pylibcudf(mode="read"), ngrams + self.to_pylibcudf(mode="read"), ngrams, seed ) return type(self).from_pylibcudf(result) # type: ignore[return-value] diff --git a/python/cudf/cudf/tests/text/test_text_methods.py b/python/cudf/cudf/tests/text/test_text_methods.py index 9a62285403f..86e1e46c1a2 100644 --- a/python/cudf/cudf/tests/text/test_text_methods.py +++ b/python/cudf/cudf/tests/text/test_text_methods.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019-2024, NVIDIA CORPORATION. +# Copyright (c) 2019-2025, NVIDIA CORPORATION. import random import string @@ -378,11 +378,11 @@ def test_hash_character_ngrams(): ), ] ) - actual = strings.str.hash_character_ngrams(5, True) + actual = strings.str.hash_character_ngrams(n=5, as_list=True) assert type(expected) is type(actual) assert_eq(expected, actual) - actual = strings.str.hash_character_ngrams(5) + actual = strings.str.hash_character_ngrams(n=5) expected = expected.explode() assert type(expected) is type(actual) assert_eq(expected, actual) diff --git a/python/pylibcudf/pylibcudf/libcudf/nvtext/generate_ngrams.pxd b/python/pylibcudf/pylibcudf/libcudf/nvtext/generate_ngrams.pxd index c7bd4da5441..a62361bb190 100644 --- a/python/pylibcudf/pylibcudf/libcudf/nvtext/generate_ngrams.pxd +++ b/python/pylibcudf/pylibcudf/libcudf/nvtext/generate_ngrams.pxd @@ -1,4 +1,5 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. +from libc.stdint cimport uint32_t from libcpp.memory cimport unique_ptr from pylibcudf.exception_handler cimport libcudf_exception_handler from pylibcudf.libcudf.column.column cimport column @@ -22,5 +23,6 @@ cdef extern from "nvtext/generate_ngrams.hpp" namespace "nvtext" nogil: cdef unique_ptr[column] hash_character_ngrams( const column_view &strings, - size_type ngrams + size_type ngrams, + uint32_t seed ) except +libcudf_exception_handler diff --git a/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pxd b/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pxd index f15eb1f25e9..bbeb8f241a1 100644 --- a/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pxd +++ b/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pxd @@ -1,5 +1,6 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. +from libc.stdint cimport uint32_t from pylibcudf.column cimport Column from pylibcudf.libcudf.types cimport size_type from pylibcudf.scalar cimport Scalar @@ -9,4 +10,4 @@ cpdef Column generate_ngrams(Column input, size_type ngrams, Scalar separator) cpdef Column generate_character_ngrams(Column input, size_type ngrams=*) -cpdef Column hash_character_ngrams(Column input, size_type ngrams=*) +cpdef Column hash_character_ngrams(Column input, size_type ngrams, uint32_t seed) diff --git a/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pyi b/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pyi index 2757518379d..a7d4da97d2a 100644 --- a/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pyi +++ b/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pyi @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from pylibcudf.column import Column from pylibcudf.scalar import Scalar @@ -7,4 +7,4 @@ def generate_ngrams( input: Column, ngrams: int, separator: Scalar ) -> Column: ... def generate_character_ngrams(input: Column, ngrams: int = 2) -> Column: ... -def hash_character_ngrams(input: Column, ngrams: int = 2) -> Column: ... +def hash_character_ngrams(input: Column, ngrams: int, seed: int) -> Column: ... diff --git a/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pyx b/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pyx index 521bc0ef4a4..29da693e06f 100644 --- a/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pyx +++ b/python/pylibcudf/pylibcudf/nvtext/generate_ngrams.pyx @@ -1,5 +1,6 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. +from libc.stdint cimport uint32_t from libcpp.memory cimport unique_ptr from libcpp.utility cimport move from pylibcudf.column cimport Column @@ -81,7 +82,8 @@ cpdef Column generate_character_ngrams(Column input, size_type ngrams = 2): ) return Column.from_libcudf(move(c_result)) -cpdef Column hash_character_ngrams(Column input, size_type ngrams = 2): + +cpdef Column hash_character_ngrams(Column input, size_type ngrams, uint32_t seed): """ Returns a lists column of hash values of the characters in each string @@ -93,6 +95,8 @@ cpdef Column hash_character_ngrams(Column input, size_type ngrams = 2): Input strings ngram : size_type The ngram number to generate + seed : uint32_t + Seed used for the hash algorithm Returns ------- @@ -106,5 +110,6 @@ cpdef Column hash_character_ngrams(Column input, size_type ngrams = 2): c_result = cpp_hash_character_ngrams( c_strings, ngrams, + seed ) return Column.from_libcudf(move(c_result)) diff --git a/python/pylibcudf/pylibcudf/tests/test_nvtext_generate_ngrams.py b/python/pylibcudf/pylibcudf/tests/test_nvtext_generate_ngrams.py index fae4685f81b..c8f8ce4f8ff 100644 --- a/python/pylibcudf/pylibcudf/tests/test_nvtext_generate_ngrams.py +++ b/python/pylibcudf/pylibcudf/tests/test_nvtext_generate_ngrams.py @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. import pyarrow as pa import pytest @@ -40,10 +40,10 @@ def test_generate_character_ngrams(input_col, ngram): @pytest.mark.parametrize("ngram", [2, 3]) -def test_hash_character_ngrams(input_col, ngram): +@pytest.mark.parametrize("seed", [0, 3]) +def test_hash_character_ngrams(input_col, ngram, seed): result = plc.nvtext.generate_ngrams.hash_character_ngrams( - plc.interop.from_arrow(input_col), - ngram, + plc.interop.from_arrow(input_col), ngram, seed ) pa_result = plc.interop.to_arrow(result) assert all( From dc479800d83136b75f73d6e33607bc2819c9fc50 Mon Sep 17 00:00:00 2001 From: Tianyu Liu Date: Tue, 18 Feb 2025 10:15:30 -0500 Subject: [PATCH 049/129] Fix the build error due to KvikIO update (#18025) KvikIO has just introduced a breaking change (https://github.com/rapidsai/kvikio/pull/608) where the compatibility mode handling is delegated to `CompatModeManager`. This PR updates the way cuDF queries the compatibility mode to avoid build errors. Authors: - Tianyu Liu (https://github.com/kingcrimsontianyu) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Yunsong Wang (https://github.com/PointKernel) - Nghia Truong (https://github.com/ttnghia) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18025 --- cpp/src/io/utilities/data_sink.cpp | 2 +- cpp/src/io/utilities/datasource.cpp | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cpp/src/io/utilities/data_sink.cpp b/cpp/src/io/utilities/data_sink.cpp index e8a05f431bd..a8f73e600f5 100644 --- a/cpp/src/io/utilities/data_sink.cpp +++ b/cpp/src/io/utilities/data_sink.cpp @@ -37,7 +37,7 @@ class file_sink : public data_sink { _kvikio_file = kvikio::FileHandle(filepath, "w"); CUDF_EXPECTS(!_kvikio_file.closed(), "KvikIO did not open the file successfully."); CUDF_LOG_INFO("Writing a file using kvikIO, with compatibility mode %s.", - _kvikio_file.is_compat_mode_preferred() ? "on" : "off"); + _kvikio_file.get_compat_mode_manager().is_compat_mode_preferred() ? "on" : "off"); } // Marked as NOLINT because we are calling a virtual method in the destructor diff --git a/cpp/src/io/utilities/datasource.cpp b/cpp/src/io/utilities/datasource.cpp index 14b6bc6f774..2cb2b303cb3 100644 --- a/cpp/src/io/utilities/datasource.cpp +++ b/cpp/src/io/utilities/datasource.cpp @@ -54,7 +54,7 @@ class file_source : public datasource { _kvikio_file = kvikio::FileHandle(filepath, "r"); CUDF_EXPECTS(!_kvikio_file.closed(), "KvikIO did not open the file successfully."); CUDF_LOG_INFO("Reading a file using kvikIO, with compatibility mode %s.", - _kvikio_file.is_compat_mode_preferred() ? "on" : "off"); + _kvikio_file.get_compat_mode_manager().is_compat_mode_preferred() ? "on" : "off"); } std::unique_ptr host_read(size_t offset, size_t size) override From 81bb6f1ea46d633548b9bf534c93a28ba66909ea Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Tue, 18 Feb 2025 09:36:11 -0800 Subject: [PATCH 050/129] Pass dtype objects to Column.astype (#18008) Broken off from https://github.com/rapidsai/cudf/pull/17978 This passes more dtype objects to `Column.astype` but does not _ensure_ that a dtype object is passes unlike in https://github.com/rapidsai/cudf/pull/17978 Also it appears we avoid surfacing from warnings from pandas within our tests. Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/18008 --- python/cudf/cudf/core/_base_index.py | 5 +- python/cudf/cudf/core/_internals/timezones.py | 6 ++- python/cudf/cudf/core/column/column.py | 26 ++++----- python/cudf/cudf/core/column/datetime.py | 52 +++++++++++------- python/cudf/cudf/core/column/numerical.py | 2 +- python/cudf/cudf/core/column/string.py | 43 ++++++++++----- python/cudf/cudf/core/column/timedelta.py | 54 +++++++++++++------ python/cudf/cudf/core/dataframe.py | 15 +++--- python/cudf/cudf/core/frame.py | 7 ++- python/cudf/cudf/core/groupby/groupby.py | 10 ++-- python/cudf/cudf/core/index.py | 24 +++++---- python/cudf/cudf/core/indexed_frame.py | 4 +- python/cudf/cudf/core/join/join.py | 4 +- python/cudf/cudf/core/multiindex.py | 4 +- python/cudf/cudf/core/resample.py | 26 ++++----- python/cudf/cudf/core/reshape.py | 9 ++-- python/cudf/cudf/core/series.py | 8 +-- python/cudf/cudf/core/subword_tokenizer.py | 3 +- python/cudf/cudf/core/tools/datetimes.py | 21 +++++--- python/cudf/cudf/core/tools/numeric.py | 6 +-- python/cudf/cudf/core/window/ewm.py | 6 ++- python/cudf/cudf/core/window/rolling.py | 13 +++-- python/cudf/cudf/tests/test_dataframe.py | 3 +- python/cudf/cudf/tests/test_datetime.py | 10 ++-- python/cudf/cudf/tests/test_string_udfs.py | 4 +- python/cudf/cudf/utils/utils.py | 6 +-- 26 files changed, 222 insertions(+), 149 deletions(-) diff --git a/python/cudf/cudf/core/_base_index.py b/python/cudf/cudf/core/_base_index.py index a5e1e88c960..05fb6f531a0 100644 --- a/python/cudf/cudf/core/_base_index.py +++ b/python/cudf/cudf/core/_base_index.py @@ -6,6 +6,7 @@ from functools import cached_property from typing import TYPE_CHECKING, Any, Literal +import numpy as np import pandas as pd from typing_extensions import Self @@ -2159,7 +2160,7 @@ def _get_result_name(left_name, right_name): return left_name if _is_same_name(left_name, right_name) else None -def _return_get_indexer_result(result): +def _return_get_indexer_result(result: cupy.ndarray) -> cupy.ndarray: if cudf.get_option("mode.pandas_compatible"): - return result.astype("int64") + return result.astype(np.dtype(np.int64)) return result diff --git a/python/cudf/cudf/core/_internals/timezones.py b/python/cudf/cudf/core/_internals/timezones.py index 4d001577581..d5de78bc2bb 100644 --- a/python/cudf/cudf/core/_internals/timezones.py +++ b/python/cudf/cudf/core/_internals/timezones.py @@ -1,4 +1,4 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. from __future__ import annotations import datetime @@ -124,7 +124,9 @@ def _read_tzfile_as_columns( from cudf.core.column.column import as_column # this happens for UTC-like zones - min_date = np.int64(np.iinfo("int64").min + 1).astype("M8[s]") + min_date = np.int64(np.iinfo("int64").min + 1).astype( + np.dtype("M8[s]") + ) return (as_column([min_date]), as_column([np.timedelta64(0, "s")])) # type: ignore[return-value] return tuple(transition_times_and_offsets) # type: ignore[return-value] diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index 6268ffb356d..94e75e9d07a 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -191,7 +191,7 @@ def _prep_pandas_compat_repr(self) -> StringColumn | Self: * null (other types)= str(pd.NA) """ if self.has_nulls(): - return self.astype("str").fillna(self._PANDAS_NA_REPR) + return self.astype(CUDF_STRING_DTYPE).fillna(self._PANDAS_NA_REPR) return self def to_pandas( @@ -1182,7 +1182,7 @@ def astype(self, dtype: Dtype, copy: bool = False) -> ColumnBase: elif ( isinstance(dtype, str) and dtype == "interval" - and isinstance(self.dtype, cudf.IntervalDtype) + and isinstance(self.dtype, IntervalDtype) ): # astype("interval") (the string only) should no-op result = self @@ -1201,7 +1201,7 @@ def astype(self, dtype: Dtype, copy: bool = False) -> ColumnBase: f"Casting {self.dtype} columns not currently supported" ) result = self - elif isinstance(dtype, cudf.core.dtypes.DecimalDtype): + elif isinstance(dtype, DecimalDtype): result = self.as_decimal_column(dtype) elif dtype.kind == "M": result = self.as_datetime_column(dtype) @@ -1728,7 +1728,7 @@ def reduce(self, reduction_op: str, dtype=None, **kwargs) -> ScalarLike: precision = max(min(new_p, col_dtype.MAX_PRECISION), 0) new_dtype = type(col_dtype)(precision, scale) result_col = result_col.astype(new_dtype) - elif isinstance(col_dtype, cudf.IntervalDtype): + elif isinstance(col_dtype, IntervalDtype): result_col = type(self).from_struct_column( # type: ignore[attr-defined] result_col, closed=col_dtype.closed ) @@ -2090,7 +2090,7 @@ def as_column( ) elif dtype is None and pa.types.is_null(arbitrary.type): # default "empty" type - dtype = "str" + dtype = CUDF_STRING_DTYPE col = ColumnBase.from_arrow(arbitrary) if dtype is not None: @@ -2156,7 +2156,7 @@ def as_column( and dtype is None ): # Conversion to arrow converts IntervalDtype to StructDtype - dtype = cudf.CategoricalDtype.from_pandas(arbitrary.dtype) + dtype = CategoricalDtype.from_pandas(arbitrary.dtype) return as_column( pa.array(arbitrary, from_pandas=True), nan_as_null=nan_as_null, @@ -2355,7 +2355,7 @@ def as_column( raise NotImplementedError( "Use `tz_localize()` to construct timezone aware data." ) - elif isinstance(dtype, cudf.core.dtypes.DecimalDtype): + elif isinstance(dtype, DecimalDtype): # Arrow throws a type error if the input is of # mixed-precision and cannot fit into the provided # decimal type properly, see: @@ -2366,11 +2366,11 @@ def as_column( arbitrary, type=pa.decimal128(precision=dtype.precision, scale=dtype.scale), ) - if isinstance(dtype, cudf.core.dtypes.Decimal128Dtype): + if isinstance(dtype, cudf.Decimal128Dtype): return cudf.core.column.Decimal128Column.from_arrow(data) - elif isinstance(dtype, cudf.core.dtypes.Decimal64Dtype): + elif isinstance(dtype, cudf.Decimal64Dtype): return cudf.core.column.Decimal64Column.from_arrow(data) - elif isinstance(dtype, cudf.core.dtypes.Decimal32Dtype): + elif isinstance(dtype, cudf.Decimal32Dtype): return cudf.core.column.Decimal32Column.from_arrow(data) else: raise NotImplementedError(f"{dtype} not implemented") @@ -2378,9 +2378,9 @@ def as_column( dtype, ( pd.CategoricalDtype, - cudf.CategoricalDtype, + CategoricalDtype, pd.IntervalDtype, - cudf.IntervalDtype, + IntervalDtype, ), ) or dtype in { "category", @@ -2391,7 +2391,7 @@ def as_column( object, np.dtype(object), }: - if isinstance(dtype, (cudf.CategoricalDtype, cudf.IntervalDtype)): + if isinstance(dtype, (CategoricalDtype, IntervalDtype)): dtype = dtype.to_pandas() elif dtype == object and not cudf.get_option("mode.pandas_compatible"): # Unlike pandas, interpret object as "str" instead of "python object" diff --git a/python/cudf/cudf/core/column/datetime.py b/python/cudf/cudf/core/column/datetime.py index 1373febb47d..439dd7991d6 100644 --- a/python/cudf/cudf/core/column/datetime.py +++ b/python/cudf/cudf/core/column/datetime.py @@ -265,8 +265,8 @@ def __contains__(self, item: ScalarLike) -> bool: return False elif ts.tzinfo is not None: ts = ts.tz_convert(None) - return ts.to_numpy().astype("int64") in cast( - "cudf.core.column.NumericalColumn", self.astype("int64") + return ts.to_numpy().astype(np.dtype(np.int64)) in cast( + "cudf.core.column.NumericalColumn", self.astype(np.dtype(np.int64)) ) @functools.cached_property @@ -506,7 +506,7 @@ def round(self, freq: str) -> ColumnBase: def isocalendar(self) -> dict[str, ColumnBase]: return { - field: self.strftime(format=directive).astype("uint32") + field: self.strftime(format=directive).astype(np.dtype(np.uint32)) for field, directive in zip( ["year", "week", "day"], ["%G", "%V", "%u"] ) @@ -559,7 +559,7 @@ def normalize_binop_value( # type: ignore[override] ) if other_time_unit not in {"s", "ms", "ns", "us"}: - other = other.astype("timedelta64[s]") + other = other.astype(np.dtype("timedelta64[s]")) return cudf.Scalar(other) elif isinstance(other, str): @@ -656,7 +656,8 @@ def as_string_column(self) -> cudf.core.column.StringColumn: def mean(self, skipna=None, min_count: int = 0) -> ScalarLike: return pd.Timestamp( cast( - "cudf.core.column.NumericalColumn", self.astype("int64") + "cudf.core.column.NumericalColumn", + self.astype(np.dtype(np.int64)), ).mean(skipna=skipna, min_count=min_count), unit=self.time_unit, ).as_unit(self.time_unit) @@ -668,16 +669,18 @@ def std( ddof: int = 1, ) -> pd.Timedelta: return pd.Timedelta( - cast("cudf.core.column.NumericalColumn", self.astype("int64")).std( - skipna=skipna, min_count=min_count, ddof=ddof - ) + cast( + "cudf.core.column.NumericalColumn", + self.astype(np.dtype(np.int64)), + ).std(skipna=skipna, min_count=min_count, ddof=ddof) * _unit_to_nanoseconds_conversion[self.time_unit], ).as_unit(self.time_unit) def median(self, skipna: bool | None = None) -> pd.Timestamp: return pd.Timestamp( cast( - "cudf.core.column.NumericalColumn", self.astype("int64") + "cudf.core.column.NumericalColumn", + self.astype(np.dtype(np.int64)), ).median(skipna=skipna), unit=self.time_unit, ).as_unit(self.time_unit) @@ -688,8 +691,13 @@ def cov(self, other: DatetimeColumn) -> float: f"cannot perform cov with types {self.dtype}, {other.dtype}" ) return cast( - "cudf.core.column.NumericalColumn", self.astype("int64") - ).cov(cast("cudf.core.column.NumericalColumn", other.astype("int64"))) + "cudf.core.column.NumericalColumn", self.astype(np.dtype(np.int64)) + ).cov( + cast( + "cudf.core.column.NumericalColumn", + other.astype(np.dtype(np.int64)), + ) + ) def corr(self, other: DatetimeColumn) -> float: if not isinstance(other, DatetimeColumn): @@ -697,8 +705,13 @@ def corr(self, other: DatetimeColumn) -> float: f"cannot perform corr with types {self.dtype}, {other.dtype}" ) return cast( - "cudf.core.column.NumericalColumn", self.astype("int64") - ).corr(cast("cudf.core.column.NumericalColumn", other.astype("int64"))) + "cudf.core.column.NumericalColumn", self.astype(np.dtype(np.int64)) + ).corr( + cast( + "cudf.core.column.NumericalColumn", + other.astype(np.dtype(np.int64)), + ) + ) def quantile( self, @@ -707,7 +720,7 @@ def quantile( exact: bool, return_scalar: bool, ) -> ColumnBase: - result = self.astype("int64").quantile( + result = self.astype(np.dtype(np.int64)).quantile( q=q, interpolation=interpolation, exact=exact, @@ -811,13 +824,16 @@ def indices_of( self, value: ScalarLike ) -> cudf.core.column.NumericalColumn: value = ( - pd.to_datetime(value).to_numpy().astype(self.dtype).astype("int64") + pd.to_datetime(value) + .to_numpy() + .astype(self.dtype) + .astype(np.dtype(np.int64)) ) - return self.astype("int64").indices_of(value) + return self.astype(np.dtype(np.int64)).indices_of(value) @property def is_unique(self) -> bool: - return self.astype("int64").is_unique + return self.astype(np.dtype(np.int64)).is_unique def isin(self, values: Sequence) -> ColumnBase: return cudf.core.tools.datetimes._isin_datetimelike(self, values) @@ -880,7 +896,7 @@ def _find_ambiguous_and_nonexistent( If no transitions occur, the tuple `(False, False)` is returned. """ transition_times, offsets = get_tz_data(zone_name) - offsets = offsets.astype(f"timedelta64[{self.time_unit}]") # type: ignore[assignment] + offsets = offsets.astype(np.dtype(f"timedelta64[{self.time_unit}]")) # type: ignore[assignment] if len(offsets) == 1: # no transitions return False, False diff --git a/python/cudf/cudf/core/column/numerical.py b/python/cudf/cudf/core/column/numerical.py index 1abd55b110d..eecb294acee 100644 --- a/python/cudf/cudf/core/column/numerical.py +++ b/python/cudf/cudf/core/column/numerical.py @@ -214,7 +214,7 @@ def _binaryop(self, other: ColumnBinaryOperand, op: str) -> ColumnBase: if op in {"__truediv__", "__rtruediv__"}: # Division with integer types results in a suitable float. if truediv_type := int_float_dtype_mapping.get(self.dtype.type): - return self.astype(truediv_type)._binaryop(other, op) + return self.astype(np.dtype(truediv_type))._binaryop(other, op) elif op in { "__lt__", "__gt__", diff --git a/python/cudf/cudf/core/column/string.py b/python/cudf/cudf/core/column/string.py index 5d8fa6a90a4..7cc0e75b4bd 100644 --- a/python/cudf/cudf/core/column/string.py +++ b/python/cudf/cudf/core/column/string.py @@ -28,6 +28,7 @@ from cudf.core.scalar import pa_scalar_to_plc_scalar from cudf.utils.docutils import copy_docstring from cudf.utils.dtypes import ( + CUDF_STRING_DTYPE, SIZE_TYPE_DTYPE, can_convert_to_column, dtype_to_pylibcudf_type, @@ -329,13 +330,15 @@ def cat(self, others=None, sep=None, na_rep=None): ) ): other_cols = ( - column.as_column(frame.reindex(parent_index), dtype="str") + column.as_column( + frame.reindex(parent_index), dtype=CUDF_STRING_DTYPE + ) if ( parent_index is not None and isinstance(frame, cudf.Series) and not frame.index.equals(parent_index) ) - else column.as_column(frame, dtype="str") + else column.as_column(frame, dtype=CUDF_STRING_DTYPE) for frame in others ) elif others is not None and not isinstance(others, StringMethods): @@ -346,7 +349,9 @@ def cat(self, others=None, sep=None, na_rep=None): ): others = others.reindex(parent_index) - other_cols = [column.as_column(others, dtype="str")] + other_cols = [ + column.as_column(others, dtype=CUDF_STRING_DTYPE) + ] else: raise TypeError( "others must be Series, Index, DataFrame, np.ndarrary " @@ -819,10 +824,14 @@ def contains( # TODO: we silently ignore the `regex=` flag here if case is False: input_column = self.lower()._column # type: ignore[union-attr] - col_pat = cudf.Index(pat, dtype="str").str.lower()._column # type: ignore[union-attr] + col_pat = ( + cudf.Index(pat, dtype=CUDF_STRING_DTYPE) + .str.lower() + ._column + ) # type: ignore[union-attr] else: input_column = self._column - col_pat = column.as_column(pat, dtype="str") + col_pat = column.as_column(pat, dtype=CUDF_STRING_DTYPE) with acquire_spill_lock(): plc_result = plc.strings.find.contains( input_column.to_pylibcudf(mode="read"), @@ -1049,15 +1058,21 @@ def replace( plc_result = plc.strings.replace_re.replace_re( self._column.to_pylibcudf(mode="read"), list(pat), - column.as_column(repl, dtype="str").to_pylibcudf( - mode="read" - ), + column.as_column( + repl, dtype=CUDF_STRING_DTYPE + ).to_pylibcudf(mode="read"), ) result = Column.from_pylibcudf(plc_result) else: result = self._column.replace_multiple( - cast(StringColumn, column.as_column(pat, dtype="str")), - cast(StringColumn, column.as_column(repl, dtype="str")), + cast( + StringColumn, + column.as_column(pat, dtype=CUDF_STRING_DTYPE), + ), + cast( + StringColumn, + column.as_column(repl, dtype=CUDF_STRING_DTYPE), + ), ) return self._return_or_inplace(result) # Pandas treats 0 as all @@ -3946,9 +3961,9 @@ def _starts_ends_with( if isinstance(pat, str): plc_pat = pa_scalar_to_plc_scalar(pa.scalar(pat, type=pa.string())) elif isinstance(pat, tuple) and all(isinstance(p, str) for p in pat): - plc_pat = column.as_column(pat, dtype="str").to_pylibcudf( - mode="read" - ) + plc_pat = column.as_column( + pat, dtype=CUDF_STRING_DTYPE + ).to_pylibcudf(mode="read") else: raise TypeError( f"expected a string or tuple, not {type(pat).__name__}" @@ -5552,7 +5567,7 @@ def _massage_string_arg( if allow_col: if isinstance(value, list): - return column.as_column(value, dtype="str") # type: ignore[return-value] + return column.as_column(value, dtype=CUDF_STRING_DTYPE) # type: ignore[return-value] if isinstance(value, StringColumn): return value diff --git a/python/cudf/cudf/core/column/timedelta.py b/python/cudf/cudf/core/column/timedelta.py index b45c62589d7..1cbbac0f8cc 100644 --- a/python/cudf/cudf/core/column/timedelta.py +++ b/python/cudf/cudf/core/column/timedelta.py @@ -133,8 +133,8 @@ def __contains__(self, item: DatetimeLikeScalar) -> bool: # np.timedelta64 raises ValueError, hence `item` # cannot exist in `self`. return False - return item.view("int64") in cast( - "cudf.core.column.NumericalColumn", self.astype("int64") + return item.view(np.dtype(np.int64)) in cast( + "cudf.core.column.NumericalColumn", self.astype(np.dtype(np.int64)) ) @property @@ -182,7 +182,9 @@ def to_arrow(self) -> pa.Array: self.mask_array_view(mode="read").copy_to_host() ) data = pa.py_buffer( - self.astype("int64").data_array_view(mode="read").copy_to_host() + self.astype(np.dtype(np.int64)) + .data_array_view(mode="read") + .copy_to_host() ) pa_dtype = np_to_pa_dtype(self.dtype) return pa.Array.from_buffers( @@ -219,7 +221,11 @@ def _binaryop(self, other: ColumnBinaryOperand, op: str) -> ColumnBase: out_dtype = determine_out_dtype(self.dtype, other.dtype) elif op in {"__truediv__", "__floordiv__"}: common_dtype = determine_out_dtype(self.dtype, other.dtype) - out_dtype = np.float64 if op == "__truediv__" else np.int64 + out_dtype = ( + np.dtype(np.float64) + if op == "__truediv__" + else np.dtype(np.int64) + ) this = self.astype(common_dtype).astype(out_dtype) if isinstance(other, cudf.Scalar): if other.is_valid(): @@ -302,10 +308,10 @@ def total_seconds(self) -> ColumnBase: # Typecast to decimal128 to avoid floating point precision issues # https://github.com/rapidsai/cudf/issues/17664 return ( - (self.astype("int64") * conversion) + (self.astype(np.dtype(np.int64)) * conversion) .astype(cudf.Decimal128Dtype(38, 9)) .round(decimals=abs(int(math.log10(conversion)))) - .astype("float64") + .astype(np.dtype(np.float64)) ) def ceil(self, freq: str) -> ColumnBase: @@ -414,7 +420,8 @@ def mean(self, skipna=None) -> pd.Timedelta: def median(self, skipna: bool | None = None) -> pd.Timedelta: return pd.Timedelta( cast( - "cudf.core.column.NumericalColumn", self.astype("int64") + "cudf.core.column.NumericalColumn", + self.astype(np.dtype(np.int64)), ).median(skipna=skipna), unit=self.time_unit, ).as_unit(self.time_unit) @@ -429,7 +436,7 @@ def quantile( exact: bool, return_scalar: bool, ) -> ColumnBase: - result = self.astype("int64").quantile( + result = self.astype(np.dtype(np.int64)).quantile( q=q, interpolation=interpolation, exact=exact, @@ -451,7 +458,7 @@ def sum( # Since sum isn't overridden in Numerical[Base]Column, mypy only # sees the signature from Reducible (which doesn't have the extra # parameters from ColumnBase._reduce) so we have to ignore this. - self.astype("int64").sum( # type: ignore + self.astype(np.dtype(np.int64)).sum( # type: ignore skipna=skipna, min_count=min_count, dtype=dtype ), unit=self.time_unit, @@ -464,9 +471,10 @@ def std( ddof: int = 1, ) -> pd.Timedelta: return pd.Timedelta( - cast("cudf.core.column.NumericalColumn", self.astype("int64")).std( - skipna=skipna, min_count=min_count, ddof=ddof - ), + cast( + "cudf.core.column.NumericalColumn", + self.astype(np.dtype(np.int64)), + ).std(skipna=skipna, min_count=min_count, ddof=ddof), unit=self.time_unit, ).as_unit(self.time_unit) @@ -476,8 +484,13 @@ def cov(self, other: TimeDeltaColumn) -> float: f"cannot perform cov with types {self.dtype}, {other.dtype}" ) return cast( - "cudf.core.column.NumericalColumn", self.astype("int64") - ).cov(cast("cudf.core.column.NumericalColumn", other.astype("int64"))) + "cudf.core.column.NumericalColumn", self.astype(np.dtype(np.int64)) + ).cov( + cast( + "cudf.core.column.NumericalColumn", + other.astype(np.dtype(np.int64)), + ) + ) def corr(self, other: TimeDeltaColumn) -> float: if not isinstance(other, TimeDeltaColumn): @@ -485,8 +498,13 @@ def corr(self, other: TimeDeltaColumn) -> float: f"cannot perform corr with types {self.dtype}, {other.dtype}" ) return cast( - "cudf.core.column.NumericalColumn", self.astype("int64") - ).corr(cast("cudf.core.column.NumericalColumn", other.astype("int64"))) + "cudf.core.column.NumericalColumn", self.astype(np.dtype(np.int64)) + ).corr( + cast( + "cudf.core.column.NumericalColumn", + other.astype(np.dtype(np.int64)), + ) + ) def components(self) -> dict[str, ColumnBase]: """ @@ -604,7 +622,9 @@ def nanoseconds(self) -> cudf.core.column.NumericalColumn: # of nanoseconds. if self.time_unit != "ns": - res_col = column.as_column(0, length=len(self), dtype="int64") + res_col = column.as_column( + 0, length=len(self), dtype=np.dtype(np.int64) + ) if self.nullable: res_col = res_col.set_mask(self.mask) return cast("cudf.core.column.NumericalColumn", res_col) diff --git a/python/cudf/cudf/core/dataframe.py b/python/cudf/cudf/core/dataframe.py index a52fd49b001..8e0cb606b2e 100644 --- a/python/cudf/cudf/core/dataframe.py +++ b/python/cudf/cudf/core/dataframe.py @@ -20,7 +20,7 @@ MutableMapping, Sequence, ) -from typing import TYPE_CHECKING, Any, Literal, cast +from typing import TYPE_CHECKING, Any, Literal import cupy import numba @@ -87,6 +87,7 @@ from cudf.utils.docutils import copy_docstring from cudf.utils.dtypes import ( CUDF_STRING_DTYPE, + SIZE_TYPE_DTYPE, can_convert_to_column, cudf_dtype_from_pydata_dtype, find_common_type, @@ -2457,15 +2458,11 @@ def scatter_by_map( # Convert float to integer if map_index.dtype.kind == "f": - map_index = map_index.astype(np.int32) + map_index = map_index.astype(SIZE_TYPE_DTYPE) # Convert string or categorical to integer if isinstance(map_index, cudf.core.column.StringColumn): - cat_index = cast( - cudf.core.column.CategoricalColumn, - map_index.astype("category"), - ) - map_index = cat_index.codes + map_index = map_index._label_encoding(map_index.unique()) warnings.warn( "Using StringColumn for map_index in scatter_by_map. " "Use an integer array/column for better performance." @@ -6371,7 +6368,7 @@ def _prepare_for_rowwise_op(self, method, skipna, numeric_only): coerced = filtered.astype(common_dtype, copy=False) if is_pure_dt: # Further convert into cupy friendly types - coerced = coerced.astype("int64", copy=False) + coerced = coerced.astype(np.dtype(np.int64), copy=False) return coerced, mask, common_dtype @_performance_tracking @@ -8081,7 +8078,7 @@ def value_counts( dropna=dropna, ) .size() - .astype("int64") + .astype(np.dtype(np.int64)) ) if sort: result = result.sort_values(ascending=ascending) diff --git a/python/cudf/cudf/core/frame.py b/python/cudf/cudf/core/frame.py index 2e0e7244719..41b9c81198e 100644 --- a/python/cudf/cudf/core/frame.py +++ b/python/cudf/cudf/core/frame.py @@ -28,6 +28,7 @@ from cudf.core.column import ( ColumnBase, as_column, + column_empty, deserialize_columns, serialize_columns, ) @@ -999,7 +1000,11 @@ def from_arrow(cls, data: pa.Table) -> Self: # of column is 0 (i.e., empty) then we will have an # int8 column in result._data[name] returned by libcudf, # which needs to be type-casted to 'category' dtype. - result[name] = result[name].astype("category") + result[name] = result[name].astype( + cudf.CategoricalDtype( + categories=column_empty(0, dtype=result[name].dtype) + ) + ) elif ( pandas_dtypes.get(name) == "empty" and np_dtypes.get(name) == "object" diff --git a/python/cudf/cudf/core/groupby/groupby.py b/python/cudf/cudf/core/groupby/groupby.py index 94e0f9155f6..8abdf88ea12 100644 --- a/python/cudf/cudf/core/groupby/groupby.py +++ b/python/cudf/cudf/core/groupby/groupby.py @@ -735,7 +735,7 @@ def rank(x): if cudf.get_option("mode.pandas_compatible"): # pandas always returns floats: - return result.astype("float64") + return result.astype(np.dtype(np.float64)) return result @@ -1017,7 +1017,7 @@ def agg(self, func=None, *args, engine=None, engine_kwargs=None, **kwargs): col = col._with_type_metadata(cudf.ListDtype(orig_dtype)) if agg_kind in {"COUNT", "SIZE", "ARGMIN", "ARGMAX"}: - data[key] = col.astype("int64") + data[key] = col.astype(np.dtype(np.int64)) elif ( self.obj.empty and ( @@ -1966,7 +1966,7 @@ def mult(df): ) if self.obj.empty: if func in {"count", "size", "idxmin", "idxmax"}: - res = cudf.Series([], dtype="int64") + res = cudf.Series([], dtype=np.dtype(np.int64)) else: res = self.obj.copy(deep=True) res.index = self.grouping.keys @@ -1975,7 +1975,7 @@ def mult(df): # will need to result in `int64` type. for name, col in res._column_labels_and_values: if col.dtype.kind == "b": - res._data[name] = col.astype("int") + res._data[name] = col.astype(np.dtype(np.int64)) return res if not callable(func): @@ -3226,7 +3226,7 @@ def value_counts( ] .count() .sort_index() - .astype(np.int64) + .astype(np.dtype(np.int64)) ) if normalize: diff --git a/python/cudf/cudf/core/index.py b/python/cudf/cudf/core/index.py index 08dc114a66d..bdd85ebf7eb 100644 --- a/python/cudf/cudf/core/index.py +++ b/python/cudf/cudf/core/index.py @@ -1453,12 +1453,12 @@ def __repr__(self) -> str: if isinstance(preprocess, CategoricalIndex): if preprocess.categories.dtype.kind == "f": output = repr( - preprocess.astype("str") + preprocess.astype(CUDF_STRING_DTYPE) .to_pandas() .astype( dtype=pd.CategoricalDtype( categories=preprocess.dtype.categories.astype( - "str" + CUDF_STRING_DTYPE ).to_pandas(), ordered=preprocess.dtype.ordered, ) @@ -2016,7 +2016,7 @@ def strftime(self, date_format: str) -> Index: @property def asi8(self) -> cupy.ndarray: - return self._column.astype("int64").values + return self._column.astype(np.dtype(np.int64)).values @property def inferred_freq(self) -> cudf.DateOffset | None: @@ -2330,7 +2330,8 @@ def microsecond(self) -> Index: # Need to manually promote column to int32 because # pandas-matching binop behaviour requires that this # __mul__ returns an int16 column. - self._column.millisecond.astype("int32") * np.int32(1000) + self._column.millisecond.astype(np.dtype(np.int32)) + * np.int32(1000) ) + self._column.microsecond, name=self.name, @@ -2490,7 +2491,9 @@ def quarter(self) -> Index: >>> gIndex.quarter Index([2, 4], dtype='int8') """ - return Index._from_column(self._column.quarter.astype("int8")) + return Index._from_column( + self._column.quarter.astype(np.dtype(np.int8)) + ) @_performance_tracking def day_name(self, locale: str | None = None) -> Index: @@ -2932,7 +2935,7 @@ def to_pytimedelta(self) -> np.ndarray: @property def asi8(self) -> cupy.ndarray: - return self._column.astype("int64").values + return self._column.astype(np.dtype(np.int64)).values def sum(self, *, skipna: bool = True, axis: int | None = 0): return self._column.sum(skipna=skipna) @@ -2990,7 +2993,7 @@ def days(self) -> cudf.Index: """ # Need to specifically return `int64` to avoid overflow. return Index._from_column( - self._column.days.astype("int64"), name=self.name + self._column.days.astype(np.dtype(np.int64)), name=self.name ) @property # type: ignore @@ -3000,7 +3003,7 @@ def seconds(self) -> cudf.Index: Number of seconds (>= 0 and less than 1 day) for each element. """ return Index._from_column( - self._column.seconds.astype("int32"), name=self.name + self._column.seconds.astype(np.dtype(np.int32)), name=self.name ) @property # type: ignore @@ -3010,7 +3013,8 @@ def microseconds(self) -> cudf.Index: Number of microseconds (>= 0 and less than 1 second) for each element. """ return Index._from_column( - self._column.microseconds.astype("int32"), name=self.name + self._column.microseconds.astype(np.dtype(np.int32)), + name=self.name, ) @property # type: ignore @@ -3021,7 +3025,7 @@ def nanoseconds(self) -> cudf.Index: element. """ return Index._from_column( - self._column.nanoseconds.astype("int32"), name=self.name + self._column.nanoseconds.astype(np.dtype(np.int32)), name=self.name ) @property # type: ignore diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index aaf73e122ed..b0ef779bab8 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -426,7 +426,7 @@ def _scan(self, op, axis=None, skipna=True): if cast_to_int and result_col.dtype.kind in "uib": # For reductions that accumulate a value (e.g. sum, not max) # pandas returns an int64 dtype for all int or bool dtypes. - result_col = result_col.astype(np.int64) + result_col = result_col.astype(np.dtype(np.int64)) results.append(getattr(result_col, op)()) return self._from_data_like_self( self._data._from_columns_like_self(results) @@ -2010,7 +2010,7 @@ def interpolate( FutureWarning, ) if col.nullable: - col = col.astype("float64").fillna(np.nan) + col = col.astype(np.dtype(np.float64)).fillna(np.nan) columns.append( cudf.core.algorithms._interpolation(col, index=interp_index) diff --git a/python/cudf/cudf/core/join/join.py b/python/cudf/cudf/core/join/join.py index b8b8324784c..d319f9e71d9 100644 --- a/python/cudf/cudf/core/join/join.py +++ b/python/cudf/cudf/core/join/join.py @@ -293,8 +293,8 @@ def perform_merge(self) -> cudf.DataFrame: and isinstance(lcol.dtype, cudf.CategoricalDtype) and isinstance(rcol.dtype, cudf.CategoricalDtype) ): - lcol_casted = lcol_casted.astype("category") - rcol_casted = rcol_casted.astype("category") + lcol_casted = lcol_casted.astype(lcol.dtype) + rcol_casted = rcol_casted.astype(rcol.dtype) left_key.set(self.lhs, lcol_casted) right_key.set(self.rhs, rcol_casted) diff --git a/python/cudf/cudf/core/multiindex.py b/python/cudf/cudf/core/multiindex.py index 514760d79f8..24e8ed8cfc2 100644 --- a/python/cudf/cudf/core/multiindex.py +++ b/python/cudf/cudf/core/multiindex.py @@ -169,7 +169,7 @@ def __init__( for code in codes: if not (is_list_like(code) or is_column_like(code)): raise TypeError("Each code must be list-like") - new_code = column.as_column(code).astype("int64") + new_code = column.as_column(code, dtype=np.dtype(np.int64)) if copy and new_code is code: new_code = new_code.copy(deep=True) new_codes.append(new_code) @@ -341,7 +341,7 @@ def _maybe_materialize_codes_and_levels(self: Self) -> Self: codes = [] for col in self._data.values(): code, cats = factorize(col) - codes.append(column.as_column(code.astype(np.int64))) + codes.append(column.as_column(code.astype(np.dtype(np.int64)))) levels.append(cats) self._levels = levels self._codes = codes diff --git a/python/cudf/cudf/core/resample.py b/python/cudf/cudf/core/resample.py index 391ee31f125..061363eee9a 100644 --- a/python/cudf/cudf/core/resample.py +++ b/python/cudf/cudf/core/resample.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2021-2024, NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2021-2025, NVIDIA CORPORATION & AFFILIATES. # All rights reserved. # SPDX-License-Identifier: Apache-2.0 # @@ -255,19 +255,21 @@ def _handle_frequency_grouper(self, by): # 'datetime64[s]'. libcudf requires the bin labels and key # column to have the same dtype, so we compute a `result_type` # and cast them both to that type. - try: - result_type = np.dtype(f"datetime64[{offset.rule_code}]") - # TODO: Ideally, we can avoid one cast by having `date_range` - # generate timestamps of a given dtype. Currently, it can - # only generate timestamps with 'ns' precision - cast_key_column = key_column.astype(result_type) - cast_bin_labels = bin_labels.astype(result_type) - except TypeError: + if offset.rule_code.lower() in {"d", "h"}: # unsupported resolution (we don't support resolutions >s) - # fall back to using datetime64[s] result_type = np.dtype("datetime64[s]") - cast_key_column = key_column.astype(result_type) - cast_bin_labels = bin_labels.astype(result_type) + else: + try: + result_type = np.dtype(f"datetime64[{offset.rule_code}]") + # TODO: Ideally, we can avoid one cast by having `date_range` + # generate timestamps of a given dtype. Currently, it can + # only generate timestamps with 'ns' precision + except TypeError: + # unsupported resolution (we don't support resolutions >s) + # fall back to using datetime64[s] + result_type = np.dtype("datetime64[s]") + cast_key_column = key_column.astype(result_type) + cast_bin_labels = bin_labels.astype(result_type) # bin the key column: with acquire_spill_lock(): diff --git a/python/cudf/cudf/core/reshape.py b/python/cudf/cudf/core/reshape.py index 36cbb196ec0..991eb86fa8f 100644 --- a/python/cudf/cudf/core/reshape.py +++ b/python/cudf/cudf/core/reshape.py @@ -20,7 +20,7 @@ from cudf.utils.dtypes import SIZE_TYPE_DTYPE, min_unsigned_type if TYPE_CHECKING: - from cudf._typing import Dtype + from cudf._typing import DtypeObj _AXIS_MAP = {0: 0, 1: 1, "index": 0, "columns": 1} @@ -810,6 +810,8 @@ def get_dummies( if sparse: raise NotImplementedError("sparse is not supported yet") + dtype = cudf.dtype(dtype) + if isinstance(data, cudf.DataFrame): encode_fallback_dtypes = ["object", "category"] @@ -1316,7 +1318,7 @@ def _one_hot_encode_column( categories: ColumnBase, prefix: str | None, prefix_sep: str | None, - dtype: Dtype | None, + dtype: DtypeObj, drop_first: bool, ) -> dict[str, ColumnBase]: """Encode a single column with one hot encoding. The return dictionary @@ -1348,8 +1350,7 @@ def _one_hot_encode_column( data.pop(next(iter(data))) if prefix is not None and prefix_sep is not None: data = {f"{prefix}{prefix_sep}{col}": enc for col, enc in data.items()} - if dtype: - data = {k: v.astype(dtype) for k, v in data.items()} + data = {k: v.astype(dtype) for k, v in data.items()} return data diff --git a/python/cudf/cudf/core/series.py b/python/cudf/cudf/core/series.py index 6a50d5da523..f6f1b31dc43 100644 --- a/python/cudf/cudf/core/series.py +++ b/python/cudf/cudf/core/series.py @@ -4181,9 +4181,9 @@ def microsecond(self) -> Series: # Need to manually promote column to int32 because # pandas-matching binop behaviour requires that this # __mul__ returns an int16 column. - extra = self.series._column.millisecond.astype("int32") * np.int32( - 1000 - ) + extra = self.series._column.millisecond.astype( + np.dtype(np.int32) + ) * np.int32(1000) return self._return_result_like_self(micro + extra) @property # type: ignore @@ -4443,7 +4443,7 @@ def quarter(self) -> Series: dtype: int8 """ return self._return_result_like_self( - self.series._column.quarter.astype(np.int8) + self.series._column.quarter.astype(np.dtype(np.int8)) ) @_performance_tracking diff --git a/python/cudf/cudf/core/subword_tokenizer.py b/python/cudf/cudf/core/subword_tokenizer.py index 50d1a11c39b..c59a16f99f5 100644 --- a/python/cudf/cudf/core/subword_tokenizer.py +++ b/python/cudf/cudf/core/subword_tokenizer.py @@ -5,6 +5,7 @@ import warnings import cupy as cp +import numpy as np import pylibcudf as plc @@ -19,7 +20,7 @@ def _cast_to_appropriate_type(ar, cast_type): elif cast_type == "tf": from tensorflow.experimental.dlpack import from_dlpack - return from_dlpack(ar.astype("int32").toDlpack()) + return from_dlpack(ar.astype(np.dtype(np.int32)).toDlpack()) class SubwordTokenizer: diff --git a/python/cudf/cudf/core/tools/datetimes.py b/python/cudf/cudf/core/tools/datetimes.py index 22d0832b27f..0fc4d5edba8 100644 --- a/python/cudf/cudf/core/tools/datetimes.py +++ b/python/cudf/cudf/core/tools/datetimes.py @@ -21,6 +21,7 @@ from cudf.core.buffer import acquire_spill_lock from cudf.core.index import ensure_index from cudf.core.scalar import pa_scalar_to_plc_scalar +from cudf.utils.dtypes import CUDF_STRING_DTYPE if TYPE_CHECKING: from collections.abc import Sequence @@ -214,11 +215,11 @@ def to_datetime( ) new_series = ( - arg[unit_rev["year"]].astype("str") + arg[unit_rev["year"]].astype(CUDF_STRING_DTYPE) + "-" - + arg[unit_rev["month"]].astype("str").str.zfill(2) + + arg[unit_rev["month"]].astype(CUDF_STRING_DTYPE).str.zfill(2) + "-" - + arg[unit_rev["day"]].astype("str").str.zfill(2) + + arg[unit_rev["day"]].astype(CUDF_STRING_DTYPE).str.zfill(2) ) format = "%Y-%m-%d" for u in ["h", "m", "s", "ms", "us", "ns"]: @@ -255,9 +256,13 @@ def to_datetime( # float dtype we don't want to type-cast if current_col.dtype.kind in ("O"): try: - current_col = current_col.astype(dtype="int64") + current_col = current_col.astype( + np.dtype(np.int64) + ) except ValueError: - current_col = current_col.astype(dtype="float64") + current_col = current_col.astype( + np.dtype(np.float64) + ) factor = ( column.datetime._unit_to_nanoseconds_conversion[u] @@ -269,7 +274,7 @@ def to_datetime( else: times_column = times_column + (current_col * factor) if times_column is not None: - col = (col.astype(dtype="int64") + times_column).astype( + col = (col.astype(np.dtype(np.int64)) + times_column).astype( dtype=col.dtype ) col = _process_col( @@ -336,7 +341,7 @@ def _process_col( # parsing against `format`. col = ( col.astype(np.dtype(np.int64)) - .astype("str") + .astype(CUDF_STRING_DTYPE) .strptime( dtype=np.dtype("datetime64[us]") if "%f" in format @@ -356,7 +361,7 @@ def _process_col( col = col * factor if format is not None: - col = col.astype("str").strptime( + col = col.astype(CUDF_STRING_DTYPE).strptime( dtype=np.dtype(_unit_dtype_map[unit]), format=format ) else: diff --git a/python/cudf/cudf/core/tools/numeric.py b/python/cudf/cudf/core/tools/numeric.py index 9a4d773d5d6..9746234cfb1 100644 --- a/python/cudf/cudf/core/tools/numeric.py +++ b/python/cudf/cudf/core/tools/numeric.py @@ -127,8 +127,8 @@ def to_numeric( if dtype.kind in "mM": col = col.astype(np.dtype(np.int64)) elif isinstance(dtype, CategoricalDtype): - cat_dtype = col.dtype.type - if _is_non_decimal_numeric_dtype(cat_dtype): + cat_dtype = col.dtype.categories.dtype + if cat_dtype.kind in "iufb": col = col.astype(cat_dtype) else: try: @@ -187,7 +187,7 @@ def to_numeric( else: if col.has_nulls(): # To match pandas, always return a floating type filled with nan. - col = col.astype(float).fillna(np.nan) + col = col.astype(np.dtype(np.float64)).fillna(np.nan) return col.values diff --git a/python/cudf/cudf/core/window/ewm.py b/python/cudf/cudf/core/window/ewm.py index c4a063a50e8..3e8a6ab400c 100644 --- a/python/cudf/cudf/core/window/ewm.py +++ b/python/cudf/cudf/core/window/ewm.py @@ -1,4 +1,4 @@ -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. from __future__ import annotations import warnings @@ -192,7 +192,9 @@ def _apply_agg_column( # pandas does nans in the same positions mathematically. # as such we need to convert the nans to nulls before # passing them in. - to_libcudf_column = source_column.astype("float64").nans_to_nulls() + to_libcudf_column = source_column.astype( + np.dtype(np.float64) + ).nans_to_nulls() return to_libcudf_column.scan( agg_name, True, com=self.com, adjust=self.adjust ) diff --git a/python/cudf/cudf/core/window/rolling.py b/python/cudf/cudf/core/window/rolling.py index 187d1b58dca..23b0d7006b4 100644 --- a/python/cudf/cudf/core/window/rolling.py +++ b/python/cudf/cudf/core/window/rolling.py @@ -19,6 +19,7 @@ from cudf.core.column.column import as_column from cudf.core.mixins import Reducible from cudf.utils import cudautils +from cudf.utils.dtypes import SIZE_TYPE_DTYPE from cudf.utils.utils import GetAttrGetItemMixin if TYPE_CHECKING: @@ -273,12 +274,16 @@ def _apply_agg_column(self, source_column, agg_name): closed=None, step=None, ) - start = as_column(start, dtype="int32") - end = as_column(end, dtype="int32") + start = as_column(start, dtype=SIZE_TYPE_DTYPE) + end = as_column(end, dtype=SIZE_TYPE_DTYPE) idx = as_column(range(len(start))) - preceding_window = (idx - start + np.int32(1)).astype("int32") - following_window = (end - idx - np.int32(1)).astype("int32") + preceding_window = (idx - start + np.int32(1)).astype( + SIZE_TYPE_DTYPE + ) + following_window = (end - idx - np.int32(1)).astype( + SIZE_TYPE_DTYPE + ) window = None else: preceding_window = as_column(self.window) diff --git a/python/cudf/cudf/tests/test_dataframe.py b/python/cudf/cudf/tests/test_dataframe.py index 4851eccd8fd..05bc221bf9d 100644 --- a/python/cudf/cudf/tests/test_dataframe.py +++ b/python/cudf/cudf/tests/test_dataframe.py @@ -2603,8 +2603,7 @@ def test_comparison_binops_df_reindexing(request, pdf, gdf, binop, other): pdf[pdf == 1.0] = 2 gdf[gdf == 1.0] = 2 try: - with pytest.warns(FutureWarning): - d = binop(pdf, other) + d = binop(pdf, other) except Exception: if isinstance(other, (pd.Series, pd.DataFrame)): cudf_other = cudf.from_pandas(other) diff --git a/python/cudf/cudf/tests/test_datetime.py b/python/cudf/cudf/tests/test_datetime.py index f8fb5ccae25..4af7f776c44 100644 --- a/python/cudf/cudf/tests/test_datetime.py +++ b/python/cudf/cudf/tests/test_datetime.py @@ -1639,11 +1639,7 @@ def test_date_range_raise_overflow(): periods = 2 freq = cudf.DateOffset(months=1) with pytest.raises(pd.errors.OutOfBoundsDatetime): - # Extending beyond the max value will trigger a warning when pandas - # does an internal conversion to a Python built-in datetime.datetime - # object, which only supports down to microsecond resolution. - with pytest.warns(UserWarning): - cudf.date_range(start=start, periods=periods, freq=freq) + cudf.date_range(start=start, periods=periods, freq=freq) @pytest.mark.parametrize( @@ -1683,7 +1679,9 @@ def test_date_range_raise_unsupported(freqstr_unsupported): if freqstr_unsupported != "3MS": freqstr_unsupported = freqstr_unsupported.lower() with pytest.raises(ValueError, match="does not yet support"): - with expect_warning_if(PANDAS_GE_220): + with expect_warning_if( + PANDAS_GE_220 and freqstr_unsupported not in {"b", "bh"} + ): cudf.date_range(start=s, end=e, freq=freqstr_unsupported) diff --git a/python/cudf/cudf/tests/test_string_udfs.py b/python/cudf/cudf/tests/test_string_udfs.py index c1369a03031..6eaf25f02ed 100644 --- a/python/cudf/cudf/tests/test_string_udfs.py +++ b/python/cudf/cudf/tests/test_string_udfs.py @@ -101,7 +101,7 @@ def run_udf_test(data, func, dtype): else: result = output - got = cudf.Series._from_column(result.astype(dtype)) + got = cudf.Series._from_column(result.astype(cudf.dtype(dtype))) assert_eq(expect, got, check_dtype=False) with _CUDFNumbaConfig(): udf_str_kernel.forall(len(data))(str_views, output) @@ -110,7 +110,7 @@ def run_udf_test(data, func, dtype): else: result = output - got = cudf.Series._from_column(result.astype(dtype)) + got = cudf.Series._from_column(result.astype(cudf.dtype(dtype))) assert_eq(expect, got, check_dtype=False) diff --git a/python/cudf/cudf/utils/utils.py b/python/cudf/cudf/utils/utils.py index c63d7816d14..fd946937945 100644 --- a/python/cudf/cudf/utils/utils.py +++ b/python/cudf/cudf/utils/utils.py @@ -439,12 +439,12 @@ def _datetime_timedelta_find_and_replace( if replacement.can_cast_safely(original_column.dtype): replacement = replacement.astype(original_column.dtype) if isinstance(to_replace, original_col_class): - to_replace = to_replace.as_numerical_column(dtype=np.dtype("int64")) + to_replace = to_replace.astype(np.dtype(np.int64)) if isinstance(replacement, original_col_class): - replacement = replacement.as_numerical_column(dtype=np.dtype("int64")) + replacement = replacement.astype(np.dtype(np.int64)) try: result_col = ( - original_column.as_numerical_column(dtype=np.dtype("int64")) + original_column.astype(np.dtype(np.int64)) .find_and_replace(to_replace, replacement, all_nan) .astype(original_column.dtype) ) From 0556701708c403d2203c055ba99345a46ac97535 Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Tue, 18 Feb 2025 17:05:07 -0600 Subject: [PATCH 051/129] Compatibility with dask.dataframe's `is_scalar` (#18030) Makes the import of dask dataframe's `is_scalar` dependent on the dask version. Closes https://github.com/rapidsai/cudf/issues/18028 Authors: - Tom Augspurger (https://github.com/TomAugspurger) Approvers: - Benjamin Zaitlen (https://github.com/quasiben) - Matthew Murray (https://github.com/Matt711) URL: https://github.com/rapidsai/cudf/pull/18030 --- python/dask_cudf/dask_cudf/_expr/__init__.py | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/python/dask_cudf/dask_cudf/_expr/__init__.py b/python/dask_cudf/dask_cudf/_expr/__init__.py index 1f757476ce5..e8051eedafb 100644 --- a/python/dask_cudf/dask_cudf/_expr/__init__.py +++ b/python/dask_cudf/dask_cudf/_expr/__init__.py @@ -1,5 +1,9 @@ # Copyright (c) 2024-2025, NVIDIA CORPORATION. +import importlib.metadata + +from packaging.version import Version + import dask import dask.dataframe.dask_expr._shuffle as _shuffle_module from dask.dataframe import get_collection_type @@ -34,7 +38,6 @@ from dask.dataframe.dask_expr._util import ( _convert_to_list, _raise_if_object_series, - is_scalar, ) from dask.dataframe.dask_expr.io.io import ( FusedIO, @@ -46,6 +49,18 @@ ReadParquetPyarrowFS, ) +_dask_version = importlib.metadata.version("dask") + +# TODO: change ">2025.2.0" to ">={next-version}" when released. +DASK_2025_3_0 = Version(_dask_version) > Version("2025.2.0") + + +if DASK_2025_3_0: + from dask.dataframe.utils import is_scalar +else: + from dask.dataframe.dask_expr._util import is_scalar + + __all__ = [ "CumulativeBlockwise", "DXDataFrame", From 000758a278e08b63fba7f8e1828d72f8a806bbee Mon Sep 17 00:00:00 2001 From: Peixin Date: Wed, 19 Feb 2025 08:43:24 +0800 Subject: [PATCH 052/129] Update spark-rapids-jni CI image version to cuda12.8.0 (#18024) spark-rapids-jni is going to drop support for the 12.2.0 CI image, this change is to switch image version to cuda12.8.0 one https://hub.docker.com/r/rapidsai/ci-spark-rapids-jni/tags Authors: - Peixin (https://github.com/pxLi) Approvers: - James Lamb (https://github.com/jameslamb) URL: https://github.com/rapidsai/cudf/pull/18024 --- .github/workflows/spark-rapids-jni.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/spark-rapids-jni.yaml b/.github/workflows/spark-rapids-jni.yaml index 097e97df8c5..996f2212c3f 100644 --- a/.github/workflows/spark-rapids-jni.yaml +++ b/.github/workflows/spark-rapids-jni.yaml @@ -7,7 +7,7 @@ jobs: spark-rapids-jni-build: runs-on: linux-amd64-cpu8 container: - image: rapidsai/ci-spark-rapids-jni:rockylinux8-cuda12.2.0 + image: rapidsai/ci-spark-rapids-jni:rockylinux8-cuda12.8.0 steps: - uses: actions/checkout@v4 with: From f2518b02c4846b5d8087ee9dc724e6a7b7489d9d Mon Sep 17 00:00:00 2001 From: Shruti Shivakumar Date: Tue, 18 Feb 2025 17:54:24 -0800 Subject: [PATCH 053/129] Limit buffer size in reallocation policy in JSON reader (#17940) Addresses https://github.com/rapidsai/cudf/issues/17058 We can split the buffer at the last but one line and pass two buffers to the tokenizer, provided that the size of each of those buffers is less than 2GB. We only need to verify for the second buffer since the first one is guaranteed to be under the limit by the batching logic. Authors: - Shruti Shivakumar (https://github.com/shrshi) Approvers: - David Wendt (https://github.com/davidwendt) - MithunR (https://github.com/mythrocks) - Tianyu Liu (https://github.com/kingcrimsontianyu) URL: https://github.com/rapidsai/cudf/pull/17940 --- cpp/src/io/json/read_json.cu | 352 +++++++++++++++++--------- cpp/tests/io/json/json_test.cpp | 29 ++- cpp/tests/large_strings/json_tests.cu | 134 ++++++++++ 3 files changed, 397 insertions(+), 118 deletions(-) diff --git a/cpp/src/io/json/read_json.cu b/cpp/src/io/json/read_json.cu index 4b0af7d6e81..0c95c2b05e8 100644 --- a/cpp/src/io/json/read_json.cu +++ b/cpp/src/io/json/read_json.cu @@ -78,7 +78,7 @@ class compressed_host_buffer_source final : public datasource { } } - size_t host_read(size_t offset, size_t size, uint8_t* dst) override + std::size_t host_read(std::size_t offset, std::size_t size, uint8_t* dst) override { auto ch_buffer = host_span(reinterpret_cast(_dbuf_ptr->data()), _dbuf_ptr->size()); @@ -97,7 +97,7 @@ class compressed_host_buffer_source final : public datasource { return count; } - std::unique_ptr host_read(size_t offset, size_t size) override + std::unique_ptr host_read(std::size_t offset, std::size_t size) override { auto ch_buffer = host_span(reinterpret_cast(_dbuf_ptr->data()), _dbuf_ptr->size()); @@ -114,10 +114,10 @@ class compressed_host_buffer_source final : public datasource { return std::make_unique(_decompressed_buffer.data() + offset, count); } - std::future device_read_async(size_t offset, - size_t size, - uint8_t* dst, - rmm::cuda_stream_view stream) override + std::future device_read_async(std::size_t offset, + std::size_t size, + uint8_t* dst, + rmm::cuda_stream_view stream) override { auto& thread_pool = pools::tpool(); return thread_pool.submit_task([this, offset, size, dst, stream] { @@ -131,12 +131,12 @@ class compressed_host_buffer_source final : public datasource { [[nodiscard]] bool supports_device_read() const override { return true; } - [[nodiscard]] size_t size() const override { return _decompressed_ch_buffer_size; } + [[nodiscard]] std::size_t size() const override { return _decompressed_ch_buffer_size; } private: std::unique_ptr _dbuf_ptr; compression_type _comptype; - size_t _decompressed_ch_buffer_size; + std::size_t _decompressed_ch_buffer_size; std::vector _decompressed_buffer; }; @@ -208,22 +208,33 @@ size_type find_first_delimiter(device_span d_data, } /** - * @brief Get the byte range between record starts and ends starting from the given range. + * @brief Get the byte range between record starts and ends starting from the given range. The + * actual byte range read and returned will contain complete JSONL records, and will include the + * delimiter at the end of the last record. * * if get_byte_range_offset == 0, then we can skip the first delimiter search * if get_byte_range_offset != 0, then we need to search for the first delimiter in given range. * if not found, skip this chunk, if found, then search for first delimiter in next range until we - * find a delimiter. Use this as actual range for parsing. + * find a delimiter. Use this as actual range for parsing. If the size of actual byte range to be + * parsed is greater than the integer limit (or the requested batch size), then split the ingested + * buffer in two. Note that as long as no single record in the JSONL input is of size larger than + * the requested batch size, we are guaranteed that each of the two buffers will be within the batch + * size limit - the size of the first buffer is capped at the batch limit by the batching logic + * itself, and the second buffer contains only the last record which was incomplete in the initial + * byte range requested. If the size of the actual byte range to be parsed does not exceed batch + * limits, then the second buffer is empty. * * @param sources Data sources to read from * @param reader_opts JSON reader options with range offset and range size * @param stream CUDA stream used for device memory operations and kernel launches - * @returns Data source owning buffer enclosing the bytes read + * @returns A pair of data source owning buffers together enclosing the bytes read. The second + * buffer may or may not be empty depending on the condition described above. */ -datasource::owning_buffer get_record_range_raw_input( - host_span> sources, - json_reader_options const& reader_opts, - rmm::cuda_stream_view stream) +std::pair, + std::optional>> +get_record_range_raw_input(host_span> sources, + json_reader_options const& reader_opts, + rmm::cuda_stream_view stream) { CUDF_FUNC_RANGE(); @@ -232,13 +243,10 @@ datasource::owning_buffer get_record_range_raw_input( auto const delimiter = reader_opts.get_delimiter(); auto const num_extra_delimiters = num_delimiter_chars * sources.size(); std::size_t const chunk_offset = reader_opts.get_byte_range_offset(); - std::size_t chunk_size = reader_opts.get_byte_range_size(); - - CUDF_EXPECTS(total_source_size ? chunk_offset < total_source_size : !chunk_offset, - "Invalid offsetting", - std::invalid_argument); - auto should_load_till_last_source = !chunk_size || chunk_size >= total_source_size - chunk_offset; - chunk_size = should_load_till_last_source ? total_source_size - chunk_offset : chunk_size; + std::size_t const chunk_size = reader_opts.get_byte_range_size(); + // Sanity checks for the byte range offset and size are handled by the batching logic. + // We only need to check if we are reading until the end of the last source in this function. + auto const should_load_till_last_source = chunk_offset + chunk_size == total_source_size; int num_subchunks_prealloced = should_load_till_last_source ? 0 : max_subchunks_prealloced; std::size_t const size_per_subchunk = estimate_size_per_subchunk(chunk_size); @@ -253,14 +261,30 @@ datasource::owning_buffer get_record_range_raw_input( std::int64_t buffer_offset = 0; auto readbufspan = ingest_raw_input(bufspan, sources, chunk_offset, chunk_size, delimiter, stream); + auto const requested_size = readbufspan.size(); auto const shift_for_nonzero_offset = std::min(chunk_offset, 1); auto const first_delim_pos = chunk_offset == 0 ? 0 : find_first_delimiter(readbufspan, delimiter, stream); + + // If we read till the end of the last source, we cannot be sure + // if the last record read ends with a delimiter. In such cases, we add a delimiter + // nevertheless; even if the record terminates + // with a delimiter, adding a extra delimiter does not affect the table constructed since the + // parser ignores empty lines. + auto insert_delimiter = [delimiter, stream](device_span subspan) { + auto last_char = delimiter; + cudf::detail::cuda_memcpy(subspan, host_span(&last_char, 1, false), stream); + }; + + // If the requested byte range ends with a delimiter at the end of line n, we will still need to + // continue reading since the next batch begins at the start of the n+1^th record and skips the + // entire line until the first delimiter is encountered at the end of the line. if (first_delim_pos == -1) { // return empty owning datasource buffer auto empty_buf = rmm::device_buffer(0, stream); - return datasource::owning_buffer(std::move(empty_buf)); + return std::make_pair(datasource::owning_buffer(std::move(empty_buf)), + std::nullopt); } else if (!should_load_till_last_source) { // Find next delimiter std::int64_t next_delim_pos = -1; @@ -285,7 +309,9 @@ datasource::owning_buffer get_record_range_raw_input( // If we have reached the end of source list but the source does not terminate with a // delimiter character next_delim_pos = buffer_offset + readbufspan.size(); + insert_delimiter(bufspan.subspan(next_delim_pos, 1)); } else { + // Reallocate-and-retry policy // Our buffer_size estimate is insufficient to read until the end of the line! We need to // allocate more memory and try again! num_subchunks_prealloced *= 2; @@ -298,73 +324,136 @@ datasource::owning_buffer get_record_range_raw_input( } } - auto const batch_limit = static_cast(std::numeric_limits::max()); - CUDF_EXPECTS(static_cast(next_delim_pos - first_delim_pos - shift_for_nonzero_offset) < - batch_limit, - "The size of the JSON buffer returned by every batch cannot exceed INT_MAX bytes"); - return datasource::owning_buffer( - std::move(buffer), - reinterpret_cast(buffer.data()) + first_delim_pos + shift_for_nonzero_offset, - next_delim_pos - first_delim_pos - shift_for_nonzero_offset); + // If the size of the ingested buffer is less than the batch size, we can simply return the + // buffer as is, and set the optional second buffer to null. + // If the size of the ingested buffer exceed the batch size limits due to the + // reallocate-and-retry policy, we split the ingested buffer in two parts. The second part + // only contains the last record in the buffer, while the first part contains all the remaining + // lines. + // As long as the size of no record exceeds the batch size limit placed, we are guaranteed that + // the returned buffer(s) will be below the batch limit. + auto const batch_size = getenv_or( + "LIBCUDF_JSON_BATCH_SIZE", static_cast(std::numeric_limits::max())); + if (static_cast(next_delim_pos - first_delim_pos - shift_for_nonzero_offset) < + batch_size) { + return std::make_pair( + datasource::owning_buffer( + std::move(buffer), + reinterpret_cast(buffer.data()) + first_delim_pos + shift_for_nonzero_offset, + next_delim_pos - first_delim_pos - shift_for_nonzero_offset + 1), + std::nullopt); + } + device_span bufsubspan = + bufspan.subspan(first_delim_pos + shift_for_nonzero_offset, + requested_size - first_delim_pos - shift_for_nonzero_offset); + auto rev_it_begin = thrust::make_reverse_iterator(bufsubspan.end()); + auto rev_it_end = thrust::make_reverse_iterator(bufsubspan.begin()); + auto const second_last_delimiter_it = + thrust::find(rmm::exec_policy(stream), rev_it_begin, rev_it_end, delimiter); + CUDF_EXPECTS(second_last_delimiter_it != rev_it_end, + "A single JSON line cannot be larger than the batch size limit"); + auto const last_line_size = + next_delim_pos - requested_size + + static_cast(thrust::distance(rev_it_begin, second_last_delimiter_it)); + CUDF_EXPECTS(last_line_size < batch_size, + "A single JSON line cannot be larger than the batch size limit"); + + rmm::device_buffer second_buffer(bufsubspan.data() + static_cast(thrust::distance( + second_last_delimiter_it, rev_it_end)), + last_line_size + 1, + stream); + + return std::make_pair( + datasource::owning_buffer( + std::move(buffer), + reinterpret_cast(buffer.data()) + first_delim_pos + shift_for_nonzero_offset, + next_delim_pos - first_delim_pos - shift_for_nonzero_offset - last_line_size), + datasource::owning_buffer( + std::move(second_buffer), + reinterpret_cast(second_buffer.data()), + second_buffer.size())); } // Add delimiter to end of buffer - possibly adding an empty line to the input buffer - iff we are - // reading till the end of the last source i.e. should_load_till_last_source is true Note that the - // table generated from the JSONL input remains unchanged since empty lines are ignored by the + // reading till the end of the last source i.e. should_load_till_last_source is true. Note that + // the table generated from the JSONL input remains unchanged since empty lines are ignored by the // parser. - size_t num_chars = readbufspan.size() - first_delim_pos - shift_for_nonzero_offset; + std::size_t num_chars = readbufspan.size() - first_delim_pos - shift_for_nonzero_offset; if (num_chars) { - auto last_char = delimiter; - cudf::detail::cuda_memcpy_async( - device_span(reinterpret_cast(buffer.data()), buffer.size()) - .subspan(readbufspan.size(), 1), - host_span(&last_char, 1, false), - stream); + insert_delimiter(bufspan.subspan(readbufspan.size(), 1)); num_chars++; } - return datasource::owning_buffer( - std::move(buffer), - reinterpret_cast(buffer.data()) + first_delim_pos + shift_for_nonzero_offset, - num_chars); + return std::make_pair( + datasource::owning_buffer( + std::move(buffer), + reinterpret_cast(buffer.data()) + first_delim_pos + shift_for_nonzero_offset, + num_chars), + std::nullopt); } -// Helper function to read the current batch using byte range offsets and size -// passed -table_with_metadata read_batch(host_span> sources, - json_reader_options const& reader_opts, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) +/** + * @brief Helper function to read the current batch using the byte range offsets and size + * passed, normalize it, and construct a partial table. + */ +std::pair> read_batch( + host_span> sources, + json_reader_options const& reader_opts, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) { CUDF_FUNC_RANGE(); - datasource::owning_buffer bufview = - get_record_range_raw_input(sources, reader_opts, stream); + // The second owning buffer in the pair returned by get_record_range_raw_input may not be + // populated depending on the size of the actual byte range read. The first owning buffer will + // always be non-empty. + auto owning_buffers = get_record_range_raw_input(sources, reader_opts, stream); // If input JSON buffer has single quotes and option to normalize single quotes is enabled, // invoke pre-processing FST if (reader_opts.is_enabled_normalize_single_quotes()) { - normalize_single_quotes( - bufview, reader_opts.get_delimiter(), stream, cudf::get_current_device_resource_ref()); + normalize_single_quotes(owning_buffers.first, + reader_opts.get_delimiter(), + stream, + cudf::get_current_device_resource_ref()); + stream.synchronize(); } - auto buffer = - cudf::device_span(reinterpret_cast(bufview.data()), bufview.size()); - stream.synchronize(); - return device_parse_nested_json(buffer, reader_opts, stream, mr); + auto buffer = cudf::device_span( + reinterpret_cast(owning_buffers.first.data()), owning_buffers.first.size()); + auto first_partial_table = device_parse_nested_json(buffer, reader_opts, stream, mr); + if (!owning_buffers.second.has_value()) + return std::make_pair(std::move(first_partial_table), std::nullopt); + + // Repeat the normalization and table construction steps for the second buffer if it exists + if (reader_opts.is_enabled_normalize_single_quotes()) { + normalize_single_quotes(owning_buffers.second.value(), + reader_opts.get_delimiter(), + stream, + cudf::get_current_device_resource_ref()); + stream.synchronize(); + } + buffer = cudf::device_span( + reinterpret_cast(owning_buffers.second.value().data()), + owning_buffers.second.value().size()); + auto second_partial_table = device_parse_nested_json(buffer, reader_opts, stream, mr); + return std::make_pair(std::move(first_partial_table), std::move(second_partial_table)); } +/** + * @brief Helper function that implements the batching logic for the JSONL reader. + * The goal of the batched reader is to handle reading multiple JSONL sources whose total cumulative + * size exceeds the integer limit imposed by the JSON tokenizer. The batching logic divides the + * requested input byte range spanning sources into smaller batches, each of which itself spans + * multiple sources. The batches are constructed such that the byte subrange in each batch does not + * exceed the batch size, which is either set using the environment variable + * LIBCUDF_JSON_BATCH_SIZE, or is set to a little under the integer limit. Note that batching + * sources does not work for for regular JSON inputs. + */ table_with_metadata read_json_impl(host_span> sources, json_reader_options const& reader_opts, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) { - /* - * The batched JSON reader enforces that the size of each batch is at most INT_MAX - * bytes (~2.14GB). Batches are defined to be byte range chunks - characterized by - * chunk offset and chunk size - that may span across multiple source files. - * Note that the batched reader does not work for compressed inputs or for regular - * JSON inputs. - */ std::size_t const total_source_size = sources_size(sources, 0, 0); // Batching is enabled only for JSONL inputs, not regular JSON files @@ -372,19 +461,20 @@ table_with_metadata read_json_impl(host_span> source reader_opts.is_enabled_lines() || total_source_size < std::numeric_limits::max(), "Parsing Regular JSON inputs of size greater than INT_MAX bytes is not supported"); - std::size_t chunk_offset = reader_opts.get_byte_range_offset(); + // Sanity checks of byte range offset and clamping of byte range size + std::size_t const chunk_offset = reader_opts.get_byte_range_offset(); + CUDF_EXPECTS(total_source_size ? chunk_offset < total_source_size : !chunk_offset, + "Invalid byte range offset", + std::invalid_argument); std::size_t chunk_size = reader_opts.get_byte_range_size(); chunk_size = !chunk_size ? total_source_size - chunk_offset : std::min(chunk_size, total_source_size - chunk_offset); std::size_t const batch_size = get_batch_size(chunk_size); - /* - * Identify the position (zero-indexed) of starting source file from which to begin - * batching based on byte range offset. If the offset is larger than the sum of all - * source sizes, then start_source is total number of source files i.e. no file is - * read - */ - + // Identify the position (zero-indexed) of starting source file from which to begin + // batching based on byte range offset. If the offset is larger than the sum of all + // source sizes, then start_source is total number of source files i.e. no file is + // read. // Prefix sum of source file sizes std::size_t pref_source_size = 0; // Starting source file from which to being batching evaluated using byte range offset @@ -395,12 +485,10 @@ table_with_metadata read_json_impl(host_span> source } return sources.size(); }(); - /* - * Construct batches of byte ranges spanning source files, with the starting position of batches - * indicated by `batch_offsets`. `pref_bytes_size` gives the bytes position from which the current - * batch begins, and `end_bytes_size` gives the terminal bytes position after which reading - * stops. - */ + // Construct batches of byte ranges spanning source files, with the starting position of batches + // indicated by `batch_offsets`. `pref_bytes_size` gives the bytes position from which the current + // batch begins, and `end_bytes_size` gives the terminal bytes position after which reading + // stops. std::size_t pref_bytes_size = chunk_offset; std::size_t end_bytes_size = chunk_offset + chunk_size; std::vector batch_offsets{pref_bytes_size}; @@ -416,15 +504,30 @@ table_with_metadata read_json_impl(host_span> source } i++; } - /* - * If there is a single batch, then we can directly return the table without the - * unnecessary concatenate. The size of batch_offsets is 1 if all sources are empty, - * or if end_bytes_size is larger than total_source_size. - */ - if (batch_offsets.size() <= 2) return read_batch(sources, reader_opts, stream, mr); std::vector partial_tables; json_reader_options batched_reader_opts{reader_opts}; + batched_reader_opts.set_byte_range_offset(chunk_offset); + batched_reader_opts.set_byte_range_size(chunk_size); + + // lambda to insert the partial tables into the vector. Since read_batch function returns a pair + // of partial tables where the second table is optional, we insert a table into the vector only if + // it is non-empty + auto insert_partial_tables = + [&partial_tables]( + std::pair>&& partial_table_pair) { + if (partial_table_pair.first.tbl->num_columns() == 0 && + partial_table_pair.first.tbl->num_rows() == 0) + return false; + partial_tables.emplace_back(std::move(partial_table_pair.first)); + if (partial_table_pair.second.has_value()) { + if (partial_table_pair.second.value().tbl->num_columns() == 0 && + partial_table_pair.second.value().tbl->num_rows() == 0) + return false; + partial_tables.emplace_back(std::move(partial_table_pair.second.value())); + } + return true; + }; // recursive lambda to construct schema_element. Here, we assume that the table from the // first batch contains all the columns in the concatenated table, and that the partial tables @@ -474,38 +577,52 @@ table_with_metadata read_json_impl(host_span> source return schema; }; - batched_reader_opts.set_byte_range_offset(batch_offsets[0]); - batched_reader_opts.set_byte_range_size(batch_offsets[1] - batch_offsets[0]); - partial_tables.emplace_back( - read_batch(sources, batched_reader_opts, stream, cudf::get_current_device_resource_ref())); - - auto& tbl = partial_tables.back().tbl; - std::vector children; - for (size_type j = 0; j < tbl->num_columns(); j++) { - children.emplace_back(tbl->get_column(j)); - } - batched_reader_opts.set_dtypes( - construct_schema(children, partial_tables.back().metadata.schema_info, schema)); - batched_reader_opts.enable_prune_columns(true); - - // Dispatch individual batches to read_batch and push the resulting table into - // partial_tables array. Note that the reader options need to be updated for each - // batch to adjust byte range offset and byte range size. - for (std::size_t batch_offset_pos = 1; batch_offset_pos < batch_offsets.size() - 1; - batch_offset_pos++) { - batched_reader_opts.set_byte_range_offset(batch_offsets[batch_offset_pos]); - batched_reader_opts.set_byte_range_size(batch_offsets[batch_offset_pos + 1] - - batch_offsets[batch_offset_pos]); - auto partial_table = - read_batch(sources, batched_reader_opts, stream, cudf::get_current_device_resource_ref()); - if (partial_table.tbl->num_columns() == 0 && partial_table.tbl->num_rows() == 0) { - CUDF_EXPECTS(batch_offset_pos == batch_offsets.size() - 2, - "Only the partial table generated by the last batch can be empty"); - break; + + if (batch_offsets.size() <= 2) { + // single batch + auto has_inserted = insert_partial_tables( + read_batch(sources, batched_reader_opts, stream, cudf::get_current_device_resource_ref())); + if (!has_inserted) { + return table_with_metadata{std::make_unique
(std::vector>{}), + {std::vector{}}}; + } + } else { + // multiple batches + batched_reader_opts.set_byte_range_offset(batch_offsets[0]); + batched_reader_opts.set_byte_range_size(batch_offsets[1] - batch_offsets[0]); + insert_partial_tables( + read_batch(sources, batched_reader_opts, stream, cudf::get_current_device_resource_ref())); + + auto& tbl = partial_tables.back().tbl; + std::vector children; + for (size_type j = 0; j < tbl->num_columns(); j++) { + children.emplace_back(tbl->get_column(j)); + } + batched_reader_opts.set_dtypes( + construct_schema(children, partial_tables.back().metadata.schema_info, schema)); + batched_reader_opts.enable_prune_columns(true); + + // Dispatch individual batches to read_batch and push the resulting table into + // partial_tables array. Note that the reader options need to be updated for each + // batch to adjust byte range offset and byte range size. + for (std::size_t batch_offset_pos = 1; batch_offset_pos < batch_offsets.size() - 1; + batch_offset_pos++) { + batched_reader_opts.set_byte_range_offset(batch_offsets[batch_offset_pos]); + batched_reader_opts.set_byte_range_size(batch_offsets[batch_offset_pos + 1] - + batch_offsets[batch_offset_pos]); + auto has_inserted = insert_partial_tables( + read_batch(sources, batched_reader_opts, stream, cudf::get_current_device_resource_ref())); + + if (!has_inserted) { + CUDF_EXPECTS(batch_offset_pos == batch_offsets.size() - 2, + "Only the partial table generated by the last batch can be empty"); + break; + } } - partial_tables.emplace_back(std::move(partial_table)); } + // If there is a single partial table, then there is no need to concatenate + if (partial_tables.size() == 1) return std::move(partial_tables[0]); auto expects_schema_equality = std::all_of(partial_tables.begin() + 1, partial_tables.end(), @@ -538,7 +655,7 @@ device_span ingest_raw_input(device_span buffer, // line of file i+1 don't end up on the same JSON line, if file i does not already end with a line // delimiter. auto constexpr num_delimiter_chars = 1; - std::vector> thread_tasks; + std::vector> thread_tasks; auto delimiter_map = cudf::detail::make_empty_host_vector(sources.size(), stream); std::vector prefsum_source_sizes(sources.size()); @@ -556,7 +673,7 @@ device_span ingest_raw_input(device_span buffer, auto const total_bytes_to_read = std::min(range_size, prefsum_source_sizes.back() - range_offset); range_offset -= start_source ? prefsum_source_sizes[start_source - 1] : 0; - size_t const num_streams = + std::size_t const num_streams = std::min({sources.size() - start_source + 1, cudf::detail::global_cuda_stream_pool().get_stream_pool_size(), pools::tpool().get_thread_count()}); @@ -605,7 +722,8 @@ device_span ingest_raw_input(device_span buffer, thread_tasks.begin(), thread_tasks.end(), std::size_t{0}, [](std::size_t sum, auto& task) { return sum + task.get(); }); - CUDF_EXPECTS(bytes_read == total_bytes_to_read, "something's fishy"); + CUDF_EXPECTS(bytes_read == total_bytes_to_read, + "Incorrect number of bytes read by multithreaded reader"); } return buffer.first(bytes_read + (delimiter_map.size() * num_delimiter_chars)); diff --git a/cpp/tests/io/json/json_test.cpp b/cpp/tests/io/json/json_test.cpp index 00f46975fdc..89666c073cd 100644 --- a/cpp/tests/io/json/json_test.cpp +++ b/cpp/tests/io/json/json_test.cpp @@ -660,13 +660,40 @@ TEST_P(JsonReaderParamTest, JsonLinesFileInput) CUDF_TEST_EXPECT_COLUMNS_EQUAL(result.tbl->get_column(1), float64_wrapper{{1.1, 2.2}}); } -TEST_F(JsonReaderTest, JsonLinesByteRange) +TEST_F(JsonReaderTest, JsonLinesByteRangeCompleteRecord) { const std::string fname = temp_env->get_temp_dir() + "JsonLinesByteRangeTest.json"; std::ofstream outfile(fname, std::ofstream::out); outfile << "[1000]\n[2000]\n[3000]\n[4000]\n[5000]\n[6000]\n[7000]\n[8000]\n[9000]\n"; outfile.close(); + // Requesting 0]\n[3000]\n[4000]\n[5000]\n but reading 0]\n[3000]\n[4000]\n[5000]\n[6000]\n + cudf::io::json_reader_options in_options = + cudf::io::json_reader_options::builder(cudf::io::source_info{fname}) + .lines(true) + .byte_range_offset(11) + .byte_range_size(24); + + cudf::io::table_with_metadata result = cudf::io::read_json(in_options); + + EXPECT_EQ(result.tbl->num_columns(), 1); + EXPECT_EQ(result.tbl->num_rows(), 4); + + EXPECT_EQ(result.tbl->get_column(0).type().id(), cudf::type_id::INT64); + EXPECT_EQ(result.metadata.schema_info[0].name, "0"); + + CUDF_TEST_EXPECT_COLUMNS_EQUAL(result.tbl->get_column(0), + int64_wrapper{{3000, 4000, 5000, 6000}}); +} + +TEST_F(JsonReaderTest, JsonLinesByteRangeIncompleteRecord) +{ + const std::string fname = temp_env->get_temp_dir() + "JsonLinesByteRangeTest.json"; + std::ofstream outfile(fname, std::ofstream::out); + outfile << "[1000]\n[2000]\n[3000]\n[4000]\n[5000]\n[6000]\n[7000]\n[8000]\n[9000]\n"; + outfile.close(); + + // Reading 0]\n[3000]\n[4000]\n[50 cudf::io::json_reader_options in_options = cudf::io::json_reader_options::builder(cudf::io::source_info{fname}) .lines(true) diff --git a/cpp/tests/large_strings/json_tests.cu b/cpp/tests/large_strings/json_tests.cu index 205fb12c4dd..b3f6a99ed51 100644 --- a/cpp/tests/large_strings/json_tests.cu +++ b/cpp/tests/large_strings/json_tests.cu @@ -16,8 +16,11 @@ #include "../io/json/json_utils.cuh" #include "io/comp/comp.hpp" +#include "io/comp/io_uncomp.hpp" #include "large_strings_fixture.hpp" +#include +#include #include #include @@ -195,3 +198,134 @@ TEST_P(JsonLargeReaderTest, MultiBatchWithNulls) // Read full test data via existing, nested JSON lines reader CUDF_EXPECT_NO_THROW(cudf::io::read_json(cjson_lines_options)); } + +TEST_P(JsonLargeReaderTest, MultiBatchDoubleBufferInput) +{ + cudf::io::compression_type const comptype = GetParam(); + + // This test constructs a JSON input of size two times the batch size but sets the batch boundary + // after the start of the last record in the batch i.e. the input is constructed such that the + // size of the last record is approximately the same as the size of all preceding records. Since + // the reader now ends up reading twice the allowed batch size per batch, it has to split the read + // buffer in two, each part of size <= the batch size. + std::string json_string = R"( + { "a": { "y" : 6}, "b" : [1, 2, 3], "c": "11" } + { "a": { "y" : 6}, "b" : [4, 5 ], "c": "12" } + { "a": { "y" : 6}, "b" : [6 ], "c": "13" } + { "a": { "y" : 6}, "b" : [7 ], "c": "14" } + )"; + std::size_t const batch_size = json_string.size() + 1; + // set smaller batch_size to reduce file size and execution time + this->set_batch_size(batch_size); + + std::string really_long_string = R"(libcudf)"; + std::size_t const log_repetitions = static_cast( + std::floor(std::log2(static_cast(json_string.size()) / really_long_string.size()))); + really_long_string.reserve(really_long_string.size() * (1UL << log_repetitions)); + for (std::size_t i = 0; i < log_repetitions; i++) { + really_long_string += really_long_string; + } + std::string last_line = R"({ "a": { "y" : 6}, "b" : [1, 2, 3], "c": ")"; + last_line += really_long_string + "\" }\n"; + json_string += last_line; + + std::vector cdata; + if (comptype != cudf::io::compression_type::NONE) { + cdata = cudf::io::detail::compress( + comptype, + cudf::host_span(reinterpret_cast(json_string.data()), + json_string.size()), + cudf::get_default_stream()); + } else { + cdata = std::vector( + reinterpret_cast(json_string.data()), + reinterpret_cast(json_string.data()) + json_string.size()); + } + + constexpr int num_sources = 3; + std::vector> chostbufs( + num_sources, + cudf::host_span(reinterpret_cast(cdata.data()), cdata.size())); + + // Initialize parsing options (reading json lines) + cudf::io::json_reader_options cjson_lines_options = + cudf::io::json_reader_options::builder( + cudf::io::source_info{ + cudf::host_span>(chostbufs.data(), chostbufs.size())}) + .lines(true) + .compression(comptype); + + // Read full test data via existing, nested JSON lines reader + auto const result = cudf::io::read_json(cjson_lines_options); + + ASSERT_EQ(result.tbl->num_columns(), 3); + ASSERT_EQ(result.tbl->num_rows(), 15); + + ASSERT_EQ(result.metadata.schema_info.size(), 3); + EXPECT_EQ(result.metadata.schema_info[0].name, "a"); + EXPECT_EQ(result.metadata.schema_info[1].name, "b"); + EXPECT_EQ(result.metadata.schema_info[2].name, "c"); + + EXPECT_EQ(result.tbl->get_column(2).type().id(), cudf::type_id::STRING); + auto expected_c_col = std::vector{"11", "12", "13", "14", really_long_string}; + auto single_src_ccol_size = expected_c_col.size(); + expected_c_col.resize(single_src_ccol_size * num_sources); + for (int i = 1; i <= num_sources - 1; i++) + std::copy(expected_c_col.begin(), + expected_c_col.begin() + single_src_ccol_size, + expected_c_col.begin() + (i * single_src_ccol_size)); + + CUDF_TEST_EXPECT_COLUMNS_EQUAL( + result.tbl->get_column(2), + cudf::test::strings_column_wrapper(expected_c_col.begin(), expected_c_col.end())); +} + +TEST_P(JsonLargeReaderTest, OverBatchLimitLine) +{ + cudf::io::compression_type const comptype = GetParam(); + + // This test constructs a JSONL input of size three times the batch limit. The input contains a + // single JSONL which will be completely read in the first batch itself. Since we cannot divide a + // single line, we expect the test to throw + std::string json_string = R"({ "a": { "y" : 6}, "b" : [1, 2, 3], "c": ")"; + std::string really_long_string = R"(libcudf)"; + std::size_t const log_repetitions = 5; + really_long_string.reserve(really_long_string.size() * (1UL << log_repetitions)); + for (std::size_t i = 0; i < log_repetitions; i++) { + really_long_string += really_long_string; + } + json_string += really_long_string + "\" }\n"; + + std::size_t const batch_size = json_string.size() / 3; + // set smaller batch_size to reduce file size and execution time + this->set_batch_size(batch_size); + + std::vector cdata; + if (comptype != cudf::io::compression_type::NONE) { + cdata = cudf::io::detail::compress( + comptype, + cudf::host_span(reinterpret_cast(json_string.data()), + json_string.size()), + cudf::get_default_stream()); + } else { + cdata = std::vector( + reinterpret_cast(json_string.data()), + reinterpret_cast(json_string.data()) + json_string.size()); + } + + constexpr int num_sources = 1; + std::vector> chostbufs( + num_sources, + cudf::host_span(reinterpret_cast(cdata.data()), cdata.size())); + + // Initialize parsing options (reading json lines) + cudf::io::json_reader_options cjson_lines_options = + cudf::io::json_reader_options::builder( + cudf::io::source_info{ + cudf::host_span>(chostbufs.data(), chostbufs.size())}) + .lines(true) + .compression(comptype); + + // Read full test data via existing, nested JSON lines reader + EXPECT_THROW(cudf::io::read_json(cjson_lines_options), cudf::logic_error); +} From b71a464bc2ff7f5463c3d2190cd39c3c081220a6 Mon Sep 17 00:00:00 2001 From: Michael Schellenberger Costa Date: Wed, 19 Feb 2025 03:04:47 +0100 Subject: [PATCH 054/129] Replace `cub::Int2Type` with `cuda::std::integral_constant` (#18013) `cub::Int2Type` is deprecated and will be removed in a future CCCL release Authors: - Michael Schellenberger Costa (https://github.com/miscco) - David Wendt (https://github.com/davidwendt) Approvers: - David Wendt (https://github.com/davidwendt) - Basit Ayantunde (https://github.com/lamarrr) - Yunsong Wang (https://github.com/PointKernel) URL: https://github.com/rapidsai/cudf/pull/18013 --- .../cudf/column/column_device_view.cuh | 4 +- .../cudf/detail/utilities/integer_utils.hpp | 6 +- cpp/src/io/fst/agent_dfa.cuh | 121 ++++++++++-------- 3 files changed, 75 insertions(+), 56 deletions(-) diff --git a/cpp/include/cudf/column/column_device_view.cuh b/cpp/include/cudf/column/column_device_view.cuh index 990dfee2d17..62da6860192 100644 --- a/cpp/include/cudf/column/column_device_view.cuh +++ b/cpp/include/cudf/column/column_device_view.cuh @@ -59,8 +59,8 @@ namespace CUDF_EXPORT cudf { * */ struct nullate { - struct YES : cuda::std::bool_constant {}; - struct NO : cuda::std::bool_constant {}; + struct YES : cuda::std::true_type {}; + struct NO : cuda::std::false_type {}; /** * @brief `nullate::DYNAMIC` defers the determination of nullability to run time rather than * compile time. The calling code is responsible for specifying whether or not nulls are diff --git a/cpp/include/cudf/detail/utilities/integer_utils.hpp b/cpp/include/cudf/detail/utilities/integer_utils.hpp index 135f645817e..2589b84ec04 100644 --- a/cpp/include/cudf/detail/utilities/integer_utils.hpp +++ b/cpp/include/cudf/detail/utilities/integer_utils.hpp @@ -120,7 +120,7 @@ CUDF_HOST_DEVICE constexpr S div_rounding_up_unsafe(S const& dividend, T const& namespace detail { template -CUDF_HOST_DEVICE constexpr I div_rounding_up_safe(cuda::std::integral_constant, +CUDF_HOST_DEVICE constexpr I div_rounding_up_safe(cuda::std::false_type, I dividend, I divisor) noexcept { @@ -130,7 +130,7 @@ CUDF_HOST_DEVICE constexpr I div_rounding_up_safe(cuda::std::integral_constant -CUDF_HOST_DEVICE constexpr I div_rounding_up_safe(cuda::std::integral_constant, +CUDF_HOST_DEVICE constexpr I div_rounding_up_safe(cuda::std::true_type, I dividend, I divisor) noexcept { @@ -160,7 +160,7 @@ CUDF_HOST_DEVICE constexpr I div_rounding_up_safe(cuda::std::integral_constant CUDF_HOST_DEVICE constexpr I div_rounding_up_safe(I dividend, I divisor) noexcept { - using i_is_a_signed_type = cuda::std::integral_constant>; + using i_is_a_signed_type = cuda::std::bool_constant>; return detail::div_rounding_up_safe(i_is_a_signed_type{}, dividend, divisor); } diff --git a/cpp/src/io/fst/agent_dfa.cuh b/cpp/src/io/fst/agent_dfa.cuh index d8f8e13a164..a4b55fb8501 100644 --- a/cpp/src/io/fst/agent_dfa.cuh +++ b/cpp/src/io/fst/agent_dfa.cuh @@ -147,7 +147,7 @@ class DFAWriteCallbackWrapper { StateIndexT const new_state, SymbolIndexT const symbol_id, SymbolT const read_symbol, - cub::Int2Type /*MaxTranslatedOutChars*/) + cuda::std::integral_constant /*MaxTranslatedOutChars*/) { uint32_t const count = transducer_table(old_state, symbol_id, read_symbol); @@ -174,7 +174,7 @@ class DFAWriteCallbackWrapper { StateIndexT const new_state, SymbolIndexT const symbol_id, SymbolT const read_symbol, - cub::Int2Type) + cuda::std::integral_constant) { uint32_t const count = transducer_table(old_state, symbol_id, read_symbol); @@ -197,7 +197,7 @@ class DFAWriteCallbackWrapper { new_state, symbol_id, read_symbol, - cub::Int2Type{}); + cuda::std::integral_constant{}); } __device__ __forceinline__ void TearDown() {} @@ -444,15 +444,12 @@ struct AgentDFA { { } - template + template __device__ __forceinline__ static void ThreadParse(SymbolMatcherT const& symbol_matcher, CharT const* chars, SymbolIndexT const& max_num_chars, CallbackOpT callback_op, - cub::Int2Type /*IS_FULL_BLOCK*/) + cuda::std::bool_constant) { // Iterate over symbols #pragma unroll @@ -467,16 +464,18 @@ struct AgentDFA { template - __device__ __forceinline__ void GetThreadStateTransitions( - SymbolMatcherT const& symbol_matcher, - CharT const* chars, - SymbolIndexT const& max_num_chars, - StateTransitionOpT& state_transition_op, - cub::Int2Type /*IS_FULL_BLOCK*/) + bool IS_FULL_BLOCK> + __device__ __forceinline__ void GetThreadStateTransitions(SymbolMatcherT const& symbol_matcher, + CharT const* chars, + SymbolIndexT const& max_num_chars, + StateTransitionOpT& state_transition_op, + cuda::std::bool_constant) { - ThreadParse( - symbol_matcher, chars, max_num_chars, state_transition_op, cub::Int2Type()); + ThreadParse(symbol_matcher, + chars, + max_num_chars, + state_transition_op, + cuda::std::bool_constant()); } //--------------------------------------------------------------------- @@ -486,8 +485,8 @@ struct AgentDFA { __device__ __forceinline__ void LoadBlock(CharInItT d_chars, OffsetT const block_offset, OffsetT const num_total_symbols, - cub::Int2Type /*IS_FULL_BLOCK*/, - cub::Int2Type<1> /*ALIGNMENT*/) + cuda::std::true_type /*IS_FULL_BLOCK*/, + cuda::std::integral_constant /*ALIGNMENT*/) { CharT thread_chars[SYMBOLS_PER_THREAD]; @@ -507,8 +506,8 @@ struct AgentDFA { __device__ __forceinline__ void LoadBlock(CharInItT d_chars, OffsetT const block_offset, OffsetT const num_total_symbols, - cub::Int2Type /*IS_FULL_BLOCK*/, - cub::Int2Type<1> /*ALIGNMENT*/) + cuda::std::false_type /*IS_FULL_BLOCK*/, + cuda::std::integral_constant /*ALIGNMENT*/) { CharT thread_chars[SYMBOLS_PER_THREAD]; @@ -530,11 +529,12 @@ struct AgentDFA { //--------------------------------------------------------------------- // LOADING FULL BLOCK OF CHARACTERS, ALIASED //--------------------------------------------------------------------- - __device__ __forceinline__ void LoadBlock(CharT const* d_chars, - OffsetT const block_offset, - OffsetT const num_total_symbols, - cub::Int2Type /*IS_FULL_BLOCK*/, - cub::Int2Type /*ALIGNMENT*/) + __device__ __forceinline__ void LoadBlock( + CharT const* d_chars, + OffsetT const block_offset, + OffsetT const num_total_symbols, + cuda::std::true_type /*IS_FULL_BLOCK*/, + cuda::std::integral_constant /*ALIGNMENT*/) { AliasedLoadT thread_units[UINTS_PER_THREAD]; @@ -551,11 +551,12 @@ struct AgentDFA { //--------------------------------------------------------------------- // LOADING PARTIAL BLOCK OF CHARACTERS, ALIASED //--------------------------------------------------------------------- - __device__ __forceinline__ void LoadBlock(CharT const* d_chars, - OffsetT const block_offset, - OffsetT const num_total_symbols, - cub::Int2Type /*IS_FULL_BLOCK*/, - cub::Int2Type /*ALIGNMENT*/) + __device__ __forceinline__ void LoadBlock( + CharT const* d_chars, + OffsetT const block_offset, + OffsetT const num_total_symbols, + cuda::std::false_type /*IS_FULL_BLOCK*/, + cuda::std::integral_constant /*ALIGNMENT*/) { AliasedLoadT thread_units[UINTS_PER_THREAD]; @@ -586,19 +587,31 @@ struct AgentDFA { // Check if pointer is aligned to four bytes if (((uintptr_t)(void const*)(d_chars + block_offset) % 4) == 0) { if (block_offset + SYMBOLS_PER_UINT_BLOCK < num_total_symbols) { - LoadBlock( - d_chars, block_offset, num_total_symbols, cub::Int2Type(), cub::Int2Type<4>()); + LoadBlock(d_chars, + block_offset, + num_total_symbols, + cuda::std::true_type(), + cuda::std::integral_constant()); } else { - LoadBlock( - d_chars, block_offset, num_total_symbols, cub::Int2Type(), cub::Int2Type<1>()); + LoadBlock(d_chars, + block_offset, + num_total_symbols, + cuda::std::false_type(), + cuda::std::integral_constant()); } } else { if (block_offset + SYMBOLS_PER_UINT_BLOCK < num_total_symbols) { - LoadBlock( - d_chars, block_offset, num_total_symbols, cub::Int2Type(), cub::Int2Type<1>()); + LoadBlock(d_chars, + block_offset, + num_total_symbols, + cuda::std::true_type(), + cuda::std::integral_constant()); } else { - LoadBlock( - d_chars, block_offset, num_total_symbols, cub::Int2Type(), cub::Int2Type<1>()); + LoadBlock(d_chars, + block_offset, + num_total_symbols, + cuda::std::false_type(), + cuda::std::integral_constant()); } } } @@ -610,11 +623,17 @@ struct AgentDFA { { // Check if we are loading a full tile of data if (block_offset + SYMBOLS_PER_UINT_BLOCK < num_total_symbols) { - LoadBlock( - d_chars, block_offset, num_total_symbols, cub::Int2Type(), cub::Int2Type<1>()); + LoadBlock(d_chars, + block_offset, + num_total_symbols, + cuda::std::true_type(), + cuda::std::integral_constant()); } else { - LoadBlock( - d_chars, block_offset, num_total_symbols, cub::Int2Type(), cub::Int2Type<1>()); + LoadBlock(d_chars, + block_offset, + num_total_symbols, + cuda::std::false_type(), + cuda::std::integral_constant()); } } @@ -648,14 +667,14 @@ struct AgentDFA { // Parse thread's symbols and transition the state-vector if (is_full_block) { GetThreadStateTransitions( - symbol_matcher, t_chars, num_block_chars, transition_op, cub::Int2Type()); + symbol_matcher, t_chars, num_block_chars, transition_op, cuda::std::true_type()); } else { GetThreadStateTransitions( - symbol_matcher, t_chars, num_block_chars, transition_op, cub::Int2Type()); + symbol_matcher, t_chars, num_block_chars, transition_op, cuda::std::false_type()); } } - template @@ -667,7 +686,7 @@ struct AgentDFA { OffsetT const num_total_symbols, StateIndexT& state, CallbackOpT& callback_op, - cub::Int2Type) + cuda::std::bool_constant) { using StateTransitionOpT = StateTransitionOp; @@ -693,10 +712,10 @@ struct AgentDFA { // Parse thread's symbols and transition the state-vector if (is_full_block) { GetThreadStateTransitions( - symbol_matcher, t_chars, num_block_chars, transition_op, cub::Int2Type()); + symbol_matcher, t_chars, num_block_chars, transition_op, cuda::std::true_type()); } else { GetThreadStateTransitions( - symbol_matcher, t_chars, num_block_chars, transition_op, cub::Int2Type()); + symbol_matcher, t_chars, num_block_chars, transition_op, cuda::std::false_type()); } callback_op.TearDown(); @@ -893,7 +912,7 @@ __launch_bounds__(int32_t(AgentDFAPolicy::BLOCK_THREADS)) CUDF_KERNEL num_chars, state, count_chars_callback_op, - cub::Int2Type()); + cuda::std::bool_constant()); __syncthreads(); @@ -954,7 +973,7 @@ __launch_bounds__(int32_t(AgentDFAPolicy::BLOCK_THREADS)) CUDF_KERNEL num_chars, t_start_state, write_translated_callback_op, - cub::Int2Type()); + cuda::std::true_type()); } } From 78e59c9f71d97a6aa9d2d57a2e1191cf5bc5fe65 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Tue, 18 Feb 2025 18:09:02 -0800 Subject: [PATCH 055/129] Remove cudf._lib.column in favor of pylibcudf. (#17760) Removes `cudf._lib.column.Column` and moves its methods and attributes to `column.core.column.ColumnBase` * Some methods in `cudf.core._internals` needed to start returning `pylibcudf.Column`s to avoid circular imports * Folds in some doc updates from reviews in https://github.com/rapidsai/cudf/pull/17701 closes https://github.com/rapidsai/cudf/issues/17317 Authors: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Bradley Dice (https://github.com/bdice) - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/17760 --- python/cudf/cudf/_lib/CMakeLists.txt | 2 +- python/cudf/cudf/_lib/column.pxd | 43 - python/cudf/cudf/_lib/column.pyi | 82 -- python/cudf/cudf/_lib/column.pyx | 913 ------------------ python/cudf/cudf/core/_base_index.py | 40 +- python/cudf/cudf/core/_internals/binaryop.py | 9 +- python/cudf/cudf/core/_internals/copying.py | 24 +- python/cudf/cudf/core/_internals/search.py | 17 +- python/cudf/cudf/core/_internals/sorting.py | 36 +- .../cudf/core/_internals/stream_compaction.py | 15 +- python/cudf/cudf/core/_internals/timezones.py | 12 +- python/cudf/cudf/core/column/categorical.py | 4 +- python/cudf/cudf/core/column/column.py | 499 +++++++++- python/cudf/cudf/core/column/datetime.py | 7 +- python/cudf/cudf/core/column/string.py | 108 ++- python/cudf/cudf/core/cut.py | 7 +- python/cudf/cudf/core/dataframe.py | 18 +- python/cudf/cudf/core/frame.py | 45 +- python/cudf/cudf/core/groupby/groupby.py | 22 +- python/cudf/cudf/core/index.py | 31 +- python/cudf/cudf/core/indexed_frame.py | 88 +- python/cudf/cudf/core/join/join.py | 48 +- python/cudf/cudf/core/multiindex.py | 6 +- python/cudf/cudf/core/resample.py | 4 +- python/cudf/cudf/core/reshape.py | 3 +- python/cudf/cudf/core/scalar.py | 6 +- python/cudf/cudf/core/tools/datetimes.py | 3 +- python/cudf/cudf/core/udf/utils.py | 5 +- python/cudf/cudf/core/window/rolling.py | 6 +- python/cudf/cudf/io/avro.py | 4 +- python/cudf/cudf/io/csv.py | 4 +- python/cudf/cudf/io/json.py | 11 +- python/cudf/cudf/io/orc.py | 12 +- python/cudf/cudf/io/parquet.py | 9 +- python/cudf/cudf/io/text.py | 4 +- python/cudf/cudf/tests/test_string_udfs.py | 10 +- python/cudf/cudf/utils/dtypes.py | 29 + 37 files changed, 812 insertions(+), 1374 deletions(-) delete mode 100644 python/cudf/cudf/_lib/column.pxd delete mode 100644 python/cudf/cudf/_lib/column.pyi delete mode 100644 python/cudf/cudf/_lib/column.pyx diff --git a/python/cudf/cudf/_lib/CMakeLists.txt b/python/cudf/cudf/_lib/CMakeLists.txt index 0ec9350e6ee..a21fe7cb85f 100644 --- a/python/cudf/cudf/_lib/CMakeLists.txt +++ b/python/cudf/cudf/_lib/CMakeLists.txt @@ -12,7 +12,7 @@ # the License. # ============================================================================= -set(cython_sources column.pyx strings_udf.pyx) +set(cython_sources strings_udf.pyx) set(linked_libraries cudf::cudf) rapids_cython_create_modules( diff --git a/python/cudf/cudf/_lib/column.pxd b/python/cudf/cudf/_lib/column.pxd deleted file mode 100644 index 58745d91fc0..00000000000 --- a/python/cudf/cudf/_lib/column.pxd +++ /dev/null @@ -1,43 +0,0 @@ -# Copyright (c) 2020-2025, NVIDIA CORPORATION. - -from typing import Literal - -from libcpp cimport bool -from libcpp.memory cimport unique_ptr - -from pylibcudf.libcudf.column.column cimport column -from pylibcudf.libcudf.column.column_view cimport ( - column_view, - mutable_column_view, -) -from pylibcudf.libcudf.types cimport size_type -from rmm.librmm.device_buffer cimport device_buffer - -cdef dtype_from_column_view(column_view cv) - -cdef class Column: - cdef public: - cdef int _offset - cdef int _size - cdef object _dtype - cdef object _base_children - cdef object _base_data - cdef object _base_mask - cdef object _children - cdef object _data - cdef object _mask - cdef object _null_count - cdef object _distinct_count - - cdef column_view _view(self, size_type null_count) except * - cdef column_view view(self) except * - cdef mutable_column_view mutable_view(self) except * - cpdef to_pylibcudf(self, mode: Literal["read", "write"]) - - @staticmethod - cdef Column from_unique_ptr( - unique_ptr[column] c_col, bint data_ptr_exposed=* - ) - - @staticmethod - cdef Column from_column_view(column_view, object) diff --git a/python/cudf/cudf/_lib/column.pyi b/python/cudf/cudf/_lib/column.pyi deleted file mode 100644 index bdd90be45b8..00000000000 --- a/python/cudf/cudf/_lib/column.pyi +++ /dev/null @@ -1,82 +0,0 @@ -# Copyright (c) 2021, NVIDIA CORPORATION. - -from __future__ import annotations - -from typing import Literal - -from typing_extensions import Self - -import pylibcudf as plc - -from cudf._typing import Dtype, DtypeObj, ScalarLike -from cudf.core.buffer import Buffer -from cudf.core.column import ColumnBase - -class Column: - _data: Buffer | None - _mask: Buffer | None - _base_data: Buffer | None - _base_mask: Buffer | None - _dtype: DtypeObj - _size: int - _offset: int - _null_count: int - _children: tuple[ColumnBase, ...] - _base_children: tuple[ColumnBase, ...] - _distinct_count: dict[bool, int] - - def __init__( - self, - data: Buffer | None, - size: int, - dtype: Dtype, - mask: Buffer | None = None, - offset: int | None = None, - null_count: int | None = None, - children: tuple[ColumnBase, ...] = (), - ) -> None: ... - @property - def base_size(self) -> int: ... - @property - def dtype(self) -> DtypeObj: ... - @property - def size(self) -> int: ... - @property - def base_data(self) -> Buffer | None: ... - @property - def data(self) -> Buffer | None: ... - @property - def data_ptr(self) -> int: ... - def set_base_data(self, value: Buffer) -> None: ... - @property - def nullable(self) -> bool: ... - def has_nulls(self, include_nan: bool = False) -> bool: ... - @property - def base_mask(self) -> Buffer | None: ... - @property - def mask(self) -> Buffer | None: ... - @property - def mask_ptr(self) -> int: ... - def set_base_mask(self, value: Buffer | None) -> None: ... - def set_mask(self, value: ColumnBase | Buffer | None) -> Self: ... - @property - def null_count(self) -> int: ... - @property - def offset(self) -> int: ... - @property - def base_children(self) -> tuple[ColumnBase, ...]: ... - @property - def children(self) -> tuple[ColumnBase, ...]: ... - def set_base_children(self, value: tuple[ColumnBase, ...]) -> None: ... - def _mimic_inplace( - self, other_col: ColumnBase, inplace=False - ) -> Self | None: ... - - # TODO: The val parameter should be Scalar, not ScalarLike - @staticmethod - def from_scalar(val: ScalarLike, size: int) -> ColumnBase: ... - @staticmethod - def from_pylibcudf( - col: plc.Column, data_ptr_exposed: bool = False - ) -> ColumnBase: ... - def to_pylibcudf(self, mode: Literal["read", "write"]) -> plc.Column: ... diff --git a/python/cudf/cudf/_lib/column.pyx b/python/cudf/cudf/_lib/column.pyx deleted file mode 100644 index 00ecd53e70d..00000000000 --- a/python/cudf/cudf/_lib/column.pyx +++ /dev/null @@ -1,913 +0,0 @@ -# Copyright (c) 2020-2025, NVIDIA CORPORATION. - - -from typing import Literal - -import cupy as cp -import numpy as np -import pandas as pd - -import pylibcudf -import rmm - -import cudf -from cudf.core.buffer import ( - Buffer, - ExposureTrackedBuffer, - SpillableBuffer, - acquire_spill_lock, - as_buffer, - cuda_array_interface_wrapper, -) -from cudf.utils.dtypes import ( - _get_base_dtype, - dtype_to_pylibcudf_type, - PYLIBCUDF_TO_SUPPORTED_NUMPY_TYPES, -) - -from cpython.buffer cimport PyObject_CheckBuffer -from libc.stdint cimport uintptr_t, int32_t -from libcpp.memory cimport make_shared, make_unique, shared_ptr, unique_ptr -from libcpp.utility cimport move -from libcpp.vector cimport vector - -from rmm.pylibrmm.device_buffer cimport DeviceBuffer - -from pylibcudf cimport ( - DataType as plc_DataType, - Column as plc_Column, - Scalar as plc_Scalar, -) -cimport pylibcudf.libcudf.copying as cpp_copying -cimport pylibcudf.libcudf.types as libcudf_types -cimport pylibcudf.libcudf.unary as libcudf_unary -from pylibcudf.libcudf.column.column cimport column, column_contents -from pylibcudf.libcudf.column.column_factories cimport ( - make_numeric_column -) -from pylibcudf.libcudf.column.column_view cimport column_view -from pylibcudf.libcudf.lists.lists_column_view cimport lists_column_view -from pylibcudf.libcudf.scalar.scalar cimport scalar - - -cdef get_element(column_view col_view, size_type index): - - cdef unique_ptr[scalar] c_output - with nogil: - c_output = move( - cpp_copying.get_element(col_view, index) - ) - plc_scalar = plc_Scalar.from_libcudf(move(c_output)) - return pylibcudf.interop.to_arrow(plc_scalar).as_py() - - -def dtype_from_pylibcudf_column(plc_Column col not None): - type_ = col.type() - tid = type_.id() - - if tid == pylibcudf.TypeId.LIST: - child = col.list_view().child() - return cudf.ListDtype(dtype_from_pylibcudf_column(child)) - elif tid == pylibcudf.TypeId.STRUCT: - fields = { - str(i): dtype_from_pylibcudf_column(col.child(i)) - for i in range(col.num_children()) - } - return cudf.StructDtype(fields) - elif tid == pylibcudf.TypeId.DECIMAL64: - return cudf.Decimal64Dtype( - precision=cudf.Decimal64Dtype.MAX_PRECISION, - scale=-type_.scale() - ) - elif tid == pylibcudf.TypeId.DECIMAL32: - return cudf.Decimal32Dtype( - precision=cudf.Decimal32Dtype.MAX_PRECISION, - scale=-type_.scale() - ) - elif tid == pylibcudf.TypeId.DECIMAL128: - return cudf.Decimal128Dtype( - precision=cudf.Decimal128Dtype.MAX_PRECISION, - scale=-type_.scale() - ) - else: - return PYLIBCUDF_TO_SUPPORTED_NUMPY_TYPES[tid] - - -cdef dtype_from_lists_column_view(column_view cv): - # lists_column_view have no default constructor, so we heap - # allocate it to get around Cython's limitation of requiring - # default constructors for stack allocated objects - cdef shared_ptr[lists_column_view] lv = make_shared[lists_column_view](cv) - cdef column_view child = lv.get()[0].child() - - if child.type().id() == libcudf_types.type_id.LIST: - return cudf.ListDtype(dtype_from_lists_column_view(child)) - else: - return cudf.ListDtype(dtype_from_column_view(child)) - - -cdef dtype_from_column_view(column_view cv): - cdef libcudf_types.type_id tid = cv.type().id() - if tid == libcudf_types.type_id.LIST: - return dtype_from_lists_column_view(cv) - elif tid == libcudf_types.type_id.STRUCT: - fields = { - str(i): dtype_from_column_view(cv.child(i)) - for i in range(cv.num_children()) - } - return cudf.StructDtype(fields) - elif tid == libcudf_types.type_id.DECIMAL64: - return cudf.Decimal64Dtype( - precision=cudf.Decimal64Dtype.MAX_PRECISION, - scale=-cv.type().scale() - ) - elif tid == libcudf_types.type_id.DECIMAL32: - return cudf.Decimal32Dtype( - precision=cudf.Decimal32Dtype.MAX_PRECISION, - scale=-cv.type().scale() - ) - elif tid == libcudf_types.type_id.DECIMAL128: - return cudf.Decimal128Dtype( - precision=cudf.Decimal128Dtype.MAX_PRECISION, - scale=-cv.type().scale() - ) - else: - return PYLIBCUDF_TO_SUPPORTED_NUMPY_TYPES[(tid)] - - -cdef class Column: - """ - A Column stores columnar data in device memory. - A Column may be composed of: - - * A *data* Buffer - * One or more (optional) *children* Columns - * An (optional) *mask* Buffer representing the nullmask - - The *dtype* indicates the Column's element type. - """ - def __init__( - self, - object data, - int size, - object dtype, - object mask=None, - int offset=0, - object null_count=None, - tuple children=() - ): - if size < 0: - raise ValueError("size must be >=0") - self._size = size - self._distinct_count = {} - self._dtype = dtype - self._offset = offset - self._null_count = null_count - self.set_base_children(children) - self.set_base_data(data) - self.set_base_mask(mask) - - @property - def base_size(self): - return int(self.base_data.size / self.dtype.itemsize) - - @property - def dtype(self): - return self._dtype - - @property - def size(self): - return self._size - - @property - def base_data(self): - return self._base_data - - @property - def data(self): - if self.base_data is None: - return None - if self._data is None: - start = self.offset * self.dtype.itemsize - end = start + self.size * self.dtype.itemsize - self._data = self.base_data[start:end] - return self._data - - @property - def data_ptr(self): - if self.data is None: - return 0 - else: - return self.data.get_ptr(mode="write") - - def set_base_data(self, value): - if value is not None and not isinstance(value, Buffer): - raise TypeError( - "Expected a Buffer or None for data, " - f"got {type(value).__name__}" - ) - - self._data = None - self._base_data = value - - @property - def nullable(self): - return self.base_mask is not None - - def has_nulls(self, include_nan=False): - return int(self.null_count) != 0 - - @property - def base_mask(self): - return self._base_mask - - @property - def mask(self): - if self._mask is None: - if self.base_mask is None or self.offset == 0: - self._mask = self.base_mask - else: - with acquire_spill_lock(): - self._mask = as_buffer( - pylibcudf.null_mask.copy_bitmask(self.to_pylibcudf(mode="read")) - ) - return self._mask - - @property - def mask_ptr(self): - if self.mask is None: - return 0 - else: - return self.mask.get_ptr(mode="write") - - def set_base_mask(self, value): - """ - Replaces the base mask buffer of the column inplace. This does not - modify size or offset in any way, so the passed mask is expected to be - compatible with the current offset. - """ - if value is not None and not isinstance(value, Buffer): - raise TypeError( - "Expected a Buffer or None for mask, " - f"got {type(value).__name__}" - ) - - if value is not None: - # bitmask size must be relative to offset = 0 data. - required_size = pylibcudf.null_mask.bitmask_allocation_size_bytes( - self.base_size - ) - if value.size < required_size: - error_msg = ( - "The Buffer for mask is smaller than expected, " - f"got {value.size} bytes, expected {required_size} bytes." - ) - if self.offset > 0 or self.size < self.base_size: - error_msg += ( - "\n\nNote: The mask is expected to be sized according " - "to the base allocation as opposed to the offsetted or" - " sized allocation." - ) - raise ValueError(error_msg) - - self._mask = None - self._children = None - self._base_mask = value - self._clear_cache() - - def _clear_cache(self): - self._distinct_count = {} - attrs = ("memory_usage", "is_monotonic_increasing", "is_monotonic_decreasing") - for attr in attrs: - try: - delattr(self, attr) - except AttributeError: - # attr was not called yet, so ignore. - pass - self._null_count = None - - def set_mask(self, value): - """ - Replaces the mask buffer of the column and returns a new column. This - will zero the column offset, compute a new mask buffer if necessary, - and compute new data Buffers zero-copy that use pointer arithmetic to - properly adjust the pointer. - """ - mask_size = pylibcudf.null_mask.bitmask_allocation_size_bytes(self.size) - required_num_bytes = -(-self.size // 8) # ceiling divide - error_msg = ( - "The value for mask is smaller than expected, got {} bytes, " - "expected " + str(required_num_bytes) + " bytes." - ) - if value is None: - mask = None - elif hasattr(value, "__cuda_array_interface__"): - if value.__cuda_array_interface__["typestr"] not in ("|i1", "|u1"): - if isinstance(value, Column): - value = value.data_array_view(mode="write") - value = cp.asarray(value).view('|u1') - mask = as_buffer(value) - if mask.size < required_num_bytes: - raise ValueError(error_msg.format(str(value.size))) - if mask.size < mask_size: - dbuf = rmm.DeviceBuffer(size=mask_size) - dbuf.copy_from_device(value) - mask = as_buffer(dbuf) - elif hasattr(value, "__array_interface__"): - value = np.asarray(value).view("u1")[:mask_size] - if value.size < required_num_bytes: - raise ValueError(error_msg.format(str(value.size))) - dbuf = rmm.DeviceBuffer(size=mask_size) - dbuf.copy_from_host(value) - mask = as_buffer(dbuf) - elif PyObject_CheckBuffer(value): - value = np.asarray(value).view("u1")[:mask_size] - if value.size < required_num_bytes: - raise ValueError(error_msg.format(str(value.size))) - dbuf = rmm.DeviceBuffer(size=mask_size) - dbuf.copy_from_host(value) - mask = as_buffer(dbuf) - else: - raise TypeError( - "Expected a Buffer object or None for mask, " - f"got {type(value).__name__}" - ) - - return cudf.core.column.build_column( - data=self.data, - dtype=self.dtype, - mask=mask, - size=self.size, - offset=0, - children=self.children - ) - - @property - def null_count(self): - if self._null_count is None: - if not self.nullable or self.size == 0: - self._null_count = 0 - else: - with acquire_spill_lock(): - self._null_count = pylibcudf.null_mask.null_count( - self.base_mask.get_ptr(mode="read"), - self.offset, - self.offset + self.size - ) - return self._null_count - - @property - def offset(self): - return self._offset - - @property - def base_children(self): - return self._base_children - - @property - def children(self): - if (self.offset == 0) and (self.size == self.base_size): - self._children = self.base_children - if self._children is None: - if self.base_children == (): - self._children = () - else: - children = Column.from_unique_ptr( - move(make_unique[column](self.view())) - ).base_children - dtypes = [ - base_child.dtype for base_child in self.base_children - ] - self._children = tuple( - child._with_type_metadata(dtype) for child, dtype in zip( - children, dtypes - ) - ) - return self._children - - def set_base_children(self, value): - if not isinstance(value, tuple): - raise TypeError("Expected a tuple of Columns for children, got " + - type(value).__name__) - - for child in value: - if not isinstance(child, Column): - raise TypeError( - "Expected each of children to be a Column, got " + - type(child).__name__ - ) - - self._children = None - self._base_children = value - - def _mimic_inplace(self, other_col, inplace=False): - """ - Given another column, update the attributes of this column to mimic an - inplace operation. This does not modify the memory of Buffers, but - instead replaces the Buffers and other attributes underneath the column - object with the Buffers and attributes from the other column. - """ - if inplace: - self._offset = other_col.offset - self._size = other_col.size - self._dtype = other_col._dtype - self.set_base_data(other_col.base_data) - self.set_base_children(other_col.base_children) - self.set_base_mask(other_col.base_mask) - else: - return other_col - - cdef mutable_column_view mutable_view(self) except *: - if isinstance(self.dtype, cudf.CategoricalDtype): - col = self.base_children[0] - data_dtype = col.dtype - elif isinstance(self.dtype, pd.DatetimeTZDtype): - col = self - data_dtype = _get_base_dtype(col.dtype) - else: - col = self - data_dtype = col.dtype - - cdef plc_DataType dtype = dtype_to_pylibcudf_type(data_dtype) - cdef libcudf_types.size_type offset = self.offset - cdef vector[mutable_column_view] children - cdef void* data - - if col.base_data is None: - data = NULL - else: - data = ( - col.base_data.get_ptr(mode="write") - ) - - cdef Column child_column - if col.base_children: - for child_column in col.base_children: - children.push_back(child_column.mutable_view()) - - cdef libcudf_types.bitmask_type* mask - if self.nullable: - mask = ( - self.base_mask.get_ptr(mode="write") - ) - else: - mask = NULL - - null_count = self._null_count - - if null_count is None: - null_count = 0 - cdef libcudf_types.size_type c_null_count = null_count - - self._mask = None - self._null_count = None - self._children = None - self._data = None - - return mutable_column_view( - dtype.c_obj, - self.size, - data, - mask, - c_null_count, - offset, - children) - - cdef column_view view(self) except *: - null_count = self.null_count - if null_count is None: - null_count = 0 - cdef libcudf_types.size_type c_null_count = null_count - return self._view(c_null_count) - - cdef column_view _view(self, libcudf_types.size_type null_count) except *: - if isinstance(self.dtype, cudf.CategoricalDtype): - col = self.base_children[0] - data_dtype = col.dtype - elif isinstance(self.dtype, pd.DatetimeTZDtype): - col = self - data_dtype = _get_base_dtype(col.dtype) - else: - col = self - data_dtype = col.dtype - - cdef plc_DataType dtype = dtype_to_pylibcudf_type(data_dtype) - cdef libcudf_types.size_type offset = self.offset - cdef vector[column_view] children - cdef void* data - - if col.base_data is None: - data = NULL - else: - data = (col.base_data.get_ptr(mode="read")) - - cdef Column child_column - if col.base_children: - for child_column in col.base_children: - children.push_back(child_column.view()) - - cdef libcudf_types.bitmask_type* mask - if self.nullable: - mask = ( - self.base_mask.get_ptr(mode="read") - ) - else: - mask = NULL - - cdef libcudf_types.size_type c_null_count = null_count - - return column_view( - dtype.c_obj, - self.size, - data, - mask, - c_null_count, - offset, - children) - - # TODO: Consider whether this function should support some sort of `copy` - # parameter. Not urgent until this functionality is moved up to the Frame - # layer and made public. This function will also need to mark the - # underlying buffers as exposed before this function can itself be exposed - # publicly. User requests to convert to pylibcudf must assume that the - # data may be modified afterwards. - cpdef to_pylibcudf(self, mode: Literal["read", "write"]): - """Convert this Column to a pylibcudf.Column. - - This function will generate a pylibcudf Column pointing to the same - data, mask, and children as this one. - - Parameters - ---------- - mode : str - Supported values are {"read", "write"} If "write", the data pointed - to may be modified by the caller. If "read", the data pointed to - must not be modified by the caller. Failure to fulfill this - contract will cause incorrect behavior. - - Returns - ------- - pylibcudf.Column - A new pylibcudf.Column referencing the same data. - """ - - # TODO: Categoricals will need to be treated differently eventually. - # There is no 1-1 correspondence between cudf and libcudf for - # categoricals because cudf supports ordered and unordered categoricals - # while libcudf supports only unordered categoricals (see - # https://github.com/rapidsai/cudf/pull/8567). - if isinstance(self.dtype, cudf.CategoricalDtype): - col = self.base_children[0] - else: - col = self - - dtype = dtype_to_pylibcudf_type(col.dtype) - - data = None - if col.base_data is not None: - cai = cuda_array_interface_wrapper( - ptr=col.base_data.get_ptr(mode=mode), - size=col.base_data.size, - owner=col.base_data, - ) - data = pylibcudf.gpumemoryview(cai) - - mask = None - if self.nullable: - # TODO: Are we intentionally use self's mask instead of col's? - # Where is the mask stored for categoricals? - cai = cuda_array_interface_wrapper( - ptr=self.base_mask.get_ptr(mode=mode), - size=self.base_mask.size, - owner=self.base_mask, - ) - mask = pylibcudf.gpumemoryview(cai) - - cdef Column child_column - children = [] - if col.base_children: - for child_column in col.base_children: - children.append(child_column.to_pylibcudf(mode=mode)) - - return pylibcudf.Column( - dtype, - self.size, - data, - mask, - self.null_count, - self.offset, - children, - ) - - @staticmethod - cdef Column from_unique_ptr( - unique_ptr[column] c_col, bint data_ptr_exposed=False - ): - """Create a Column from a column - - Typically, this is called on the result of a libcudf operation. - If the data of the libcudf result has been exposed, set - `data_ptr_exposed=True` to expose the memory of the returned Column - as well. - """ - cdef column_view view = c_col.get()[0].view() - cdef libcudf_types.type_id tid = view.type().id() - cdef libcudf_types.data_type c_dtype - cdef size_type length = view.size() - cdef libcudf_types.mask_state mask_state - if tid == libcudf_types.type_id.TIMESTAMP_DAYS: - c_dtype = libcudf_types.data_type( - libcudf_types.type_id.TIMESTAMP_SECONDS - ) - with nogil: - c_col = move(libcudf_unary.cast(view, c_dtype)) - elif tid == libcudf_types.type_id.EMPTY: - c_dtype = libcudf_types.data_type(libcudf_types.type_id.INT8) - mask_state = libcudf_types.mask_state.ALL_NULL - with nogil: - c_col = move(make_numeric_column(c_dtype, length, mask_state)) - - size = c_col.get()[0].size() - dtype = dtype_from_column_view(c_col.get()[0].view()) - null_count = c_col.get()[0].null_count() - - # After call to release(), c_col is unusable - cdef column_contents contents = move(c_col.get()[0].release()) - - data = as_buffer( - DeviceBuffer.c_from_unique_ptr(move(contents.data)), - exposed=data_ptr_exposed - ) - - if null_count > 0: - mask = as_buffer( - DeviceBuffer.c_from_unique_ptr(move(contents.null_mask)), - exposed=data_ptr_exposed - ) - else: - mask = None - - cdef vector[unique_ptr[column]] c_children = move(contents.children) - children = [] - if c_children.size() != 0: - # Because of a bug in Cython, we cannot set the optional - # `data_ptr_exposed` argument within a comprehension. - for i in range(c_children.size()): - child = Column.from_unique_ptr( - move(c_children[i]), - data_ptr_exposed=data_ptr_exposed - ) - children.append(child) - - return cudf.core.column.build_column( - data, - dtype=dtype, - mask=mask, - size=size, - null_count=null_count, - children=tuple(children) - ) - - @staticmethod - def from_pylibcudf( - col, bint data_ptr_exposed=False - ): - """Create a Column from a pylibcudf.Column. - - This function will generate a Column pointing to the provided pylibcudf - Column. It will directly access the data and mask buffers of the - pylibcudf Column, so the newly created object is not tied to the - lifetime of the original pylibcudf.Column. - - Parameters - ---------- - col : pylibcudf.Column - The object to copy. - data_ptr_exposed : bool - Whether the data buffer is exposed. - - Returns - ------- - pylibcudf.Column - A new pylibcudf.Column referencing the same data. - """ - if col.type().id() == pylibcudf.TypeId.TIMESTAMP_DAYS: - col = pylibcudf.unary.cast( - col, pylibcudf.DataType(pylibcudf.TypeId.TIMESTAMP_SECONDS) - ) - elif col.type().id() == pylibcudf.TypeId.EMPTY: - new_dtype = pylibcudf.DataType(pylibcudf.TypeId.INT8) - - col = pylibcudf.column_factories.make_numeric_column( - new_dtype, - col.size(), - pylibcudf.column_factories.MaskState.ALL_NULL - ) - - dtype = dtype_from_pylibcudf_column(col) - - return cudf.core.column.build_column( - data=as_buffer( - col.data().obj, exposed=data_ptr_exposed - ) if col.data() is not None else None, - dtype=dtype, - size=col.size(), - mask=as_buffer( - col.null_mask().obj, exposed=data_ptr_exposed - ) if col.null_mask() is not None else None, - offset=col.offset(), - null_count=col.null_count(), - children=tuple([ - Column.from_pylibcudf(child, data_ptr_exposed=data_ptr_exposed) - for child in col.children() - ]) - ) - - @staticmethod - cdef Column from_column_view(column_view cv, object owner): - """ - Given a ``cudf::column_view``, constructs a ``cudf.Column`` from it, - along with referencing an ``owner`` Python object that owns the memory - lifetime. If ``owner`` is a ``cudf.Column``, we reach inside of it and - make the owner of each newly created ``Buffer`` the respective - ``Buffer`` from the ``owner`` ``cudf.Column``. - If ``owner`` is ``None``, we allocate new memory for the resulting - ``cudf.Column``. - """ - column_owner = isinstance(owner, Column) - mask_owner = owner - if column_owner and isinstance(owner.dtype, cudf.CategoricalDtype): - owner = owner.base_children[0] - - size = cv.size() - offset = cv.offset() - dtype = dtype_from_column_view(cv) - dtype_itemsize = getattr(dtype, "itemsize", 1) - - data_ptr = (cv.head[void]()) - data = None - base_size = size + offset - data_owner = owner - - if column_owner: - data_owner = owner.base_data - mask_owner = mask_owner.base_mask - base_size = owner.base_size - base_nbytes = base_size * dtype_itemsize - # special case for string column - is_string_column = (cv.type().id() == libcudf_types.type_id.STRING) - if is_string_column: - if cv.num_children() == 0: - base_nbytes = 0 - else: - # get the size from offset child column (device to host copy) - offsets_column_index = 0 - offset_child_column = cv.child(offsets_column_index) - if offset_child_column.size() == 0: - base_nbytes = 0 - else: - chars_size = get_element( - offset_child_column, offset_child_column.size()-1) - base_nbytes = chars_size - - if data_ptr: - if data_owner is None: - buffer_size = ( - base_nbytes - if is_string_column - else ((size + offset) * dtype_itemsize) - ) - data = as_buffer( - rmm.DeviceBuffer(ptr=data_ptr, - size=buffer_size) - ) - elif ( - column_owner and - isinstance(data_owner, ExposureTrackedBuffer) - ): - data = as_buffer( - data=data_ptr, - size=base_nbytes, - owner=data_owner, - exposed=False, - ) - elif ( - # This is an optimization of the most common case where - # from_column_view creates a "view" that is identical to - # the owner. - column_owner and - isinstance(data_owner, SpillableBuffer) and - # We check that `data_owner` is spill locked (not spillable) - # and that it points to the same memory as `data_ptr`. - not data_owner.spillable and - data_owner.memory_info() == (data_ptr, base_nbytes, "gpu") - ): - data = data_owner - else: - # At this point we don't know the relationship between data_ptr - # and data_owner thus we mark both of them exposed. - # TODO: try to discover their relationship and create a - # SpillableBufferSlice instead. - data = as_buffer( - data=data_ptr, - size=base_nbytes, - owner=data_owner, - exposed=True, - ) - if isinstance(data_owner, ExposureTrackedBuffer): - # accessing the pointer marks it exposed permanently. - data_owner.mark_exposed() - elif isinstance(data_owner, SpillableBuffer): - if data_owner.is_spilled: - raise ValueError( - f"{data_owner} is spilled, which invalidates " - f"the exposed data_ptr ({hex(data_ptr)})" - ) - # accessing the pointer marks it exposed permanently. - data_owner.mark_exposed() - else: - data = as_buffer( - rmm.DeviceBuffer(ptr=data_ptr, size=0) - ) - - mask = None - mask_ptr = (cv.null_mask()) - if mask_ptr: - if mask_owner is None: - if column_owner: - # if we reached here, it means `owner` is a `Column` - # that does not have a null mask, but `cv` thinks it - # should have a null mask. This can happen in the - # following sequence of events: - # - # 1) `cv` is constructed as a view into a - # `cudf::column` that is nullable (i.e., it has - # a null mask), but contains no nulls. - # 2) `owner`, a `Column`, is constructed from the - # same `cudf::column`. Because `cudf::column` - # is memory owning, `owner` takes ownership of - # the memory owned by the - # `cudf::column`. Because the column has a null - # count of 0, it may choose to discard the null - # mask. - # 3) Now, `cv` points to a discarded null mask. - # - # TL;DR: we should not include a null mask in the - # result: - mask = None - else: - mask = as_buffer( - rmm.DeviceBuffer( - ptr=mask_ptr, - size=pylibcudf.null_mask.bitmask_allocation_size_bytes( - base_size - ) - ) - ) - else: - mask = as_buffer( - data=mask_ptr, - size=pylibcudf.null_mask.bitmask_allocation_size_bytes( - base_size - ), - owner=mask_owner, - exposed=True - ) - - if cv.has_nulls(): - null_count = cv.null_count() - else: - null_count = 0 - - children = [] - for child_index in range(cv.num_children()): - child_owner = owner - if column_owner: - child_owner = owner.base_children[child_index] - children.append( - Column.from_column_view( - cv.child(child_index), - child_owner - ) - ) - children = tuple(children) - - result = cudf.core.column.build_column( - data=data, - dtype=dtype, - mask=mask, - size=size, - offset=offset, - null_count=null_count, - children=tuple(children) - ) - - return result - - @staticmethod - def from_scalar(py_val, size_type size): - return Column.from_pylibcudf( - pylibcudf.Column.from_scalar( - py_val.device_value, size - ) - ) diff --git a/python/cudf/cudf/core/_base_index.py b/python/cudf/cudf/core/_base_index.py index 05fb6f531a0..142a9b4dac5 100644 --- a/python/cudf/cudf/core/_base_index.py +++ b/python/cudf/cudf/core/_base_index.py @@ -1941,11 +1941,14 @@ def drop_duplicates( # This utilizes the fact that all `Index` is also a `Frame`. # Except RangeIndex. return self._from_columns_like_self( - stream_compaction.drop_duplicates( - list(self._columns), - keep=keep, - nulls_are_equal=nulls_are_equal, - ), + [ + ColumnBase.from_pylibcudf(col) + for col in stream_compaction.drop_duplicates( + list(self._columns), + keep=keep, + nulls_are_equal=nulls_are_equal, + ) + ], self._column_names, ) @@ -2028,10 +2031,13 @@ def dropna(self, how="any"): data_columns = [col.nans_to_nulls() for col in self._columns] return self._from_columns_like_self( - stream_compaction.drop_nulls( - data_columns, - how=how, - ), + [ + ColumnBase.from_pylibcudf(col) + for col in stream_compaction.drop_nulls( + data_columns, + how=how, + ) + ], self._column_names, ) @@ -2050,7 +2056,12 @@ def _gather(self, gather_map, nullify=False, check_bounds=True): GatherMap(gather_map, len(self), nullify=not check_bounds or nullify) return self._from_columns_like_self( - copying.gather(self._columns, gather_map, nullify=nullify), + [ + ColumnBase.from_pylibcudf(col) + for col in copying.gather( + self._columns, gather_map, nullify=nullify + ) + ], self._column_names, ) @@ -2099,9 +2110,12 @@ def _apply_boolean_mask(self, boolean_mask): raise ValueError("boolean_mask is not boolean type.") return self._from_columns_like_self( - stream_compaction.apply_boolean_mask( - list(self._columns), boolean_mask - ), + [ + ColumnBase.from_pylibcudf(col) + for col in stream_compaction.apply_boolean_mask( + list(self._columns), boolean_mask + ) + ], column_names=self._column_names, ) diff --git a/python/cudf/cudf/core/_internals/binaryop.py b/python/cudf/cudf/core/_internals/binaryop.py index 4ad873b9825..3c11e065d21 100644 --- a/python/cudf/cudf/core/_internals/binaryop.py +++ b/python/cudf/cudf/core/_internals/binaryop.py @@ -5,13 +5,12 @@ import pylibcudf as plc -from cudf._lib.column import Column from cudf.core.buffer import acquire_spill_lock +from cudf.core.column import ColumnBase from cudf.utils.dtypes import dtype_to_pylibcudf_type if TYPE_CHECKING: from cudf._typing import Dtype - from cudf.core.column import ColumnBase from cudf.core.scalar import Scalar @@ -46,13 +45,13 @@ def binaryop( op = op.upper() op = _op_map.get(op, op) - return Column.from_pylibcudf( + return ColumnBase.from_pylibcudf( plc.binaryop.binary_operation( lhs.to_pylibcudf(mode="read") - if isinstance(lhs, Column) + if isinstance(lhs, ColumnBase) else lhs.device_value, rhs.to_pylibcudf(mode="read") - if isinstance(rhs, Column) + if isinstance(rhs, ColumnBase) else rhs.device_value, plc.binaryop.BinaryOperator[op], dtype_to_pylibcudf_type(dtype), diff --git a/python/cudf/cudf/core/_internals/copying.py b/python/cudf/cudf/core/_internals/copying.py index 9e63ec63828..6ff26f23774 100644 --- a/python/cudf/cudf/core/_internals/copying.py +++ b/python/cudf/cudf/core/_internals/copying.py @@ -5,7 +5,6 @@ import pylibcudf as plc -import cudf from cudf.core.buffer import acquire_spill_lock if TYPE_CHECKING: @@ -20,7 +19,7 @@ def gather( columns: Iterable[ColumnBase], gather_map: NumericalColumn, nullify: bool = False, -) -> list[ColumnBase]: +) -> list[plc.Column]: plc_tbl = plc.copying.gather( plc.Table([col.to_pylibcudf(mode="read") for col in columns]), gather_map.to_pylibcudf(mode="read"), @@ -28,10 +27,7 @@ def gather( if nullify else plc.copying.OutOfBoundsPolicy.DONT_CHECK, ) - return [ - cudf._lib.column.Column.from_pylibcudf(col) - for col in plc_tbl.columns() - ] + return plc_tbl.columns() @acquire_spill_lock() @@ -64,29 +60,25 @@ def scatter( f"index out of bounds for column of size {n_rows}" ) + from cudf.core.column import ColumnBase + plc_tbl = plc.copying.scatter( plc.Table([col.to_pylibcudf(mode="read") for col in sources]) # type: ignore[union-attr] - if isinstance(sources[0], cudf._lib.column.Column) + if isinstance(sources[0], ColumnBase) else sources, # type: ignore[union-attr] scatter_map.to_pylibcudf(mode="read"), plc.Table([col.to_pylibcudf(mode="read") for col in target_columns]), ) - return [ - cudf._lib.column.Column.from_pylibcudf(col) - for col in plc_tbl.columns() - ] + return plc_tbl.columns() @acquire_spill_lock() def columns_split( input_columns: Iterable[ColumnBase], splits: list[int] -) -> list[list[ColumnBase]]: +) -> list[list[plc.Column]]: return [ - [ - cudf._lib.column.Column.from_pylibcudf(col) - for col in plc_tbl.columns() - ] + plc_tbl.columns() for plc_tbl in plc.copying.split( plc.Table( [col.to_pylibcudf(mode="read") for col in input_columns] diff --git a/python/cudf/cudf/core/_internals/search.py b/python/cudf/cudf/core/_internals/search.py index a0ffe078de9..bee198800e7 100644 --- a/python/cudf/cudf/core/_internals/search.py +++ b/python/cudf/cudf/core/_internals/search.py @@ -1,11 +1,10 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from __future__ import annotations from typing import TYPE_CHECKING, Literal import pylibcudf as plc -from cudf._lib.column import Column from cudf.core.buffer import acquire_spill_lock if TYPE_CHECKING: @@ -19,7 +18,7 @@ def search_sorted( side: Literal["left", "right"], ascending: bool = True, na_position: Literal["first", "last"] = "last", -) -> ColumnBase: +) -> plc.Column: """Find indices where elements should be inserted to maintain order Parameters @@ -46,11 +45,9 @@ def search_sorted( plc.search, "lower_bound" if side == "left" else "upper_bound", ) - return Column.from_pylibcudf( - func( - plc.Table([col.to_pylibcudf(mode="read") for col in source]), - plc.Table([col.to_pylibcudf(mode="read") for col in values]), - column_order, - null_precedence, - ) + return func( + plc.Table([col.to_pylibcudf(mode="read") for col in source]), + plc.Table([col.to_pylibcudf(mode="read") for col in values]), + column_order, + null_precedence, ) diff --git a/python/cudf/cudf/core/_internals/sorting.py b/python/cudf/cudf/core/_internals/sorting.py index 69f9e7664b1..5e6f23f1368 100644 --- a/python/cudf/cudf/core/_internals/sorting.py +++ b/python/cudf/cudf/core/_internals/sorting.py @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from __future__ import annotations import itertools @@ -6,7 +6,6 @@ import pylibcudf as plc -from cudf._lib.column import Column from cudf.core.buffer import acquire_spill_lock if TYPE_CHECKING: @@ -120,7 +119,7 @@ def order_by( na_position: Literal["first", "last"], *, stable: bool, -): +) -> plc.Column: """ Get index to sort the table in ascending/descending order. @@ -146,14 +145,12 @@ def order_by( func = ( plc.sorting.stable_sorted_order if stable else plc.sorting.sorted_order ) - return Column.from_pylibcudf( - func( - plc.Table( - [col.to_pylibcudf(mode="read") for col in columns_from_table], - ), - order[0], - order[1], - ) + return func( + plc.Table( + [col.to_pylibcudf(mode="read") for col in columns_from_table], + ), + order[0], + order[1], ) @@ -165,7 +162,7 @@ def sort_by_key( na_position: list[Literal["first", "last"]], *, stable: bool, -) -> list[ColumnBase]: +) -> list[plc.Column]: """ Sort a table by given keys @@ -194,12 +191,9 @@ def sort_by_key( func = ( plc.sorting.stable_sort_by_key if stable else plc.sorting.sort_by_key ) - return [ - Column.from_pylibcudf(col) - for col in func( - plc.Table([col.to_pylibcudf(mode="read") for col in values]), - plc.Table([col.to_pylibcudf(mode="read") for col in keys]), - order[0], - order[1], - ).columns() - ] + return func( + plc.Table([col.to_pylibcudf(mode="read") for col in values]), + plc.Table([col.to_pylibcudf(mode="read") for col in keys]), + order[0], + order[1], + ).columns() diff --git a/python/cudf/cudf/core/_internals/stream_compaction.py b/python/cudf/cudf/core/_internals/stream_compaction.py index 4ccc26c2a1c..57a655688c4 100644 --- a/python/cudf/cudf/core/_internals/stream_compaction.py +++ b/python/cudf/cudf/core/_internals/stream_compaction.py @@ -1,11 +1,10 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from __future__ import annotations from typing import TYPE_CHECKING, Literal import pylibcudf as plc -from cudf._lib.column import Column from cudf.core.buffer import acquire_spill_lock if TYPE_CHECKING: @@ -18,7 +17,7 @@ def drop_nulls( how: Literal["any", "all"] = "any", keys: list[int] | None = None, thresh: int | None = None, -) -> list[ColumnBase]: +) -> list[plc.Column]: """ Drops null rows from cols depending on key columns. @@ -53,13 +52,13 @@ def drop_nulls( keys, keep_threshold, ) - return [Column.from_pylibcudf(col) for col in plc_table.columns()] + return plc_table.columns() @acquire_spill_lock() def apply_boolean_mask( columns: list[ColumnBase], boolean_mask: ColumnBase -) -> list[ColumnBase]: +) -> list[plc.Column]: """ Drops the rows which correspond to False in boolean_mask. @@ -76,7 +75,7 @@ def apply_boolean_mask( plc.Table([col.to_pylibcudf(mode="read") for col in columns]), boolean_mask.to_pylibcudf(mode="read"), ) - return [Column.from_pylibcudf(col) for col in plc_table.columns()] + return plc_table.columns() @acquire_spill_lock() @@ -85,7 +84,7 @@ def drop_duplicates( keys: list[int] | None = None, keep: Literal["first", "last", False] = "first", nulls_are_equal: bool = True, -) -> list[ColumnBase]: +) -> list[plc.Column]: """ Drops rows in source_table as per duplicate rows in keys. @@ -118,4 +117,4 @@ def drop_duplicates( else plc.types.NullEquality.UNEQUAL, plc.types.NanEquality.ALL_EQUAL, ) - return [Column.from_pylibcudf(col) for col in plc_table.columns()] + return plc_table.columns() diff --git a/python/cudf/cudf/core/_internals/timezones.py b/python/cudf/cudf/core/_internals/timezones.py index d5de78bc2bb..80129e7d71b 100644 --- a/python/cudf/cudf/core/_internals/timezones.py +++ b/python/cudf/cudf/core/_internals/timezones.py @@ -13,7 +13,6 @@ import pylibcudf as plc import cudf -from cudf._lib.column import Column if TYPE_CHECKING: from cudf.core.column.datetime import DatetimeColumn @@ -116,9 +115,7 @@ def _read_tzfile_as_columns( plc_table = plc.io.timezone.make_timezone_transition_table( tzdir, zone_name ) - transition_times_and_offsets = [ - Column.from_pylibcudf(col) for col in plc_table.columns() - ] + transition_times_and_offsets = plc_table.columns() if not transition_times_and_offsets: from cudf.core.column.column import as_column @@ -128,7 +125,12 @@ def _read_tzfile_as_columns( np.dtype("M8[s]") ) return (as_column([min_date]), as_column([np.timedelta64(0, "s")])) # type: ignore[return-value] - return tuple(transition_times_and_offsets) # type: ignore[return-value] + + from cudf.core.column import ColumnBase + + return tuple( + ColumnBase.from_pylibcudf(col) for col in transition_times_and_offsets + ) # type: ignore[return-value] def check_ambiguous_and_nonexistent( diff --git a/python/cudf/cudf/core/column/categorical.py b/python/cudf/cudf/core/column/categorical.py index 985b689f087..a789d5d5ab1 100644 --- a/python/cudf/cudf/core/column/categorical.py +++ b/python/cudf/cudf/core/column/categorical.py @@ -506,7 +506,7 @@ class CategoricalColumn(column.ColumnBase): """ dtype: CategoricalDtype - _children: tuple[NumericalColumn] + _children: tuple[NumericalColumn] # type: ignore[assignment] _VALID_REDUCTIONS = { "max", "min", @@ -1169,7 +1169,7 @@ def memory_usage(self) -> int: def _mimic_inplace( self, other_col: ColumnBase, inplace: bool = False ) -> Self | None: - out = super()._mimic_inplace(other_col, inplace=inplace) + out = super()._mimic_inplace(other_col, inplace=inplace) # type: ignore[arg-type] if inplace and isinstance(other_col, CategoricalColumn): self._codes = other_col.codes return out diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index 94e75e9d07a..d281076690a 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -23,7 +23,6 @@ import rmm import cudf -from cudf._lib.column import Column from cudf.api.types import ( _is_non_decimal_numeric_dtype, _is_pandas_nullable_extension_dtype, @@ -66,6 +65,7 @@ _maybe_convert_to_default_type, cudf_dtype_from_pa_type, cudf_dtype_to_pa_type, + dtype_from_pylibcudf_column, dtype_to_pylibcudf_type, find_common_type, get_time_unit, @@ -89,7 +89,19 @@ NumpyExtensionArray = pd.arrays.PandasArray -class ColumnBase(Column, Serializable, BinaryOperand, Reducible): +class ColumnBase(Serializable, BinaryOperand, Reducible): + """ + A ColumnBase stores columnar data in device memory. + + A ColumnBase may be composed of: + + * A *data* Buffer + * One or more (optional) *children* Columns + * An (optional) *mask* Buffer representing the nullmask + + The *dtype* indicates the ColumnBase's element type. + """ + _VALID_REDUCTIONS = { "any", "all", @@ -99,6 +111,423 @@ class ColumnBase(Column, Serializable, BinaryOperand, Reducible): _PANDAS_NA_REPR = str(pd.NA) + def __init__( + self, + data: None | Buffer, + size: int, + dtype, + mask: None | Buffer = None, + offset: int = 0, + null_count: int | None = None, + children: tuple[ColumnBase, ...] = (), + ) -> None: + if size < 0: + raise ValueError("size must be >=0") + self._size = size + self._distinct_count: dict[bool, int] = {} + self._dtype = dtype + self._offset = offset + self._null_count = null_count + self._mask = None + self._base_mask = None + self._data = None + self._children = None + self.set_base_children(children) + self.set_base_data(data) + self.set_base_mask(mask) + + @property + def base_size(self) -> int: + return int(self.base_data.size / self.dtype.itemsize) # type: ignore[union-attr] + + @property + def dtype(self): + return self._dtype + + @property + def size(self) -> int: + return self._size + + @property + def base_data(self) -> None | Buffer: + return self._base_data # type: ignore[has-type] + + @property + def data(self) -> None | Buffer: + if self.base_data is None: + return None + if self._data is None: # type: ignore[has-type] + start = self.offset * self.dtype.itemsize + end = start + self.size * self.dtype.itemsize + self._data = self.base_data[start:end] # type: ignore[assignment] + return self._data + + @property + def data_ptr(self) -> int: + if self.data is None: + return 0 + else: + return self.data.get_ptr(mode="write") + + def set_base_data(self, value: None | Buffer) -> None: + if value is not None and not isinstance(value, Buffer): + raise TypeError( + "Expected a Buffer or None for data, " + f"got {type(value).__name__}" + ) + + self._data = None # type: ignore[assignment] + self._base_data = value + + @property + def nullable(self) -> bool: + return self.base_mask is not None + + def has_nulls(self, include_nan: bool = False) -> bool: + return int(self.null_count) != 0 + + @property + def base_mask(self) -> None | Buffer: + return self._base_mask # type: ignore[has-type] + + @property + def mask(self) -> None | Buffer: + if self._mask is None: # type: ignore[has-type] + if self.base_mask is None or self.offset == 0: + self._mask = self.base_mask # type: ignore[assignment] + else: + with acquire_spill_lock(): + self._mask = as_buffer( # type: ignore[assignment] + plc.null_mask.copy_bitmask( + self.to_pylibcudf(mode="read") + ) + ) + return self._mask + + @property + def mask_ptr(self) -> int: + if self.mask is None: + return 0 + else: + return self.mask.get_ptr(mode="write") + + def set_base_mask(self, value: None | Buffer) -> None: + """ + Replaces the base mask buffer of the column inplace. This does not + modify size or offset in any way, so the passed mask is expected to be + compatible with the current offset. + """ + if value is not None and not isinstance(value, Buffer): + raise TypeError( + "Expected a Buffer or None for mask, " + f"got {type(value).__name__}" + ) + + if value is not None: + # bitmask size must be relative to offset = 0 data. + required_size = plc.null_mask.bitmask_allocation_size_bytes( + self.base_size + ) + if value.size < required_size: + error_msg = ( + "The Buffer for mask is smaller than expected, " + f"got {value.size} bytes, expected {required_size} bytes." + ) + if self.offset > 0 or self.size < self.base_size: + error_msg += ( + "\n\nNote: The mask is expected to be sized according " + "to the base allocation as opposed to the offsetted or" + " sized allocation." + ) + raise ValueError(error_msg) + + self._mask = None + self._children = None + self._base_mask = value # type: ignore[assignment] + self._clear_cache() + + def _clear_cache(self) -> None: + self._distinct_count.clear() + attrs = ( + "memory_usage", + "is_monotonic_increasing", + "is_monotonic_decreasing", + ) + for attr in attrs: + try: + delattr(self, attr) + except AttributeError: + # attr was not called yet, so ignore. + pass + self._null_count = None + + def set_mask(self, value) -> Self: + """ + Replaces the mask buffer of the column and returns a new column. This + will zero the column offset, compute a new mask buffer if necessary, + and compute new data Buffers zero-copy that use pointer arithmetic to + properly adjust the pointer. + """ + mask_size = plc.null_mask.bitmask_allocation_size_bytes(self.size) + required_num_bytes = -(-self.size // 8) # ceiling divide + error_msg = ( + "The value for mask is smaller than expected, got {} bytes, " + f"expected {required_num_bytes} bytes." + ) + if value is None: + mask = None + elif hasattr(value, "__cuda_array_interface__"): + if value.__cuda_array_interface__["typestr"] not in ("|i1", "|u1"): + if isinstance(value, ColumnBase): + value = value.data_array_view(mode="write") + value = cupy.asarray(value).view("|u1") + mask = as_buffer(value) + if mask.size < required_num_bytes: + raise ValueError(error_msg.format(str(value.size))) + if mask.size < mask_size: + dbuf = rmm.DeviceBuffer(size=mask_size) + dbuf.copy_from_device(value) + mask = as_buffer(dbuf) + elif hasattr(value, "__array_interface__"): + value = np.asarray(value).view("u1")[:mask_size] + if value.size < required_num_bytes: + raise ValueError(error_msg.format(str(value.size))) + dbuf = rmm.DeviceBuffer(size=mask_size) + dbuf.copy_from_host(value) + mask = as_buffer(dbuf) + else: + try: + value = memoryview(value) + except TypeError as err: + raise TypeError( + f"Expected a Buffer object or None for mask, got {type(value).__name__}" + ) from err + else: + value = np.asarray(value).view("u1")[:mask_size] + if value.size < required_num_bytes: + raise ValueError(error_msg.format(str(value.size))) + dbuf = rmm.DeviceBuffer(size=mask_size) + dbuf.copy_from_host(value) + mask = as_buffer(dbuf) + + return cudf.core.column.build_column( # type: ignore[return-value] + data=self.data, + dtype=self.dtype, + mask=mask, + size=self.size, + offset=0, + children=self.children, + ) + + @property + def null_count(self) -> int: + if self._null_count is None: + if not self.nullable or self.size == 0: + self._null_count = 0 + else: + with acquire_spill_lock(): + self._null_count = plc.null_mask.null_count( + self.base_mask.get_ptr(mode="read"), # type: ignore[union-attr] + self.offset, + self.offset + self.size, + ) + return self._null_count + + @property + def offset(self) -> int: + return self._offset + + @property + def base_children(self) -> tuple[ColumnBase, ...]: + return self._base_children # type: ignore[has-type] + + @property + def children(self) -> tuple[ColumnBase, ...]: + if self.offset == 0 and self.size == self.base_size: + self._children = self.base_children # type: ignore[assignment] + if self._children is None: + if not self.base_children: + self._children = () # type: ignore[assignment] + else: + # Compute children from the column view (children factoring self.size) + children = ColumnBase.from_pylibcudf( + self.to_pylibcudf(mode="read").copy() + ).base_children + dtypes = ( + base_child.dtype for base_child in self.base_children + ) + self._children = tuple( # type: ignore[assignment] + child._with_type_metadata(dtype) + for child, dtype in zip(children, dtypes) + ) + return self._children # type: ignore[return-value] + + def set_base_children(self, value: tuple[ColumnBase, ...]) -> None: + if not isinstance(value, tuple): + raise TypeError( + f"Expected a tuple of Columns for children, got {type(value).__name__}" + ) + if any(not isinstance(child, ColumnBase) for child in value): + raise TypeError("All children must be Columns.") + + self._children = None + self._base_children = value + + def _mimic_inplace( + self, other_col: Self, inplace: bool = False + ) -> None | Self: + """ + Given another column, update the attributes of this column to mimic an + inplace operation. This does not modify the memory of Buffers, but + instead replaces the Buffers and other attributes underneath the column + object with the Buffers and attributes from the other column. + """ + if inplace: + self._offset = other_col.offset + self._size = other_col.size + self._dtype = other_col._dtype + self.set_base_data(other_col.base_data) + self.set_base_children(other_col.base_children) + self.set_base_mask(other_col.base_mask) + # TODO: self._clear_cache here? + return None + else: + return other_col + + # TODO: Consider whether this function should support some sort of `copy` + # parameter. Not urgent until this functionality is moved up to the Frame + # layer and made public. This function will also need to mark the + # underlying buffers as exposed before this function can itself be exposed + # publicly. User requests to convert to pylibcudf must assume that the + # data may be modified afterwards. + def to_pylibcudf(self, mode: Literal["read", "write"]) -> plc.Column: + """Convert this Column to a pylibcudf.Column. + + This function will generate a pylibcudf Column pointing to the same + data, mask, and children as this one. + + Parameters + ---------- + mode : str + Supported values are {"read", "write"} If "write", the data pointed + to may be modified by the caller. If "read", the data pointed to + must not be modified by the caller. Failure to fulfill this + contract will cause incorrect behavior. + + Returns + ------- + pylibcudf.Column + A new pylibcudf.Column referencing the same data. + """ + + # TODO: Categoricals will need to be treated differently eventually. + # There is no 1-1 correspondence between cudf and libcudf for + # categoricals because cudf supports ordered and unordered categoricals + # while libcudf supports only unordered categoricals (see + # https://github.com/rapidsai/cudf/pull/8567). + if isinstance(self.dtype, cudf.CategoricalDtype): + col = self.base_children[0] + else: + col = self + + dtype = dtype_to_pylibcudf_type(col.dtype) + + data = None + if col.base_data is not None: + cai = cuda_array_interface_wrapper( + ptr=col.base_data.get_ptr(mode=mode), + size=col.base_data.size, + owner=col.base_data, + ) + data = plc.gpumemoryview(cai) + + mask = None + if self.nullable: + # TODO: Are we intentionally use self's mask instead of col's? + # Where is the mask stored for categoricals? + cai = cuda_array_interface_wrapper( + ptr=self.base_mask.get_ptr(mode=mode), # type: ignore[union-attr] + size=self.base_mask.size, # type: ignore[union-attr] + owner=self.base_mask, + ) + mask = plc.gpumemoryview(cai) + + children = [] + if col.base_children: + children = [ + child_column.to_pylibcudf(mode=mode) + for child_column in col.base_children + ] + + return plc.Column( + dtype, + self.size, + data, + mask, + self.null_count, + self.offset, + children, + ) + + @classmethod + def from_pylibcudf( + cls, col: plc.Column, data_ptr_exposed: bool = False + ) -> Self: + """Create a Column from a pylibcudf.Column. + + This function will generate a Column pointing to the provided pylibcudf + Column. It will directly access the data and mask buffers of the + pylibcudf Column, so the newly created object is not tied to the + lifetime of the original pylibcudf.Column. + + Parameters + ---------- + col : pylibcudf.Column + The object to copy. + data_ptr_exposed : bool + Whether the data buffer is exposed. + + Returns + ------- + pylibcudf.Column + A new pylibcudf.Column referencing the same data. + """ + if col.type().id() == plc.TypeId.TIMESTAMP_DAYS: + col = plc.unary.cast( + col, plc.DataType(plc.TypeId.TIMESTAMP_SECONDS) + ) + elif col.type().id() == plc.TypeId.EMPTY: + new_dtype = plc.DataType(plc.TypeId.INT8) + + col = plc.column_factories.make_numeric_column( + new_dtype, col.size(), plc.column_factories.MaskState.ALL_NULL + ) + + dtype = dtype_from_pylibcudf_column(col) + + return cudf.core.column.build_column( # type: ignore[return-value] + data=as_buffer(col.data().obj, exposed=data_ptr_exposed) + if col.data() is not None + else None, + dtype=dtype, + size=col.size(), + mask=as_buffer(col.null_mask().obj, exposed=data_ptr_exposed) + if col.null_mask() is not None + else None, + offset=col.offset(), + null_count=col.null_count(), + children=tuple( + cls.from_pylibcudf(child, data_ptr_exposed=data_ptr_exposed) + for child in col.children() + ), + ) + + @classmethod + def from_scalar(cls, slr: cudf.Scalar, size: int) -> Self: + return cls.from_pylibcudf( + plc.Column.from_scalar(slr.device_value, size) + ) + def data_array_view( self, *, mode: Literal["write", "read"] = "write" ) -> "cuda.devicearray.DeviceNDArray": @@ -296,9 +725,9 @@ def any(self, skipna: bool = True) -> bool: def dropna(self) -> Self: if self.has_nulls(): - return stream_compaction.drop_nulls([self])[0]._with_type_metadata( - self.dtype - ) # type: ignore[return-value] + return ColumnBase.from_pylibcudf( + stream_compaction.drop_nulls([self])[0] + )._with_type_metadata(self.dtype) # type: ignore[return-value] else: return self.copy() @@ -734,7 +1163,7 @@ def _scatter_by_column( with acquire_spill_lock(): plc_table = plc.copying.boolean_mask_scatter( plc.Table([value.to_pylibcudf(mode="read")]) - if isinstance(value, Column) + if isinstance(value, ColumnBase) else [value], plc.Table([self.to_pylibcudf(mode="read")]), key.to_pylibcudf(mode="read"), @@ -745,9 +1174,11 @@ def _scatter_by_column( ._with_type_metadata(self.dtype) ) else: - return copying.scatter( - [value], key, [self], bounds_check=bounds_check - )[0]._with_type_metadata(self.dtype) + return ColumnBase.from_pylibcudf( # type: ignore[return-value] + copying.scatter( + [value], key, [self], bounds_check=bounds_check + )[0] + )._with_type_metadata(self.dtype) def _check_scatter_key_length( self, num_keys: int, value: plc.Scalar | ColumnBase @@ -991,8 +1422,10 @@ def take( if indices.dtype.kind not in {"u", "i"}: indices = indices.astype(SIZE_TYPE_DTYPE) GatherMap(indices, len(self), nullify=not check_bounds or nullify) - gathered = copying.gather([self], indices, nullify=nullify) # type: ignore[arg-type] - return gathered[0]._with_type_metadata(self.dtype) # type: ignore[return-value] + gathered = ColumnBase.from_pylibcudf( + copying.gather([self], indices, nullify=nullify)[0] # type: ignore[arg-type] + ) + return gathered._with_type_metadata(self.dtype) # type: ignore[return-value] def isin(self, values: Sequence) -> ColumnBase: """Check whether values are contained in the Column. @@ -1114,7 +1547,7 @@ def contains(self, other: ColumnBase) -> ColumnBase: A column of values to search for """ with acquire_spill_lock(): - return Column.from_pylibcudf( + return ColumnBase.from_pylibcudf( plc.search.contains( self.to_pylibcudf(mode="read"), other.to_pylibcudf(mode="read"), @@ -1301,9 +1734,9 @@ def apply_boolean_mask(self, mask) -> ColumnBase: if mask.dtype.kind != "b": raise ValueError("boolean_mask is not boolean type.") - return stream_compaction.apply_boolean_mask([self], mask)[ - 0 - ]._with_type_metadata(self.dtype) + return ColumnBase.from_pylibcudf( + stream_compaction.apply_boolean_mask([self], mask)[0] + )._with_type_metadata(self.dtype) def argsort( self, @@ -1324,8 +1757,8 @@ def argsort( as_column(range(len(self) - 1, -1, -1)), ) else: - return sorting.order_by( - [self], [ascending], na_position, stable=True + return ColumnBase.from_pylibcudf( # type: ignore[return-value] + sorting.order_by([self], [ascending], na_position, stable=True) ) def __arrow_array__(self, type=None): @@ -1376,12 +1809,14 @@ def searchsorted( raise ValueError( "Column searchsorted expects values to be column of same dtype" ) - return search.search_sorted( # type: ignore[return-value] - [self], - [value], - side=side, - ascending=ascending, - na_position=na_position, + return ColumnBase.from_pylibcudf( + search.search_sorted( # type: ignore[return-value] + [self], + [value], + side=side, + ascending=ascending, + na_position=na_position, + ) ) def unique(self) -> Self: @@ -1391,9 +1826,11 @@ def unique(self) -> Self: if self.is_unique: return self.copy() else: - return stream_compaction.drop_duplicates([self], keep="first")[ # type: ignore[return-value] - 0 - ]._with_type_metadata(self.dtype) + return ColumnBase.from_pylibcudf( + stream_compaction.drop_duplicates([self], keep="first")[ # type: ignore[return-value] + 0 + ] + )._with_type_metadata(self.dtype) def serialize(self) -> tuple[dict, list]: # data model: @@ -1629,10 +2066,10 @@ def _return_sentinel_column(): del right_rows # reorder `codes` so that its values correspond to the # values of `self`: - (codes,) = sorting.sort_by_key( + plc_codes = sorting.sort_by_key( [codes], [left_gather_map], [True], ["last"], stable=True - ) - return codes.fillna(na_sentinel) + )[0] + return ColumnBase.from_pylibcudf(plc_codes).fillna(na_sentinel) @acquire_spill_lock() def copy_if_else( @@ -2027,7 +2464,7 @@ def as_column( """ if isinstance(arbitrary, (range, pd.RangeIndex, cudf.RangeIndex)): with acquire_spill_lock(): - column = Column.from_pylibcudf( + column = ColumnBase.from_pylibcudf( plc.filling.sequence( len(arbitrary), pa_scalar_to_plc_scalar( @@ -2606,7 +3043,7 @@ def concat_columns(objs: "MutableSequence[ColumnBase]") -> ColumnBase: # Filter out inputs that have 0 length, then concatenate. objs_with_len = [o for o in objs if len(o)] with acquire_spill_lock(): - return Column.from_pylibcudf( + return ColumnBase.from_pylibcudf( plc.concatenate.concatenate( [col.to_pylibcudf(mode="read") for col in objs_with_len] ) diff --git a/python/cudf/cudf/core/column/datetime.py b/python/cudf/cudf/core/column/datetime.py index 439dd7991d6..92d5c39e69d 100644 --- a/python/cudf/cudf/core/column/datetime.py +++ b/python/cudf/cudf/core/column/datetime.py @@ -19,7 +19,6 @@ import cudf import cudf.core.column.column as column -from cudf import _lib as libcudf from cudf.core._compat import PANDAS_GE_220 from cudf.core._internals import binaryop from cudf.core._internals.timezones import ( @@ -929,7 +928,7 @@ def _find_ambiguous_and_nonexistent( ambiguous_end.to_pylibcudf(mode="read"), plc.labeling.Inclusive.NO, ) - ambiguous = libcudf.column.Column.from_pylibcudf(plc_column) + ambiguous = ColumnBase.from_pylibcudf(plc_column) ambiguous = ambiguous.notnull() # At the start of a non-existent time period, Clock 2 reads less @@ -948,10 +947,10 @@ def _find_ambiguous_and_nonexistent( nonexistent_end.to_pylibcudf(mode="read"), plc.labeling.Inclusive.NO, ) - nonexistent = libcudf.column.Column.from_pylibcudf(plc_column) + nonexistent = ColumnBase.from_pylibcudf(plc_column) nonexistent = nonexistent.notnull() - return ambiguous, nonexistent + return ambiguous, nonexistent # type: ignore[return-value] def tz_localize( self, diff --git a/python/cudf/cudf/core/column/string.py b/python/cudf/cudf/core/column/string.py index 7cc0e75b4bd..04a72017c33 100644 --- a/python/cudf/cudf/core/column/string.py +++ b/python/cudf/cudf/core/column/string.py @@ -19,7 +19,6 @@ import cudf.api.types import cudf.core.column.column as column import cudf.core.column.datetime as datetime -from cudf._lib.column import Column from cudf.api.types import is_integer, is_scalar, is_string_dtype from cudf.core._internals import binaryop from cudf.core.buffer import acquire_spill_lock @@ -168,7 +167,7 @@ def len(self) -> SeriesOrIndex: plc_column = plc.strings.attributes.count_characters( self._column.to_pylibcudf(mode="read") ) - result = Column.from_pylibcudf(plc_column) + result = ColumnBase.from_pylibcudf(plc_column) return self._return_or_inplace(result) def byte_count(self) -> SeriesOrIndex: @@ -202,7 +201,7 @@ def byte_count(self) -> SeriesOrIndex: plc_column = plc.strings.attributes.count_bytes( self._column.to_pylibcudf(mode="read") ) - result = Column.from_pylibcudf(plc_column) + result = ColumnBase.from_pylibcudf(plc_column) return self._return_or_inplace(result) @overload @@ -311,7 +310,7 @@ def cat(self, others=None, sep=None, na_rep=None): pa.scalar(na_rep, type=pa.string()) ), ) - data = Column.from_pylibcudf(plc_column) + data = ColumnBase.from_pylibcudf(plc_column) else: parent_index = ( self._parent.index @@ -374,7 +373,7 @@ def cat(self, others=None, sep=None, na_rep=None): pa.scalar(na_rep, type=pa.string()) ), ) - data = Column.from_pylibcudf(plc_column) + data = ColumnBase.from_pylibcudf(plc_column) if len(data) == 1 and data.null_count == 1: data = cudf.core.column.as_column("", length=len(data)) @@ -540,7 +539,7 @@ def join( plc.strings.combine.SeparatorOnNulls.YES, plc.strings.combine.OutputIfEmptyList.NULL_ELEMENT, ) - data = Column.from_pylibcudf(plc_column) + data = ColumnBase.from_pylibcudf(plc_column) elif can_convert_to_column(sep): sep_column = column.as_column(sep) if len(sep_column) != len(strings_column): @@ -562,7 +561,7 @@ def join( plc.strings.combine.SeparatorOnNulls.YES, plc.strings.combine.OutputIfEmptyList.NULL_ELEMENT, ) - data = Column.from_pylibcudf(plc_column) + data = ColumnBase.from_pylibcudf(plc_column) else: raise TypeError( f"sep should be an str, array-like or Series object, " @@ -659,7 +658,8 @@ def extract( ) data = dict( enumerate( - Column.from_pylibcudf(col) for col in plc_result.columns() + ColumnBase.from_pylibcudf(col) + for col in plc_result.columns() ) ) if len(data) == 1 and expand is False: @@ -806,7 +806,7 @@ def contains( plc_result = plc.strings.contains.contains_re( self._column.to_pylibcudf(mode="read"), prog ) - result_col = Column.from_pylibcudf(plc_result) + result_col = ColumnBase.from_pylibcudf(plc_result) else: if case is False: input_column = self.lower()._column # type: ignore[union-attr] @@ -819,7 +819,7 @@ def contains( input_column.to_pylibcudf(mode="read"), pa_scalar_to_plc_scalar(pa.scalar(pat_normed)), ) - result_col = Column.from_pylibcudf(plc_result) + result_col = ColumnBase.from_pylibcudf(plc_result) else: # TODO: we silently ignore the `regex=` flag here if case is False: @@ -837,7 +837,7 @@ def contains( input_column.to_pylibcudf(mode="read"), col_pat.to_pylibcudf(mode="read"), ) - result_col = Column.from_pylibcudf(plc_result) + result_col = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result_col) def like(self, pat: str, esc: str | None = None) -> SeriesOrIndex: @@ -909,7 +909,7 @@ def like(self, pat: str, esc: str | None = None) -> SeriesOrIndex: pa_scalar_to_plc_scalar(pa.scalar(pat)), pa_scalar_to_plc_scalar(pa.scalar(esc)), ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) @@ -966,7 +966,7 @@ def repeat( plc_result = plc.strings.repeat.repeat_strings( self._column.to_pylibcudf(mode="read"), repeats ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def replace( @@ -1062,7 +1062,7 @@ def replace( repl, dtype=CUDF_STRING_DTYPE ).to_pylibcudf(mode="read"), ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) else: result = self._column.replace_multiple( cast( @@ -1105,7 +1105,7 @@ def replace( pa_scalar_to_plc_scalar(pa_repl), n, ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def replace_with_backrefs(self, pat: str, repl: str) -> SeriesOrIndex: @@ -1146,7 +1146,7 @@ def replace_with_backrefs(self, pat: str, repl: str) -> SeriesOrIndex: ), repl, ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def slice( @@ -1226,7 +1226,7 @@ def slice( pa_scalar_to_plc_scalar(pa.scalar(stop, param_dtype)), pa_scalar_to_plc_scalar(pa.scalar(step, param_dtype)), ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def _all_characters_of_type( @@ -1238,7 +1238,7 @@ def _all_characters_of_type( plc_column = plc.strings.char_types.all_characters_of_type( self._column.to_pylibcudf(mode="read"), char_type, case_type ) - result = Column.from_pylibcudf(plc_column) + result = ColumnBase.from_pylibcudf(plc_column) return self._return_or_inplace(result) def isinteger(self) -> SeriesOrIndex: @@ -2203,7 +2203,7 @@ def filter_alphanum( if keep else plc.strings.char_types.StringCharacterTypes.ALL_TYPES, ) - result = Column.from_pylibcudf(plc_column) + result = ColumnBase.from_pylibcudf(plc_column) return self._return_or_inplace(result) def slice_from( @@ -2250,7 +2250,7 @@ def slice_from( starts._column.to_pylibcudf(mode="read"), stops._column.to_pylibcudf(mode="read"), ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def slice_replace( @@ -2346,7 +2346,7 @@ def slice_replace( start, stop, ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def insert(self, start: int = 0, repl: str | None = None) -> SeriesOrIndex: @@ -2532,7 +2532,7 @@ def get_json_object( pa_scalar_to_plc_scalar(pa.scalar(json_path)), options, ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def split( @@ -3129,7 +3129,7 @@ def pad( side, fillchar, ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def zfill(self, width: int) -> SeriesOrIndex: @@ -3200,7 +3200,7 @@ def zfill(self, width: int) -> SeriesOrIndex: plc_result = plc.strings.padding.zfill( self._column.to_pylibcudf(mode="read"), width ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def center(self, width: int, fillchar: str = " ") -> SeriesOrIndex: @@ -3349,7 +3349,7 @@ def _strip( side, pa_scalar_to_plc_scalar(pa.scalar(to_strip, type=pa.string())), ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def strip(self, to_strip: str | None = None) -> SeriesOrIndex: @@ -3594,7 +3594,7 @@ def wrap(self, width: int, **kwargs) -> SeriesOrIndex: plc_result = plc.strings.wrap.wrap( self._column.to_pylibcudf(mode="read"), width ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def count(self, pat: str, flags: int = 0) -> SeriesOrIndex: @@ -3668,7 +3668,7 @@ def count(self, pat: str, flags: int = 0) -> SeriesOrIndex: plc_result = plc.strings.contains.count_re( self._column.to_pylibcudf(mode="read"), prog ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def _findall( @@ -3692,7 +3692,7 @@ def _findall( self._column.to_pylibcudf(mode="read"), prog, ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def findall(self, pat: str, flags: int = 0) -> SeriesOrIndex: @@ -3854,7 +3854,7 @@ def find_multiple(self, patterns: SeriesOrIndex) -> cudf.Series: self._column.to_pylibcudf(mode="read"), patterns_column.to_pylibcudf(mode="read"), ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return cudf.Series._from_column( result, @@ -3972,7 +3972,7 @@ def _starts_ends_with( plc_result = method( self._column.to_pylibcudf(mode="read"), plc_pat ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def endswith(self, pat: str | tuple[str, ...]) -> SeriesOrIndex: @@ -4104,7 +4104,7 @@ def removesuffix(self, suffix: str) -> SeriesOrIndex: ends_column = self.endswith(suffix)._column # type: ignore[union-attr] removed_column = self.slice(0, -len(suffix), None)._column # type: ignore[union-attr] - result = removed_column.copy_if_else(self._column, ends_column) + result = removed_column.copy_if_else(self._column, ends_column) # type: ignore[arg-type] return self._return_or_inplace(result) def removeprefix(self, prefix: str) -> SeriesOrIndex: @@ -4142,7 +4142,7 @@ def removeprefix(self, prefix: str) -> SeriesOrIndex: return self._return_or_inplace(self._column) starts_column = self.startswith(prefix)._column # type: ignore[union-attr] removed_column = self.slice(len(prefix), None, None)._column # type: ignore[union-attr] - result = removed_column.copy_if_else(self._column, starts_column) + result = removed_column.copy_if_else(self._column, starts_column) # type: ignore[arg-type] return self._return_or_inplace(result) def _find( @@ -4167,7 +4167,7 @@ def _find( start, end, ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def find( @@ -4447,7 +4447,7 @@ def match( plc_result = plc.strings.contains.matches_re( self._column.to_pylibcudf(mode="read"), prog ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def url_decode(self) -> SeriesOrIndex: @@ -4545,7 +4545,7 @@ def code_points(self) -> SeriesOrIndex: plc_column = plc.strings.attributes.code_points( self._column.to_pylibcudf(mode="read") ) - result = Column.from_pylibcudf(plc_column) + result = ColumnBase.from_pylibcudf(plc_column) return self._return_or_inplace(result, retain_index=False) def translate(self, table: dict) -> SeriesOrIndex: @@ -4593,7 +4593,7 @@ def translate(self, table: dict) -> SeriesOrIndex: plc_result = plc.strings.translate.translate( self._column.to_pylibcudf(mode="read"), table ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def filter_characters( @@ -4652,7 +4652,7 @@ def filter_characters( else plc.strings.translate.FilterType.REMOVE, pa_scalar_to_plc_scalar(pa.scalar(repl, type=pa.string())), ) - result = Column.from_pylibcudf(plc_result) + result = ColumnBase.from_pylibcudf(plc_result) return self._return_or_inplace(result) def normalize_spaces(self) -> SeriesOrIndex: @@ -4755,7 +4755,7 @@ def tokenize(self, delimiter: str = " ") -> SeriesOrIndex: """ delim = _massage_string_arg(delimiter, "delimiter", allow_col=True) - if isinstance(delim, Column): + if isinstance(delim, ColumnBase): result = self._return_or_inplace( self._column.tokenize_column(delim), # type: ignore[arg-type] retain_index=False, @@ -4896,7 +4896,7 @@ def token_count(self, delimiter: str = " ") -> SeriesOrIndex: dtype: int32 """ delim = _massage_string_arg(delimiter, "delimiter", allow_col=True) - if isinstance(delim, Column): + if isinstance(delim, ColumnBase): return self._return_or_inplace( self._column.count_tokens_column(delim) # type: ignore[arg-type] ) @@ -5805,7 +5805,9 @@ def sum( pa_scalar_to_plc_scalar(pa.scalar("")), pa_scalar_to_plc_scalar(pa.scalar(None, type=pa.string())), ) - return Column.from_pylibcudf(plc_column).element_indexing(0) + return ColumnBase.from_pylibcudf(plc_column).element_indexing( + 0 + ) else: return result_col @@ -5820,7 +5822,7 @@ def as_numerical_column(self, dtype: Dtype) -> NumericalColumn: plc_column = plc.strings.attributes.count_characters( self.to_pylibcudf(mode="read") ) - result = Column.from_pylibcudf(plc_column) + result = ColumnBase.from_pylibcudf(plc_column) return (result > np.int8(0)).fillna(False) elif out_dtype.kind in {"i", "u"}: if not self.is_integer().all(): @@ -5870,7 +5872,7 @@ def strptime( plc_column = plc.strings.attributes.count_characters( without_nat.to_pylibcudf(mode="read") ) - char_counts = Column.from_pylibcudf(plc_column) + char_counts = ColumnBase.from_pylibcudf(plc_column) if char_counts.distinct_count(dropna=True) != 1: # Unfortunately disables OK cases like: # ["2020-01-01", "2020-01-01 00:00:00"] @@ -5927,7 +5929,7 @@ def as_decimal_column( self.to_pylibcudf(mode="read"), dtype_to_pylibcudf_type(dtype), ) - result = Column.from_pylibcudf(plc_column) + result = ColumnBase.from_pylibcudf(plc_column) result.dtype.precision = dtype.precision # type: ignore[union-attr] return result # type: ignore[return-value] @@ -6093,7 +6095,7 @@ def _binaryop( pa.scalar(None, type=pa.string()) ), ) - return Column.from_pylibcudf(plc_column) + return ColumnBase.from_pylibcudf(plc_column) elif op in { "__eq__", "__ne__", @@ -6108,8 +6110,8 @@ def _binaryop( return binaryop.binaryop(lhs=lhs, rhs=rhs, op=op, dtype="bool") return NotImplemented - @copy_docstring(column.ColumnBase.view) - def view(self, dtype) -> "cudf.core.column.ColumnBase": + @copy_docstring(ColumnBase.view) + def view(self, dtype) -> ColumnBase: if self.null_count > 0: raise ValueError( "Can not produce a view of a string column with nulls" @@ -6255,7 +6257,7 @@ def normalize_spaces(self) -> Self: @acquire_spill_lock() def normalize_characters(self, do_lower: bool = True) -> Self: - return Column.from_pylibcudf( # type: ignore[return-value] + return ColumnBase.from_pylibcudf( # type: ignore[return-value] plc.nvtext.normalize.normalize_characters( self.to_pylibcudf(mode="read"), do_lower, @@ -6413,7 +6415,7 @@ def _modify_characters( Helper function for methods that modify characters e.g. to_lower """ plc_column = method(self.to_pylibcudf(mode="read")) - return cast(Self, Column.from_pylibcudf(plc_column)) + return cast(Self, ColumnBase.from_pylibcudf(plc_column)) def to_lower(self) -> Self: return self._modify_characters(plc.strings.case.to_lower) @@ -6440,7 +6442,7 @@ def replace_multiple(self, pattern: Self, replacements: Self) -> Self: pattern.to_pylibcudf(mode="read"), replacements.to_pylibcudf(mode="read"), ) - return cast(Self, Column.from_pylibcudf(plc_result)) + return cast(Self, ColumnBase.from_pylibcudf(plc_result)) @acquire_spill_lock() def is_hex(self) -> NumericalColumn: @@ -6500,7 +6502,7 @@ def _split_record_re( ), maxsplit, ) - return cast(Self, Column.from_pylibcudf(plc_column)) + return cast(Self, ColumnBase.from_pylibcudf(plc_column)) def split_record_re(self, pattern: str, maxsplit: int) -> Self: return self._split_record_re( @@ -6532,7 +6534,7 @@ def _split_re( ) return dict( enumerate( - Column.from_pylibcudf(col) # type: ignore[misc] + ColumnBase.from_pylibcudf(col) # type: ignore[misc] for col in plc_table.columns() ) ) @@ -6585,7 +6587,7 @@ def _split( ) return dict( enumerate( - Column.from_pylibcudf(col) # type: ignore[misc] + ColumnBase.from_pylibcudf(col) # type: ignore[misc] for col in plc_table.columns() ) ) @@ -6608,7 +6610,7 @@ def _partition( ) return dict( enumerate( - Column.from_pylibcudf(col) # type: ignore[misc] + ColumnBase.from_pylibcudf(col) # type: ignore[misc] for col in plc_table.columns() ) ) diff --git a/python/cudf/cudf/core/cut.py b/python/cudf/cudf/core/cut.py index 5bfea45a946..67c29dc59ed 100644 --- a/python/cudf/cudf/core/cut.py +++ b/python/cudf/cudf/core/cut.py @@ -1,4 +1,4 @@ -# Copyright (c) 2021-2024, NVIDIA CORPORATION. +# Copyright (c) 2021-2025, NVIDIA CORPORATION. from collections import abc @@ -9,10 +9,9 @@ import pylibcudf as plc import cudf -from cudf._lib.column import Column from cudf.api.types import is_list_like from cudf.core.buffer import acquire_spill_lock -from cudf.core.column import as_column +from cudf.core.column import ColumnBase, as_column from cudf.core.column.categorical import CategoricalColumn, as_unsigned_codes from cudf.core.index import IntervalIndex, interval_range @@ -272,7 +271,7 @@ def cut( if right_inclusive else plc.labeling.Inclusive.NO, ) - index_labels = Column.from_pylibcudf(plc_column) + index_labels = ColumnBase.from_pylibcudf(plc_column) if labels is False: # if labels is false we return the index labels, we return them diff --git a/python/cudf/cudf/core/dataframe.py b/python/cudf/cudf/core/dataframe.py index 8e0cb606b2e..69db055fe87 100644 --- a/python/cudf/cudf/core/dataframe.py +++ b/python/cudf/cudf/core/dataframe.py @@ -36,7 +36,6 @@ import cudf import cudf.core.common -from cudf import _lib as libcudf from cudf.api.extensions import no_default from cudf.api.types import ( _is_scalar_or_zero_d_array, @@ -2508,8 +2507,7 @@ def scatter_by_map( map_size, ) partitioned_columns = [ - libcudf.column.Column.from_pylibcudf(col) - for col in plc_table.columns() + ColumnBase.from_pylibcudf(col) for col in plc_table.columns() ] partitioned = self._from_columns_like_self( @@ -4132,7 +4130,7 @@ def transpose(self): ) ) result_columns = [ - libcudf.column.Column.from_pylibcudf(col, data_ptr_exposed=True) + ColumnBase.from_pylibcudf(col, data_ptr_exposed=True) for col in result_table.columns() ] @@ -5040,8 +5038,7 @@ def partition_by_hash( nparts, ) output_columns = [ - libcudf.column.Column.from_pylibcudf(col) - for col in plc_table.columns() + ColumnBase.from_pylibcudf(col) for col in plc_table.columns() ] outdf = self._from_columns_like_self( @@ -7256,8 +7253,7 @@ def stack( self.shape[0], ) tiled_index = [ - libcudf.column.Column.from_pylibcudf(plc) - for plc in plc_table.columns() + ColumnBase.from_pylibcudf(plc) for plc in plc_table.columns() ] # Assemble the final index @@ -7336,7 +7332,7 @@ def unnamed_group_generator(): ) with acquire_spill_lock(): - interleaved_col = libcudf.column.Column.from_pylibcudf( + interleaved_col = ColumnBase.from_pylibcudf( plc.reshape.interleave_columns( plc.Table( [ @@ -7841,7 +7837,7 @@ def interleave_columns(self): "interleave_columns does not support 'category' dtype." ) with acquire_spill_lock(): - result_col = libcudf.column.Column.from_pylibcudf( + result_col = ColumnBase.from_pylibcudf( plc.reshape.interleave_columns( plc.Table( [ @@ -7862,7 +7858,7 @@ def _compute_column(self, expr: str) -> ColumnBase: ), plc.expressions.to_expression(expr, self._column_names), ) - return libcudf.column.Column.from_pylibcudf(plc_column) + return ColumnBase.from_pylibcudf(plc_column) @_performance_tracking def eval(self, expr: str, inplace: bool = False, **kwargs): diff --git a/python/cudf/cudf/core/frame.py b/python/cudf/cudf/core/frame.py index 41b9c81198e..5284d4340d1 100644 --- a/python/cudf/cudf/core/frame.py +++ b/python/cudf/cudf/core/frame.py @@ -19,7 +19,6 @@ # TODO: The `numpy` import is needed for typing purposes during doc builds # only, need to figure out why the `np` alias is insufficient then remove. -from cudf import _lib as libcudf from cudf.api.types import is_dtype_equal, is_scalar from cudf.core._compat import PANDAS_LT_300 from cudf.core._internals import copying, search, sorting @@ -965,9 +964,9 @@ def from_arrow(cls, data: pa.Table) -> Self: for name, plc_codes in zip( dict_indices_table.column_names, plc_indices.columns() ): - codes = libcudf.column.Column.from_pylibcudf(plc_codes) + codes = ColumnBase.from_pylibcudf(plc_codes) categories = cudf_dictionaries_columns[name] - codes = as_unsigned_codes(len(categories), codes) + codes = as_unsigned_codes(len(categories), codes) # type: ignore[arg-type] cudf_category_frame[name] = CategoricalColumn( data=None, size=codes.size, @@ -981,7 +980,7 @@ def from_arrow(cls, data: pa.Table) -> Self: # Handle non-dict arrays cudf_non_category_frame = { - name: libcudf.column.Column.from_pylibcudf(plc_col) + name: ColumnBase.from_pylibcudf(plc_col) for name, plc_col in zip( data.column_names, plc.interop.from_arrow(data).columns() ) @@ -1354,12 +1353,14 @@ def searchsorted( for val, common_dtype in zip(values, common_dtype_list) ] - outcol = search.search_sorted( - sources, - values, - side, - ascending=ascending, - na_position=na_position, + outcol = ColumnBase.from_pylibcudf( + search.search_sorted( + sources, + values, + side, + ascending=ascending, + na_position=na_position, + ) ) # Return result as cupy array if the values is non-scalar @@ -1478,11 +1479,13 @@ def _get_sorted_inds( else: ascending_lst = list(ascending) - return sorting.order_by( - list(to_sort), - ascending_lst, - na_position, - stable=True, + return ColumnBase.from_pylibcudf( + sorting.order_by( + list(to_sort), + ascending_lst, + na_position, + stable=True, + ) ) @_performance_tracking @@ -1491,7 +1494,10 @@ def _split(self, splits: list[int]) -> list[Self]: Frames of length `len(splits) + 1`. """ return [ - self._from_columns_like_self(split, self._column_names) + self._from_columns_like_self( + [ColumnBase.from_pylibcudf(col) for col in split], + self._column_names, + ) for split in copying.columns_split(self._columns, splits) ] @@ -1501,10 +1507,9 @@ def _encode(self): plc.Table([col.to_pylibcudf(mode="read") for col in self._columns]) ) columns = [ - libcudf.column.Column.from_pylibcudf(col) - for col in plc_table.columns() + ColumnBase.from_pylibcudf(col) for col in plc_table.columns() ] - indices = libcudf.column.Column.from_pylibcudf(plc_column) + indices = ColumnBase.from_pylibcudf(plc_column) keys = self._from_columns_like_self(columns) return keys, indices @@ -1955,7 +1960,7 @@ def _repeat( if isinstance(repeats, ColumnBase): repeats = repeats.to_pylibcudf(mode="read") return [ - libcudf.column.Column.from_pylibcudf(col) + ColumnBase.from_pylibcudf(col) for col in plc.filling.repeat(plc_table, repeats).columns() ] diff --git a/python/cudf/cudf/core/groupby/groupby.py b/python/cudf/cudf/core/groupby/groupby.py index 8abdf88ea12..38b519c6d5f 100644 --- a/python/cudf/cudf/core/groupby/groupby.py +++ b/python/cudf/cudf/core/groupby/groupby.py @@ -19,7 +19,6 @@ import pylibcudf as plc import cudf -from cudf import _lib as libcudf from cudf.api.extensions import no_default from cudf.api.types import ( is_list_like, @@ -594,7 +593,10 @@ def indices(self) -> dict[ScalarLike, cp.ndarray]: ] ) - group_keys = stream_compaction.drop_duplicates(group_keys) + group_keys = [ + ColumnBase.from_pylibcudf(col) + for col in stream_compaction.drop_duplicates(group_keys) + ] if len(group_keys) > 1: index = cudf.MultiIndex.from_arrays(group_keys) else: @@ -1073,24 +1075,24 @@ def agg(self, func=None, *args, engine=None, engine_kwargs=None, **kwargs): plc_tables[1], plc.types.NullEquality.EQUAL, ) - left_order = libcudf.column.Column.from_pylibcudf(left_plc) - right_order = libcudf.column.Column.from_pylibcudf( - right_plc - ) + left_order = ColumnBase.from_pylibcudf(left_plc) + right_order = ColumnBase.from_pylibcudf(right_plc) # left order is some permutation of the ordering we # want, and right order is a matching gather map for # the result table. Get the correct order by sorting # the right gather map. - (right_order,) = sorting.sort_by_key( + right_order = sorting.sort_by_key( [right_order], [left_order], [True], ["first"], stable=False, - ) + )[0] result = result._gather( GatherMap.from_column_unchecked( - right_order, len(result), nullify=False + ColumnBase.from_pylibcudf(right_order), + len(result), + nullify=False, ) ) @@ -2523,7 +2525,7 @@ def _cov_or_corr(self, func, method_name): @acquire_spill_lock() def interleave_columns(source_columns): - return libcudf.column.Column.from_pylibcudf( + return ColumnBase.from_pylibcudf( plc.reshape.interleave_columns( plc.Table( [c.to_pylibcudf(mode="read") for c in source_columns] diff --git a/python/cudf/cudf/core/index.py b/python/cudf/cudf/core/index.py index bdd85ebf7eb..8ce8dfd2198 100644 --- a/python/cudf/cudf/core/index.py +++ b/python/cudf/cudf/core/index.py @@ -18,7 +18,6 @@ import pylibcudf as plc import cudf -from cudf import _lib as libcudf from cudf.api.extensions import no_default from cudf.api.types import ( _is_non_decimal_numeric_dtype, @@ -126,17 +125,21 @@ def _lexsorted_equal_range( else: sort_inds = None sort_vals = idx - lower_bound = search.search_sorted( - list(sort_vals._columns), - keys, - side="left", - ascending=sort_vals.is_monotonic_increasing, + lower_bound = ColumnBase.from_pylibcudf( + search.search_sorted( + list(sort_vals._columns), + keys, + side="left", + ascending=sort_vals.is_monotonic_increasing, + ) ).element_indexing(0) - upper_bound = search.search_sorted( - list(sort_vals._columns), - keys, - side="right", - ascending=sort_vals.is_monotonic_increasing, + upper_bound = ColumnBase.from_pylibcudf( + search.search_sorted( + list(sort_vals._columns), + keys, + side="right", + ascending=sort_vals.is_monotonic_increasing, + ) ).element_indexing(0) return lower_bound, upper_bound, sort_inds @@ -1367,8 +1370,8 @@ def get_indexer(self, target, method=None, limit=None, tolerance=None): plc.Table([rcol.to_pylibcudf(mode="read")]), plc.types.NullEquality.EQUAL, ) - scatter_map = libcudf.column.Column.from_pylibcudf(left_plc) - indices = libcudf.column.Column.from_pylibcudf(right_plc) + scatter_map = ColumnBase.from_pylibcudf(left_plc) + indices = ColumnBase.from_pylibcudf(right_plc) result = result._scatter_by_column(scatter_map, indices) result_series = cudf.Series._from_column(result) @@ -3394,7 +3397,7 @@ def interval_range( pa_freq = pa_freq.cast(cudf_dtype_to_pa_type(common_dtype)) with acquire_spill_lock(): - bin_edges = libcudf.column.Column.from_pylibcudf( + bin_edges = ColumnBase.from_pylibcudf( plc.filling.sequence( size=periods + 1, init=pa_scalar_to_plc_scalar(pa_start), diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index b0ef779bab8..ac4303394f7 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -26,7 +26,6 @@ import pylibcudf as plc import cudf -import cudf._lib as libcudf import cudf.core import cudf.core.algorithms from cudf.api.extensions import no_default @@ -2940,7 +2939,7 @@ def hash_values( plc_column = plc.hashing.sha512(plc_table) else: raise ValueError(f"Unsupported hashing algorithm {method}.") - result = libcudf.column.Column.from_pylibcudf(plc_column) + result = ColumnBase.from_pylibcudf(plc_column) return cudf.Series._from_column( result, index=self.index, @@ -2962,13 +2961,16 @@ def _gather( if not gather_map.nullify and len(self) != gather_map.nrows: raise IndexError("Gather map is out of bounds") return self._from_columns_like_self( - copying.gather( - itertools.chain(self.index._columns, self._columns) - if keep_index - else self._columns, - gather_map.column, - nullify=gather_map.nullify, - ), + [ + ColumnBase.from_pylibcudf(col) + for col in copying.gather( + itertools.chain(self.index._columns, self._columns) + if keep_index + else self._columns, + gather_map.column, + nullify=gather_map.nullify, + ) + ], self._column_names, self.index.names if keep_index else None, ) @@ -3058,7 +3060,7 @@ def _slice(self, arg: slice, keep_index: bool = True) -> Self: [start, stop], ) sliced = [ - libcudf.column.Column.from_pylibcudf(col) + ColumnBase.from_pylibcudf(col) for col in plc_tables[0].columns() ] result = self._from_columns_like_self( @@ -3123,14 +3125,17 @@ def drop_duplicates( subset, offset_by_index_columns=not ignore_index ) return self._from_columns_like_self( - stream_compaction.drop_duplicates( - list(self._columns) - if ignore_index - else list(self.index._columns + self._columns), - keys=keys, - keep=keep, - nulls_are_equal=nulls_are_equal, - ), + [ + ColumnBase.from_pylibcudf(col) + for col in stream_compaction.drop_duplicates( + list(self._columns) + if ignore_index + else list(self.index._columns + self._columns), + keys=keys, + keep=keep, + nulls_are_equal=nulls_are_equal, + ) + ], self._column_names, self.index.names if not ignore_index else None, ) @@ -3255,11 +3260,11 @@ def duplicated( plc.types.NullEquality.EQUAL, plc.types.NanEquality.ALL_EQUAL, ) - distinct = libcudf.column.Column.from_pylibcudf(plc_column) + distinct = ColumnBase.from_pylibcudf(plc_column) result = as_column( True, length=len(self), dtype=bool )._scatter_by_column( - distinct, + distinct, # type: ignore[arg-type] pa_scalar_to_plc_scalar(pa.scalar(False)), bounds_check=False, ) @@ -3281,8 +3286,7 @@ def _empty_like(self, keep_index: bool = True) -> Self: ) ) columns = [ - libcudf.column.Column.from_pylibcudf(col) - for col in plc_table.columns() + ColumnBase.from_pylibcudf(col) for col in plc_table.columns() ] result = self._from_columns_like_self( columns, @@ -3306,7 +3310,7 @@ def _split(self, splits, keep_index: bool = True) -> list[Self]: return [ self._from_columns_like_self( - split, + [ColumnBase.from_pylibcudf(col) for col in split], self._column_names, self.index.names if keep_index else None, ) @@ -4383,12 +4387,15 @@ def _drop_na_rows(self, how="any", subset=None, thresh=None): data_columns = [col.nans_to_nulls() for col in self._columns] return self._from_columns_like_self( - stream_compaction.drop_nulls( - [*self.index._columns, *data_columns], - how=how, - keys=self._positions_from_column_names(subset), - thresh=thresh, - ), + [ + ColumnBase.from_pylibcudf(col) + for col in stream_compaction.drop_nulls( + [*self.index._columns, *data_columns], + how=how, + keys=self._positions_from_column_names(subset), + thresh=thresh, + ) + ], self._column_names, self.index.names, ) @@ -4406,12 +4413,15 @@ def _apply_boolean_mask(self, boolean_mask: BooleanMask, keep_index=True): f"{len(boolean_mask.column)} not {len(self)}" ) return self._from_columns_like_self( - stream_compaction.apply_boolean_mask( - list(self.index._columns + self._columns) - if keep_index - else list(self._columns), - boolean_mask.column, - ), + [ + ColumnBase.from_pylibcudf(col) + for col in stream_compaction.apply_boolean_mask( + list(self.index._columns + self._columns) + if keep_index + else list(self._columns), + boolean_mask.column, + ) + ], column_names=self._column_names, index_names=self.index.names if keep_index else None, ) @@ -5387,8 +5397,7 @@ def _explode(self, explode_column: Any, ignore_index: bool): column_index + len(idx_cols), ) exploded = [ - libcudf.column.Column.from_pylibcudf(col) - for col in plc_table.columns() + ColumnBase.from_pylibcudf(col) for col in plc_table.columns() ] # We must copy inner datatype of the exploded list column to # maintain struct dtype key names @@ -5445,8 +5454,7 @@ def tile(self, count: int): count, ) tiled = [ - libcudf.column.Column.from_pylibcudf(plc) - for plc in plc_table.columns() + ColumnBase.from_pylibcudf(plc) for plc in plc_table.columns() ] return self._from_columns_like_self( tiled, @@ -6453,7 +6461,7 @@ def rank( source = source.nans_to_nulls() with acquire_spill_lock(): result_columns = [ - libcudf.column.Column.from_pylibcudf( + ColumnBase.from_pylibcudf( plc.sorting.rank( col.to_pylibcudf(mode="read"), method_enum, diff --git a/python/cudf/cudf/core/join/join.py b/python/cudf/cudf/core/join/join.py index d319f9e71d9..233f10cc21a 100644 --- a/python/cudf/cudf/core/join/join.py +++ b/python/cudf/cudf/core/join/join.py @@ -7,9 +7,9 @@ import pylibcudf as plc import cudf -from cudf import _lib as libcudf from cudf.core._internals import sorting from cudf.core.buffer import acquire_spill_lock +from cudf.core.column import ColumnBase from cudf.core.copy_types import GatherMap from cudf.core.join._join_helpers import ( _coerce_to_tuple, @@ -24,10 +24,10 @@ class Merge: @staticmethod @acquire_spill_lock() def _joiner( - lhs: list[libcudf.column.Column], - rhs: list[libcudf.column.Column], + lhs: list[ColumnBase], + rhs: list[ColumnBase], how: str, - ) -> tuple[libcudf.column.Column, libcudf.column.Column]: + ) -> tuple[ColumnBase, ColumnBase]: if how == "outer": how = "full" if (join_func := getattr(plc.join, f"{how}_join", None)) is None: @@ -38,9 +38,10 @@ def _joiner( plc.Table([col.to_pylibcudf(mode="read") for col in rhs]), plc.types.NullEquality.EQUAL, ) - return libcudf.column.Column.from_pylibcudf( - left_rows - ), libcudf.column.Column.from_pylibcudf(right_rows) + return ( + ColumnBase.from_pylibcudf(left_rows), + ColumnBase.from_pylibcudf(right_rows), + ) def __init__( self, @@ -266,14 +267,17 @@ def _gather_maps(self, left_cols, right_cols): ) for map_, n, null in zip(maps, lengths, nullify) ] - return sorting.sort_by_key( - list(maps), - # If how is right, right map is primary sort key. - key_order[:: -1 if self.how == "right" else 1], - [True] * len(key_order), - ["last"] * len(key_order), - stable=True, - ) + return [ + ColumnBase.from_pylibcudf(col) + for col in sorting.sort_by_key( + list(maps), + # If how is right, right map is primary sort key. + key_order[:: -1 if self.how == "right" else 1], + [True] * len(key_order), + ["last"] * len(key_order), + stable=True, + ) + ] def perform_merge(self) -> cudf.DataFrame: left_join_cols = [] @@ -451,7 +455,9 @@ def _sort_result(self, result: cudf.DataFrame) -> cudf.DataFrame: stable=True, ) result = result._from_columns_like_self( - result_columns, result._column_names, index_names + [ColumnBase.from_pylibcudf(col) for col in result_columns], + result._column_names, + index_names, ) return result @@ -575,11 +581,11 @@ def _validate_merge_params( class MergeSemi(Merge): @staticmethod @acquire_spill_lock() - def _joiner( - lhs: list[libcudf.column.Column], - rhs: list[libcudf.column.Column], + def _joiner( # type: ignore[override] + lhs: list[ColumnBase], + rhs: list[ColumnBase], how: str, - ) -> tuple[libcudf.column.Column, None]: + ) -> tuple[ColumnBase, None]: if ( join_func := getattr( plc.join, f"{how.replace('left', 'left_')}_join", None @@ -587,7 +593,7 @@ def _joiner( ) is None: raise ValueError(f"Invalid join type {how}") - return libcudf.column.Column.from_pylibcudf( + return ColumnBase.from_pylibcudf( join_func( plc.Table([col.to_pylibcudf(mode="read") for col in lhs]), plc.Table([col.to_pylibcudf(mode="read") for col in rhs]), diff --git a/python/cudf/cudf/core/multiindex.py b/python/cudf/cudf/core/multiindex.py index 24e8ed8cfc2..87a8849a260 100644 --- a/python/cudf/cudf/core/multiindex.py +++ b/python/cudf/cudf/core/multiindex.py @@ -16,7 +16,6 @@ import pylibcudf as plc import cudf -import cudf._lib as libcudf from cudf.api.extensions import no_default from cudf.api.types import is_integer, is_list_like, is_object_dtype, is_scalar from cudf.core import column @@ -24,6 +23,7 @@ from cudf.core._internals import sorting from cudf.core.algorithms import factorize from cudf.core.buffer import acquire_spill_lock +from cudf.core.column.column import ColumnBase from cudf.core.column_accessor import ColumnAccessor from cudf.core.frame import Frame from cudf.core.index import ( @@ -1962,8 +1962,8 @@ def get_indexer(self, target, method=None, limit=None, tolerance=None): plc_tables[1], plc.types.NullEquality.EQUAL, ) - scatter_map = libcudf.column.Column.from_pylibcudf(left_plc) - indices = libcudf.column.Column.from_pylibcudf(right_plc) + scatter_map = ColumnBase.from_pylibcudf(left_plc) + indices = ColumnBase.from_pylibcudf(right_plc) result_series = cudf.Series._from_column( result._scatter_by_column(scatter_map, indices) ) diff --git a/python/cudf/cudf/core/resample.py b/python/cudf/cudf/core/resample.py index 061363eee9a..de6c76cc0e1 100644 --- a/python/cudf/cudf/core/resample.py +++ b/python/cudf/cudf/core/resample.py @@ -24,9 +24,9 @@ import pylibcudf as plc import cudf -from cudf._lib.column import Column from cudf.core.abc import Serializable from cudf.core.buffer import acquire_spill_lock +from cudf.core.column import ColumnBase from cudf.core.groupby.groupby import ( DataFrameGroupBy, GroupBy, @@ -284,7 +284,7 @@ def _handle_frequency_grouper(self, by): if closed == "right" else plc.labeling.Inclusive.NO, ) - bin_numbers = Column.from_pylibcudf(plc_column) + bin_numbers = ColumnBase.from_pylibcudf(plc_column) if label == "right": cast_bin_labels = cast_bin_labels[1:] diff --git a/python/cudf/cudf/core/reshape.py b/python/cudf/cudf/core/reshape.py index 991eb86fa8f..21f8dc9bb8a 100644 --- a/python/cudf/cudf/core/reshape.py +++ b/python/cudf/cudf/core/reshape.py @@ -11,7 +11,6 @@ import pylibcudf as plc import cudf -from cudf._lib.column import Column from cudf.api.extensions import no_default from cudf.api.types import is_scalar from cudf.core._compat import PANDAS_LT_300 @@ -980,7 +979,7 @@ def _merge_sorted( ) result_columns = [ - Column.from_pylibcudf(col) for col in plc_table.columns() + ColumnBase.from_pylibcudf(col) for col in plc_table.columns() ] return objs[0]._from_columns_like_self( diff --git a/python/cudf/cudf/core/scalar.py b/python/cudf/cudf/core/scalar.py index d78ea83d578..cf85282cccb 100644 --- a/python/cudf/cudf/core/scalar.py +++ b/python/cudf/cudf/core/scalar.py @@ -175,7 +175,8 @@ def _to_plc_scalar(value: ScalarLike, dtype: Dtype) -> plc.Scalar: Returns ------- - plc.Scalar + pylibcudf.Scalar + pylibcudf.Scalar for cudf.Scalar._device_value """ if cudf.utils.utils.is_na_like(value): value = None @@ -225,7 +226,8 @@ def pa_scalar_to_plc_scalar(pa_scalar: pa.Scalar) -> plc.Scalar: Returns ------- - plc.Scalar + pylibcudf.Scalar + pylibcudf.Scalar to use in pylibcudf APIs """ return plc.interop.from_arrow(pa_scalar) diff --git a/python/cudf/cudf/core/tools/datetimes.py b/python/cudf/cudf/core/tools/datetimes.py index 0fc4d5edba8..546abfc4d3d 100644 --- a/python/cudf/cudf/core/tools/datetimes.py +++ b/python/cudf/cudf/core/tools/datetimes.py @@ -15,7 +15,6 @@ import pylibcudf as plc import cudf -from cudf import _lib as libcudf from cudf.api.types import is_integer, is_scalar from cudf.core import column from cudf.core.buffer import acquire_spill_lock @@ -987,7 +986,7 @@ def date_range( "months", 0 ) with acquire_spill_lock(): - res = libcudf.column.Column.from_pylibcudf( + res = column.ColumnBase.from_pylibcudf( plc.filling.calendrical_month_sequence( periods, pa_scalar_to_plc_scalar(pa.scalar(start)), diff --git a/python/cudf/cudf/core/udf/utils.py b/python/cudf/cudf/core/udf/utils.py index 94ce3001ca1..bfc5a67ab13 100644 --- a/python/cudf/cudf/core/udf/utils.py +++ b/python/cudf/cudf/core/udf/utils.py @@ -20,9 +20,8 @@ import rmm from cudf._lib import strings_udf -from cudf._lib.column import Column from cudf.api.types import is_scalar -from cudf.core.column.column import as_column +from cudf.core.column.column import ColumnBase, as_column from cudf.core.dtypes import dtype from cudf.core.udf.masked_typing import MaskedType from cudf.core.udf.strings_typing import ( @@ -333,7 +332,7 @@ def _return_arr_from_dtype(dtype, size): def _post_process_output_col(col, retty): if retty == _cudf_str_dtype: - return Column.from_pylibcudf( + return ColumnBase.from_pylibcudf( strings_udf.column_from_udf_string_array(col) ) return as_column(col, retty) diff --git a/python/cudf/cudf/core/window/rolling.py b/python/cudf/cudf/core/window/rolling.py index 23b0d7006b4..9e6d07878a2 100644 --- a/python/cudf/cudf/core/window/rolling.py +++ b/python/cudf/cudf/core/window/rolling.py @@ -12,18 +12,16 @@ import pylibcudf as plc import cudf -from cudf import _lib as libcudf from cudf.api.types import is_integer, is_number from cudf.core._internals import aggregation from cudf.core.buffer import acquire_spill_lock -from cudf.core.column.column import as_column +from cudf.core.column.column import ColumnBase, as_column from cudf.core.mixins import Reducible from cudf.utils import cudautils from cudf.utils.dtypes import SIZE_TYPE_DTYPE from cudf.utils.utils import GetAttrGetItemMixin if TYPE_CHECKING: - from cudf.core.column.column import ColumnBase from cudf.core.indexed_frame import IndexedFrame @@ -309,7 +307,7 @@ def _apply_agg_column(self, source_column, agg_name): pre = window fwd = 0 - return libcudf.column.Column.from_pylibcudf( + return ColumnBase.from_pylibcudf( plc.rolling.rolling_window( source_column.to_pylibcudf(mode="read"), pre, diff --git a/python/cudf/cudf/io/avro.py b/python/cudf/cudf/io/avro.py index c37df89dd28..1f5f6761cb3 100644 --- a/python/cudf/cudf/io/avro.py +++ b/python/cudf/cudf/io/avro.py @@ -3,7 +3,7 @@ import pylibcudf as plc import cudf -from cudf._lib.column import Column +from cudf.core.column import ColumnBase from cudf.core.column_accessor import ColumnAccessor from cudf.utils import ioutils @@ -48,7 +48,7 @@ def read_avro( plc_result = plc.io.avro.read_avro(options) data = { - name: Column.from_pylibcudf(col) + name: ColumnBase.from_pylibcudf(col) for name, col in zip( plc_result.column_names(include_children=False), plc_result.columns, diff --git a/python/cudf/cudf/io/csv.py b/python/cudf/cudf/io/csv.py index f83bbb5a8fa..3fbecff2c22 100644 --- a/python/cudf/cudf/io/csv.py +++ b/python/cudf/cudf/io/csv.py @@ -15,9 +15,9 @@ import pylibcudf as plc import cudf -from cudf._lib.column import Column from cudf.api.types import is_scalar from cudf.core.buffer import acquire_spill_lock +from cudf.core.column import ColumnBase from cudf.core.column_accessor import ColumnAccessor from cudf.utils import ioutils from cudf.utils.dtypes import ( @@ -276,7 +276,7 @@ def read_csv( table_w_meta = plc.io.csv.read_csv(options) data = { - name: Column.from_pylibcudf(col) + name: ColumnBase.from_pylibcudf(col) for name, col in zip( table_w_meta.column_names(include_children=False), table_w_meta.columns, diff --git a/python/cudf/cudf/io/json.py b/python/cudf/cudf/io/json.py index 8957ea04fd8..e12883b9850 100644 --- a/python/cudf/cudf/io/json.py +++ b/python/cudf/cudf/io/json.py @@ -5,7 +5,7 @@ import warnings from collections import abc from io import BytesIO, StringIO -from typing import TYPE_CHECKING, Any, Literal +from typing import Any, Literal import numpy as np import pandas as pd @@ -13,17 +13,14 @@ import pylibcudf as plc import cudf -from cudf._lib.column import Column from cudf.core.buffer import acquire_spill_lock +from cudf.core.column import ColumnBase from cudf.utils import ioutils from cudf.utils.dtypes import ( _maybe_convert_to_default_type, dtype_to_pylibcudf_type, ) -if TYPE_CHECKING: - from cudf.core.column import ColumnBase - def _get_cudf_schema_element_from_dtype( dtype, @@ -180,7 +177,7 @@ def read_json( ) ) data = { - name: Column.from_pylibcudf(col) + name: ColumnBase.from_pylibcudf(col) for name, col in zip(res_col_names, res_cols, strict=True) } df = cudf.DataFrame._from_data(data) @@ -207,7 +204,7 @@ def read_json( ) ) data = { - name: Column.from_pylibcudf(col) + name: ColumnBase.from_pylibcudf(col) for name, col in zip( table_w_meta.column_names(include_children=False), table_w_meta.columns, diff --git a/python/cudf/cudf/io/orc.py b/python/cudf/cudf/io/orc.py index 9fd40eff119..2c10f79e69a 100644 --- a/python/cudf/cudf/io/orc.py +++ b/python/cudf/cudf/io/orc.py @@ -3,16 +3,16 @@ import itertools import warnings -from typing import TYPE_CHECKING, Literal +from typing import Literal import pyarrow as pa import pylibcudf as plc import cudf -from cudf._lib.column import Column from cudf.api.types import is_list_like from cudf.core.buffer import acquire_spill_lock +from cudf.core.column import ColumnBase from cudf.core.column_accessor import ColumnAccessor from cudf.core.index import _index_from_data from cudf.utils import ioutils @@ -23,9 +23,6 @@ except ImportError: import json -if TYPE_CHECKING: - from cudf.core.column import ColumnBase - @ioutils.doc_read_orc_metadata() def read_orc_metadata(path): @@ -331,14 +328,15 @@ def read_orc( if actual_index_names is None: index = None data = { - name: Column.from_pylibcudf(col) + name: ColumnBase.from_pylibcudf(col) for name, col in zip( result_col_names, tbl_w_meta.columns, strict=True ) } else: result_columns = [ - Column.from_pylibcudf(col) for col in tbl_w_meta.columns + ColumnBase.from_pylibcudf(col) + for col in tbl_w_meta.columns ] index = _index_from_data( dict( diff --git a/python/cudf/cudf/io/parquet.py b/python/cudf/cudf/io/parquet.py index f2b174bc8ff..4b2f5969511 100644 --- a/python/cudf/cudf/io/parquet.py +++ b/python/cudf/cudf/io/parquet.py @@ -22,10 +22,9 @@ import pylibcudf as plc import cudf -from cudf._lib.column import Column from cudf.api.types import is_list_like from cudf.core.buffer import acquire_spill_lock -from cudf.core.column import as_column, column_empty +from cudf.core.column import ColumnBase, as_column, column_empty from cudf.core.column.categorical import CategoricalColumn, as_unsigned_codes from cudf.utils import ioutils from cudf.utils.performance_tracking import _performance_tracking @@ -40,8 +39,6 @@ from typing_extensions import Self - from cudf.core.column import ColumnBase - BYTE_SIZES = { "kb": 1000, @@ -1226,7 +1223,7 @@ def _read_parquet( tbl._columns[i] = None data = { - name: Column.from_pylibcudf(col) + name: ColumnBase.from_pylibcudf(col) for name, col in zip(column_names, concatenated_columns) } df = cudf.DataFrame._from_data(data) @@ -1270,7 +1267,7 @@ def _read_parquet( tbl_w_meta = plc.io.parquet.read_parquet(options) data = { - name: Column.from_pylibcudf(col) + name: ColumnBase.from_pylibcudf(col) for name, col in zip( tbl_w_meta.column_names(include_children=False), tbl_w_meta.columns, diff --git a/python/cudf/cudf/io/text.py b/python/cudf/cudf/io/text.py index 5e266c5ff55..09711bf36b0 100644 --- a/python/cudf/cudf/io/text.py +++ b/python/cudf/cudf/io/text.py @@ -1,4 +1,4 @@ -# Copyright (c) 2018-2024, NVIDIA CORPORATION. +# Copyright (c) 2018-2025, NVIDIA CORPORATION. from io import BytesIO, StringIO, TextIOBase @@ -63,6 +63,6 @@ def read_text( byte_range=byte_range, strip_delimiters=strip_delimiters ) plc_column = plc.io.text.multibyte_split(datasource, delimiter, options) - result = cudf._lib.column.Column.from_pylibcudf(plc_column) + result = cudf.core.column.ColumnBase.from_pylibcudf(plc_column) return cudf.Series._from_column(result) diff --git a/python/cudf/cudf/tests/test_string_udfs.py b/python/cudf/cudf/tests/test_string_udfs.py index 6eaf25f02ed..f0160834530 100644 --- a/python/cudf/cudf/tests/test_string_udfs.py +++ b/python/cudf/cudf/tests/test_string_udfs.py @@ -11,11 +11,11 @@ import rmm import cudf -from cudf._lib.column import Column from cudf._lib.strings_udf import ( column_from_udf_string_array, column_to_string_view_array, ) +from cudf.core.column import ColumnBase from cudf.core.udf.strings_typing import ( str_view_arg_handler, string_view, @@ -97,7 +97,9 @@ def run_udf_test(data, func, dtype): with _CUDFNumbaConfig(): sv_kernel.forall(len(data))(str_views, output) if dtype == "str": - result = Column.from_pylibcudf(column_from_udf_string_array(output)) + result = ColumnBase.from_pylibcudf( + column_from_udf_string_array(output) + ) else: result = output @@ -106,7 +108,9 @@ def run_udf_test(data, func, dtype): with _CUDFNumbaConfig(): udf_str_kernel.forall(len(data))(str_views, output) if dtype == "str": - result = Column.from_pylibcudf(column_from_udf_string_array(output)) + result = ColumnBase.from_pylibcudf( + column_from_udf_string_array(output) + ) else: result = output diff --git a/python/cudf/cudf/utils/dtypes.py b/python/cudf/cudf/utils/dtypes.py index c545b840c0e..489b804583a 100644 --- a/python/cudf/cudf/utils/dtypes.py +++ b/python/cudf/cudf/utils/dtypes.py @@ -634,6 +634,35 @@ def dtype_to_pylibcudf_type(dtype) -> plc.DataType: return plc.DataType(SUPPORTED_NUMPY_TO_PYLIBCUDF_TYPES[dtype]) +def dtype_from_pylibcudf_column(col: plc.Column) -> DtypeObj: + type_ = col.type() + tid = type_.id() + + if tid == plc.TypeId.LIST: + child = col.list_view().child() + return cudf.ListDtype(dtype_from_pylibcudf_column(child)) + elif tid == plc.TypeId.STRUCT: + fields = { + str(i): dtype_from_pylibcudf_column(col.child(i)) + for i in range(col.num_children()) + } + return cudf.StructDtype(fields) + elif tid == plc.TypeId.DECIMAL64: + return cudf.Decimal64Dtype( + precision=cudf.Decimal64Dtype.MAX_PRECISION, scale=-type_.scale() + ) + elif tid == plc.TypeId.DECIMAL32: + return cudf.Decimal32Dtype( + precision=cudf.Decimal32Dtype.MAX_PRECISION, scale=-type_.scale() + ) + elif tid == plc.TypeId.DECIMAL128: + return cudf.Decimal128Dtype( + precision=cudf.Decimal128Dtype.MAX_PRECISION, scale=-type_.scale() + ) + else: + return PYLIBCUDF_TO_SUPPORTED_NUMPY_TYPES[tid] + + SUPPORTED_NUMPY_TO_PYLIBCUDF_TYPES = { np.dtype("int8"): plc.types.TypeId.INT8, np.dtype("int16"): plc.types.TypeId.INT16, From 9d6953a6fa3c81bd6103d2fc21231cc47c906f88 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Wed, 19 Feb 2025 09:00:46 -0500 Subject: [PATCH 056/129] Remove deprecated single component datetime extract APIs (#18010) Follows up #17221 to remove the deprecated APIs. Note: This should have been removed in 25.02. Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - Yunsong Wang (https://github.com/PointKernel) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18010 --- cpp/include/cudf/datetime.hpp | 191 +----------------- cpp/include/cudf/detail/datetime.hpp | 92 +-------- cpp/src/datetime/datetime_ops.cu | 152 +------------- python/pylibcudf/pylibcudf/datetime.pxd | 14 +- python/pylibcudf/pylibcudf/datetime.pyi | 3 - python/pylibcudf/pylibcudf/datetime.pyx | 80 +------- .../pylibcudf/pylibcudf/libcudf/datetime.pxd | 32 +-- .../pylibcudf/tests/test_datetime.py | 22 +- 8 files changed, 7 insertions(+), 579 deletions(-) diff --git a/cpp/include/cudf/datetime.hpp b/cpp/include/cudf/datetime.hpp index 1f6e86d0389..f385ede96b9 100644 --- a/cpp/include/cudf/datetime.hpp +++ b/cpp/include/cudf/datetime.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -54,195 +54,6 @@ enum class datetime_component : uint8_t { NANOSECOND }; -/** - * @brief Extracts year from any datetime type and returns an int16_t - * cudf::column. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t years - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_year( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Extracts month from any datetime type and returns an int16_t - * cudf::column. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t months - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_month( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Extracts day from any datetime type and returns an int16_t - * cudf::column. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t days - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_day( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Extracts a weekday from any datetime type and returns an int16_t - * cudf::column. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t days - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_weekday( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Extracts hour from any datetime type and returns an int16_t - * cudf::column. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t hours - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_hour( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Extracts minute from any datetime type and returns an int16_t - * cudf::column. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t minutes - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_minute( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Extracts second from any datetime type and returns an int16_t - * cudf::column. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t seconds - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_second( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Extracts millisecond fraction from any datetime type and returns an int16_t - * cudf::column. - * - * A millisecond fraction is only the 3 digits that make up the millisecond portion of a duration. - * For example, the millisecond fraction of 1.234567890 seconds is 234. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t milliseconds - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_millisecond_fraction( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Extracts microsecond fraction from any datetime type and returns an int16_t - * cudf::column. - * - * A microsecond fraction is only the 3 digits that make up the microsecond portion of a duration. - * For example, the microsecond fraction of 1.234567890 seconds is 567. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t microseconds - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_microsecond_fraction( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - -/** - * @brief Extracts nanosecond fraction from any datetime type and returns an int16_t - * cudf::column. - * - * A nanosecond fraction is only the 3 digits that make up the nanosecond portion of a duration. - * For example, the nanosecond fraction of 1.234567890 seconds is 890. - * - * @deprecated Deprecated in 24.12, to be removed in 25.02 - * - * @param column cudf::column_view of the input datetime values - * @param stream CUDA stream used for device memory operations and kernel launches - * @param mr Device memory resource used to allocate device memory of the returned column - * - * @returns cudf::column of the extracted int16_t nanoseconds - * @throw cudf::logic_error if input column datatype is not TIMESTAMP - */ -[[deprecated]] std::unique_ptr extract_nanosecond_fraction( - cudf::column_view const& column, - rmm::cuda_stream_view stream = cudf::get_default_stream(), - rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); - /** * @brief Extracts the specified datetime component from any datetime type and * returns an int16_t cudf::column. diff --git a/cpp/include/cudf/detail/datetime.hpp b/cpp/include/cudf/detail/datetime.hpp index df3050d6494..2b01231deab 100644 --- a/cpp/include/cudf/detail/datetime.hpp +++ b/cpp/include/cudf/detail/datetime.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -25,96 +25,6 @@ namespace CUDF_EXPORT cudf { namespace datetime { namespace detail { -/** - * @copydoc cudf::extract_year(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_year(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - -/** - * @copydoc cudf::extract_month(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_month(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - -/** - * @copydoc cudf::extract_day(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_day(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - -/** - * @copydoc cudf::extract_weekday(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_weekday(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - -/** - * @copydoc cudf::extract_hour(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_hour(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - -/** - * @copydoc cudf::extract_minute(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_minute(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - -/** - * @copydoc cudf::extract_second(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_second(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - -/** - * @copydoc cudf::extract_millisecond_fraction(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_millisecond_fraction(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - -/** - * @copydoc cudf::extract_microsecond_fraction(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_microsecond_fraction(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - -/** - * @copydoc cudf::extract_nanosecond_fraction(cudf::column_view const&, rmm::cuda_stream_view, - * rmm::device_async_resource_ref) - * - */ -std::unique_ptr extract_nanosecond_fraction(cudf::column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr); - /** * @copydoc cudf::extract_datetime_component(cudf::column_view const&, datetime_component, * rmm::cuda_stream_view, rmm::device_async_resource_ref) diff --git a/cpp/src/datetime/datetime_ops.cu b/cpp/src/datetime/datetime_ops.cu index a497cedb3bc..62f702ac147 100644 --- a/cpp/src/datetime/datetime_ops.cu +++ b/cpp/src/datetime/datetime_ops.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -436,76 +436,6 @@ std::unique_ptr round_general(rounding_function round_kind, column.type(), dispatch_round{}, round_kind, component, column, stream, mr); } -std::unique_ptr extract_year(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::YEAR, stream, mr); -} - -std::unique_ptr extract_month(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::MONTH, stream, mr); -} - -std::unique_ptr extract_day(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::DAY, stream, mr); -} - -std::unique_ptr extract_weekday(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::WEEKDAY, stream, mr); -} - -std::unique_ptr extract_hour(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::HOUR, stream, mr); -} - -std::unique_ptr extract_minute(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::MINUTE, stream, mr); -} - -std::unique_ptr extract_second(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::SECOND, stream, mr); -} - -std::unique_ptr extract_millisecond_fraction(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::MILLISECOND, stream, mr); -} - -std::unique_ptr extract_microsecond_fraction(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::MICROSECOND, stream, mr); -} - -std::unique_ptr extract_nanosecond_fraction(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - return detail::extract_datetime_component(column, datetime_component::NANOSECOND, stream, mr); -} - std::unique_ptr last_day_of_month(column_view const& column, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) @@ -598,62 +528,6 @@ std::unique_ptr round_datetimes(column_view const& column, return detail::round_general(detail::rounding_function::ROUND, freq, column, stream, mr); } -std::unique_ptr extract_year(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_year(column, stream, mr); -} - -std::unique_ptr extract_month(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_month(column, stream, mr); -} - -std::unique_ptr extract_day(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_day(column, stream, mr); -} - -std::unique_ptr extract_weekday(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_weekday(column, stream, mr); -} - -std::unique_ptr extract_hour(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_hour(column, stream, mr); -} - -std::unique_ptr extract_minute(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_minute(column, stream, mr); -} - -std::unique_ptr extract_second(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_second(column, stream, mr); -} - std::unique_ptr extract_datetime_component(cudf::column_view const& column, datetime_component component, rmm::cuda_stream_view stream, @@ -663,30 +537,6 @@ std::unique_ptr extract_datetime_component(cudf::column_view const return detail::extract_datetime_component(column, component, stream, mr); } -std::unique_ptr extract_millisecond_fraction(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_millisecond_fraction(column, stream, mr); -} - -std::unique_ptr extract_microsecond_fraction(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_microsecond_fraction(column, stream, mr); -} - -std::unique_ptr extract_nanosecond_fraction(column_view const& column, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) -{ - CUDF_FUNC_RANGE(); - return detail::extract_nanosecond_fraction(column, stream, mr); -} - std::unique_ptr last_day_of_month(column_view const& column, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) diff --git a/python/pylibcudf/pylibcudf/datetime.pxd b/python/pylibcudf/pylibcudf/datetime.pxd index 335ef435f9b..ce295990d26 100644 --- a/python/pylibcudf/pylibcudf/datetime.pxd +++ b/python/pylibcudf/pylibcudf/datetime.pxd @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from pylibcudf.column cimport Column from pylibcudf.libcudf.datetime cimport datetime_component, rounding_frequency @@ -8,18 +8,6 @@ ctypedef fused ColumnOrScalar: Column Scalar -cpdef Column extract_millisecond_fraction( - Column input -) - -cpdef Column extract_microsecond_fraction( - Column input -) - -cpdef Column extract_nanosecond_fraction( - Column input -) - cpdef Column extract_datetime_component( Column input, datetime_component component diff --git a/python/pylibcudf/pylibcudf/datetime.pyi b/python/pylibcudf/pylibcudf/datetime.pyi index 6a3ae7953d9..8eedaeefe61 100644 --- a/python/pylibcudf/pylibcudf/datetime.pyi +++ b/python/pylibcudf/pylibcudf/datetime.pyi @@ -26,9 +26,6 @@ class RoundingFrequency(IntEnum): MICROSECOND = ... NANOSECOND = ... -def extract_millisecond_fraction(input: Column) -> Column: ... -def extract_microsecond_fraction(input: Column) -> Column: ... -def extract_nanosecond_fraction(input: Column) -> Column: ... def extract_datetime_component( input: Column, component: DatetimeComponent ) -> Column: ... diff --git a/python/pylibcudf/pylibcudf/datetime.pyx b/python/pylibcudf/pylibcudf/datetime.pyx index b100e3e22d0..15aee4c3e9e 100644 --- a/python/pylibcudf/pylibcudf/datetime.pyx +++ b/python/pylibcudf/pylibcudf/datetime.pyx @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from libcpp.memory cimport unique_ptr from libcpp.utility cimport move from pylibcudf.libcudf.column.column cimport column @@ -9,9 +9,6 @@ from pylibcudf.libcudf.datetime cimport ( day_of_year as cpp_day_of_year, days_in_month as cpp_days_in_month, extract_datetime_component as cpp_extract_datetime_component, - extract_microsecond_fraction as cpp_extract_microsecond_fraction, - extract_millisecond_fraction as cpp_extract_millisecond_fraction, - extract_nanosecond_fraction as cpp_extract_nanosecond_fraction, extract_quarter as cpp_extract_quarter, floor_datetimes as cpp_floor_datetimes, is_leap_year as cpp_is_leap_year, @@ -37,9 +34,6 @@ __all__ = [ "day_of_year", "days_in_month", "extract_datetime_component", - "extract_microsecond_fraction", - "extract_millisecond_fraction", - "extract_nanosecond_fraction", "extract_quarter", "floor_datetimes", "is_leap_year", @@ -47,78 +41,6 @@ __all__ = [ "round_datetimes", ] -cpdef Column extract_millisecond_fraction( - Column input -): - """ - Extract the millisecond from a datetime column. - - For details, see :cpp:func:`extract_millisecond_fraction`. - - Parameters - ---------- - input : Column - The column to extract the millisecond from. - - Returns - ------- - Column - Column with the extracted milliseconds. - """ - cdef unique_ptr[column] result - - with nogil: - result = cpp_extract_millisecond_fraction(input.view()) - return Column.from_libcudf(move(result)) - -cpdef Column extract_microsecond_fraction( - Column input -): - """ - Extract the microsecond fraction from a datetime column. - - For details, see :cpp:func:`extract_microsecond_fraction`. - - Parameters - ---------- - input : Column - The column to extract the microsecond fraction from. - - Returns - ------- - Column - Column with the extracted microsecond fractions. - """ - cdef unique_ptr[column] result - - with nogil: - result = cpp_extract_microsecond_fraction(input.view()) - return Column.from_libcudf(move(result)) - -cpdef Column extract_nanosecond_fraction( - Column input -): - """ - Extract the nanosecond fraction from a datetime column. - - For details, see :cpp:func:`extract_nanosecond_fraction`. - - Parameters - ---------- - input : Column - The column to extract the nanosecond fraction from. - - Returns - ------- - Column - Column with the extracted nanosecond fractions. - """ - cdef unique_ptr[column] result - - with nogil: - result = cpp_extract_nanosecond_fraction(input.view()) - return Column.from_libcudf(move(result)) - cpdef Column extract_datetime_component( Column input, datetime_component component diff --git a/python/pylibcudf/pylibcudf/libcudf/datetime.pxd b/python/pylibcudf/pylibcudf/libcudf/datetime.pxd index 049a1b06c2e..7dacab668b6 100644 --- a/python/pylibcudf/pylibcudf/libcudf/datetime.pxd +++ b/python/pylibcudf/pylibcudf/libcudf/datetime.pxd @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from libc.stdint cimport int32_t, uint8_t from libcpp.memory cimport unique_ptr @@ -21,36 +21,6 @@ cdef extern from "cudf/datetime.hpp" namespace "cudf::datetime" nogil: MICROSECOND NANOSECOND - cdef unique_ptr[column] extract_year( - const column_view& column - ) except +libcudf_exception_handler - cdef unique_ptr[column] extract_month( - const column_view& column - ) except +libcudf_exception_handler - cdef unique_ptr[column] extract_day( - const column_view& column - ) except +libcudf_exception_handler - cdef unique_ptr[column] extract_weekday( - const column_view& column - ) except +libcudf_exception_handler - cdef unique_ptr[column] extract_hour( - const column_view& column - ) except +libcudf_exception_handler - cdef unique_ptr[column] extract_minute( - const column_view& column - ) except +libcudf_exception_handler - cdef unique_ptr[column] extract_second( - const column_view& column - ) except +libcudf_exception_handler - cdef unique_ptr[column] extract_millisecond_fraction( - const column_view& column - ) except +libcudf_exception_handler - cdef unique_ptr[column] extract_microsecond_fraction( - const column_view& column - ) except +libcudf_exception_handler - cdef unique_ptr[column] extract_nanosecond_fraction( - const column_view& column - ) except +libcudf_exception_handler cdef unique_ptr[column] extract_datetime_component( const column_view& column, datetime_component component diff --git a/python/pylibcudf/pylibcudf/tests/test_datetime.py b/python/pylibcudf/pylibcudf/tests/test_datetime.py index f5f24ef28e2..6251a4bbb86 100644 --- a/python/pylibcudf/pylibcudf/tests/test_datetime.py +++ b/python/pylibcudf/pylibcudf/tests/test_datetime.py @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. import calendar import datetime @@ -77,26 +77,6 @@ def test_extract_datetime_component(datetime_column, component): assert_column_eq(expect, got) -@pytest.mark.parametrize( - "datetime_func", - [ - "extract_millisecond_fraction", - "extract_microsecond_fraction", - "extract_nanosecond_fraction", - ], -) -def test_datetime_extracting_functions(datetime_column, datetime_func): - pa_col = plc.interop.to_arrow(datetime_column) - got = getattr(plc.datetime, datetime_func)(datetime_column) - kwargs = {} - attr = datetime_func.split("_")[1] - if attr == "weekday": - kwargs = {"count_from_zero": False} - attr = "day_of_week" - expect = getattr(pc, attr)(pa_col, **kwargs).cast(pa.int16()) - assert_column_eq(expect, got) - - @pytest.mark.parametrize( "op", [ From d660873068bd9a54d9a78f6eabd3eaf53e0296b1 Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Wed, 19 Feb 2025 11:16:34 -0500 Subject: [PATCH 057/129] Refactor math_ops.cu dispatcher logic (#18006) Refactors the type-dispatcher logic and cleans up the code in `math_ops.cu` for unary operations. The 3 of the 4 dispatch functors had the same logic except for the supported types SFINAE clause. Also correcting the code for handling RINT properly created a 4th common functor. These have been refactored into a single functor and separated from the supported-types checks. The single functor now excepts the transform function as well as the supported-types expression. Also, the 2nd dispatcher call for dictionary was replaced with an if-statement to help simplify the code and minimize maintenance syncing up the supported-types clauses correctly. One side effect is that more ops are now supported appropriately with dictionary types. Referenced cleanup needed here: https://github.com/rapidsai/cudf/pull/17560#discussion_r1934160760 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Matthew Murray (https://github.com/Matt711) - Shruti Shivakumar (https://github.com/shrshi) - Vukasin Milovanovic (https://github.com/vuule) URL: https://github.com/rapidsai/cudf/pull/18006 --- cpp/src/unary/math_ops.cu | 323 +++++++++++++------------------------- 1 file changed, 112 insertions(+), 211 deletions(-) diff --git a/cpp/src/unary/math_ops.cu b/cpp/src/unary/math_ops.cu index 4e96f900bf3..aead6710082 100644 --- a/cpp/src/unary/math_ops.cu +++ b/cpp/src/unary/math_ops.cu @@ -27,9 +27,9 @@ #include +#include #include -#include #include namespace cudf { @@ -42,7 +42,7 @@ struct DeviceSin { template __device__ T operator()(T data) { - return std::sin(data); + return cuda::std::sin(data); } }; @@ -50,7 +50,7 @@ struct DeviceCos { template __device__ T operator()(T data) { - return std::cos(data); + return cuda::std::cos(data); } }; @@ -58,7 +58,7 @@ struct DeviceTan { template __device__ T operator()(T data) { - return std::tan(data); + return cuda::std::tan(data); } }; @@ -66,7 +66,7 @@ struct DeviceArcSin { template __device__ T operator()(T data) { - return std::asin(data); + return cuda::std::asin(data); } }; @@ -74,7 +74,7 @@ struct DeviceArcCos { template __device__ T operator()(T data) { - return std::acos(data); + return cuda::std::acos(data); } }; @@ -82,7 +82,7 @@ struct DeviceArcTan { template __device__ T operator()(T data) { - return std::atan(data); + return cuda::std::atan(data); } }; @@ -90,7 +90,7 @@ struct DeviceSinH { template __device__ T operator()(T data) { - return std::sinh(data); + return cuda::std::sinh(data); } }; @@ -98,7 +98,7 @@ struct DeviceCosH { template __device__ T operator()(T data) { - return std::cosh(data); + return cuda::std::cosh(data); } }; @@ -106,7 +106,7 @@ struct DeviceTanH { template __device__ T operator()(T data) { - return std::tanh(data); + return cuda::std::tanh(data); } }; @@ -114,7 +114,7 @@ struct DeviceArcSinH { template __device__ T operator()(T data) { - return std::asinh(data); + return cuda::std::asinh(data); } }; @@ -122,7 +122,7 @@ struct DeviceArcCosH { template __device__ T operator()(T data) { - return std::acosh(data); + return cuda::std::acosh(data); } }; @@ -130,7 +130,7 @@ struct DeviceArcTanH { template __device__ T operator()(T data) { - return std::atanh(data); + return cuda::std::atanh(data); } }; @@ -140,7 +140,7 @@ struct DeviceExp { template __device__ T operator()(T data) { - return std::exp(data); + return cuda::std::exp(data); } }; @@ -148,7 +148,7 @@ struct DeviceLog { template __device__ T operator()(T data) { - return std::log(data); + return cuda::std::log(data); } }; @@ -156,7 +156,7 @@ struct DeviceSqrt { template __device__ T operator()(T data) { - return std::sqrt(data); + return cuda::std::sqrt(data); } }; @@ -164,7 +164,7 @@ struct DeviceCbrt { template __device__ T operator()(T data) { - return std::cbrt(data); + return cuda::std::cbrt(data); } }; @@ -174,7 +174,7 @@ struct DeviceCeil { template __device__ T operator()(T data) { - return std::ceil(data); + return cuda::std::ceil(data); } }; @@ -182,7 +182,7 @@ struct DeviceFloor { template __device__ T operator()(T data) { - return std::floor(data); + return cuda::std::floor(data); } }; @@ -190,7 +190,7 @@ struct DeviceAbs { template std::enable_if_t, T> __device__ operator()(T data) { - return std::abs(data); + return cuda::std::abs(data); } template std::enable_if_t, T> __device__ operator()(T data) @@ -199,18 +199,13 @@ struct DeviceAbs { } }; -struct DeviceRInt { - template - std::enable_if_t, T> __device__ operator()(T data) - { - return std::rint(data); - } +// round float to int - // Dummy to handle other types, will never be executed +struct DeviceRInt { template - std::enable_if_t, T> __device__ operator()(T data) + __device__ T operator()(T data) { - return data; + return cuda::std::rint(data); } }; @@ -238,7 +233,7 @@ struct DeviceNot { struct DeviceNegate { template - T __device__ operator()(T data) + __device__ T operator()(T data) { return -data; } @@ -350,7 +345,6 @@ std::unique_ptr transform_fn(InputIterator begin, null_count, stream, mr); - if (size == 0) return output; auto output_view = output->mutable_view(); thrust::transform(rmm::exec_policy(stream), begin, end, output_view.begin(), UFN{}); @@ -358,6 +352,19 @@ std::unique_ptr transform_fn(InputIterator begin, return output; } +template +std::unique_ptr transform_fn(cudf::column_view const& input, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) +{ + return transform_fn(input.begin(), + input.end(), + detail::copy_bitmask(input, stream, mr), + input.null_count(), + stream, + mr); +} + template std::unique_ptr transform_fn(cudf::dictionary_column_view const& input, rmm::cuda_stream_view stream, @@ -377,136 +384,52 @@ std::unique_ptr transform_fn(cudf::dictionary_column_view const& i output->view(), dictionary::detail::get_indices_type_for_size(output->size()), stream, mr); } -template -struct MathOpDispatcher { - template >* = nullptr> - std::unique_ptr operator()(cudf::column_view const& input, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) - { - return transform_fn(input.begin(), - input.end(), - cudf::detail::copy_bitmask(input, stream, mr), - input.null_count(), - stream, - mr); - } - - struct dictionary_dispatch { - template >* = nullptr> - std::unique_ptr operator()(cudf::dictionary_column_view const& input, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) - { - return transform_fn(input, stream, mr); - } - - template - std::enable_if_t, std::unique_ptr> operator()(Args&&...) - { - CUDF_FAIL("dictionary keys must be numeric for this operation"); - } - }; - - template < - typename T, - std::enable_if_t and std::is_same_v>* = nullptr> - std::unique_ptr operator()(cudf::column_view const& input, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) - { - if (input.is_empty()) return empty_like(input); - auto dictionary_col = dictionary_column_view(input); - return type_dispatcher( - dictionary_col.keys().type(), dictionary_dispatch{}, dictionary_col, stream, mr); - } - - template - std::enable_if_t and !std::is_same_v, - std::unique_ptr> - operator()(Args&&...) - { - CUDF_FAIL("Unsupported data type for operation"); - } +template +struct ArithmeticOps { + static constexpr bool is_supported() { return std::is_arithmetic_v; } }; -template -struct NegateOpDispatcher { - template - static constexpr bool is_supported() - { - return std::is_signed_v || cudf::is_duration(); - } - - template ()>* = nullptr> - std::unique_ptr operator()(cudf::column_view const& input, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) - { - return transform_fn(input.begin(), - input.end(), - cudf::detail::copy_bitmask(input, stream, mr), - input.null_count(), - stream, - mr); - } - - template - std::enable_if_t(), std::unique_ptr> operator()(Args&&...) - { - CUDF_FAIL("Unsupported data type for negate operation"); - } +template +struct NegateOps { + static constexpr bool is_supported() { return std::is_signed_v || cudf::is_duration(); } }; -template -struct BitwiseOpDispatcher { - template >* = nullptr> - std::unique_ptr operator()(cudf::column_view const& input, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) - { - return transform_fn(input.begin(), - input.end(), - cudf::detail::copy_bitmask(input, stream, mr), - input.null_count(), - stream, - mr); - } - - struct dictionary_dispatch { - template >* = nullptr> - std::unique_ptr operator()(cudf::dictionary_column_view const& input, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) - { - return transform_fn(input, stream, mr); - } +template +struct BitWiseOps { + static constexpr bool is_supported() { return std::is_integral_v; } +}; - template - std::enable_if_t, std::unique_ptr> operator()(Args&&...) - { - CUDF_FAIL("dictionary keys type not supported for this operation"); - } - }; +template +struct FloatOnlyOps { + static constexpr bool is_supported() { return std::is_floating_point_v; } +}; - template and std::is_same_v>* = nullptr> +/** + * @brief Generic math-ops dispatcher + * + * Performs a transform on the input data using the operator defined by UFN. + * The Supported type determines which types are allowed by the operator. + * + * @tparam UFN The actual operator to perform on the input data + * @tparam Supported Contains the 'is_supported()' function + */ +template typename Supported> +struct MathOpDispatcher { + template ::is_supported()>* = nullptr> std::unique_ptr operator()(cudf::column_view const& input, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) { - if (input.is_empty()) return empty_like(input); - auto dictionary_col = dictionary_column_view(input); - return type_dispatcher( - dictionary_col.keys().type(), dictionary_dispatch{}, dictionary_col, stream, mr); + return (input.type().id() == type_id::DICTIONARY32) + ? transform_fn(cudf::dictionary_column_view(input), stream, mr) + : transform_fn(input, stream, mr); } template - std::enable_if_t and !std::is_same_v, - std::unique_ptr> - operator()(Args&&...) + std::enable_if_t::is_supported(), std::unique_ptr> operator()( + Args&&...) { - CUDF_FAIL("Unsupported datatype for operation"); + CUDF_FAIL("Unsupported data type for this operation"); } }; @@ -525,54 +448,26 @@ struct LogicalOpDispatcher { rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) { - return transform_fn(input.begin(), - input.end(), - cudf::detail::copy_bitmask(input, stream, mr), - input.null_count(), - - stream, - mr); - } - - struct dictionary_dispatch { - template ()>* = nullptr> - std::unique_ptr operator()(cudf::dictionary_column_view const& input, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) - { - auto dictionary_view = cudf::column_device_view::create(input.parent(), stream); + if (input.type().id() == type_id::DICTIONARY32) { + auto dictionary_view = cudf::column_device_view::create(input, stream); auto dictionary_itr = dictionary::detail::make_dictionary_iterator(*dictionary_view); return transform_fn(dictionary_itr, dictionary_itr + input.size(), - cudf::detail::copy_bitmask(input.parent(), stream, mr), + cudf::detail::copy_bitmask(input, stream, mr), input.null_count(), stream, mr); } - - template - std::enable_if_t(), std::unique_ptr> operator()(Args&&...) - { - CUDF_FAIL("dictionary keys type not supported for this operation"); - } - }; - - template () and std::is_same_v>* = nullptr> - std::unique_ptr operator()(cudf::column_view const& input, - rmm::cuda_stream_view stream, - rmm::device_async_resource_ref mr) - { - if (input.is_empty()) return make_empty_column(cudf::data_type{cudf::type_id::BOOL8}); - auto dictionary_col = dictionary_column_view(input); - return type_dispatcher( - dictionary_col.keys().type(), dictionary_dispatch{}, dictionary_col, stream, mr); + return transform_fn(input.begin(), + input.end(), + cudf::detail::copy_bitmask(input, stream, mr), + input.null_count(), + stream, + mr); } template - std::enable_if_t() and !std::is_same_v, - std::unique_ptr> - operator()(Args&&...) + std::enable_if_t(), std::unique_ptr> operator()(Args&&...) { CUDF_FAIL("Unsupported datatype for operation"); } @@ -614,79 +509,85 @@ std::unique_ptr unary_operation(cudf::column_view const& input, if (cudf::is_fixed_point(input.type())) return type_dispatcher(input.type(), detail::FixedPointOpDispatcher{}, input, op, stream, mr); + if (input.is_empty()) { + return op == cudf::unary_operator::NOT ? make_empty_column(type_id::BOOL8) : empty_like(input); + } + + // dispatch on the keys if dictionary saves a 2nd dispatch later + auto dispatch_type = input.type().id() == type_id::DICTIONARY32 + ? dictionary_column_view(input).keys().type() + : input.type(); + switch (op) { case cudf::unary_operator::SIN: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::COS: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::TAN: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::ARCSIN: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::ARCCOS: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::ARCTAN: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::SINH: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::COSH: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::TANH: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::ARCSINH: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::ARCCOSH: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::ARCTANH: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::EXP: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::LOG: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::SQRT: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::CBRT: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::CEIL: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::FLOOR: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::ABS: return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::RINT: - CUDF_EXPECTS( - (input.type().id() == type_id::FLOAT32) or (input.type().id() == type_id::FLOAT64), - "rint expects floating point values"); return cudf::type_dispatcher( - input.type(), detail::MathOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::BIT_INVERT: return cudf::type_dispatcher( - input.type(), detail::BitwiseOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); case cudf::unary_operator::NOT: return cudf::type_dispatcher( - input.type(), detail::LogicalOpDispatcher{}, input, stream, mr); + dispatch_type, detail::LogicalOpDispatcher{}, input, stream, mr); case cudf::unary_operator::NEGATE: return cudf::type_dispatcher( - input.type(), detail::NegateOpDispatcher{}, input, stream, mr); + dispatch_type, MathOpDispatcher{}, input, stream, mr); default: CUDF_FAIL("Undefined unary operation"); } } From c99f393b61a41893b02709ecdc166f7f2a1fbcb2 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Wed, 19 Feb 2025 13:31:45 -0500 Subject: [PATCH 058/129] Skip the failing connectorx polars tests (#18037) In #18015, we tried skipping the failing polars tests and applying the workaround mentioned in polars issue 21274. But pip is [unable to solve our test environment](https://github.com/rapidsai/cudf/actions/runs/13406947992/job/37463788766) in that case. This PR just skips the tests because we only need to do one or the other, not both. --- python/cudf_polars/cudf_polars/testing/plugin.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/python/cudf_polars/cudf_polars/testing/plugin.py b/python/cudf_polars/cudf_polars/testing/plugin.py index 0b52cf1c61c..e56d906833f 100644 --- a/python/cudf_polars/cudf_polars/testing/plugin.py +++ b/python/cudf_polars/cudf_polars/testing/plugin.py @@ -214,6 +214,10 @@ def pytest_configure(config: pytest.Config) -> None: "tests/unit/streaming/test_streaming_group_by.py::test_streaming_group_by_literal[1]": "May segfault w/the legacy streaming engine", # Fails in CI, but passes locally "tests/unit/streaming/test_streaming.py::test_streaming_streamable_functions": "RuntimeError: polars_python::sql::PySQLContext is unsendable, but is being dropped on another thread", + # TODO: Remove once when we support polars 1.23 + "tests/unit/io/database/test_read.py::test_read_database[uri: connectorx]": "ValueError: arrow2", + "tests/unit/io/database/test_read.py::test_read_database_cx_credentials[fakedb://123:456@account/database/schema?warehouse=warehouse&role=role]": "ValueError: arrow2", + "tests/unit/io/database/test_read.py::test_read_database_cx_credentials[fakedb://my#%us3r:p433w0rd@not_a_real_host:9999/database]": "ValueError: arrow2", } From e500794479c3b1a23c1a12c8425d9120424871f8 Mon Sep 17 00:00:00 2001 From: Matthew Murray Date: Wed, 19 Feb 2025 10:47:37 -0800 Subject: [PATCH 059/129] remove pip install --- ci/test_cudf_polars_polars_tests.sh | 2 -- 1 file changed, 2 deletions(-) diff --git a/ci/test_cudf_polars_polars_tests.sh b/ci/test_cudf_polars_polars_tests.sh index 909abbe9d1e..3466edacfc5 100755 --- a/ci/test_cudf_polars_polars_tests.sh +++ b/ci/test_cudf_polars_polars_tests.sh @@ -27,8 +27,6 @@ git clone https://github.com/pola-rs/polars.git --branch "${TAG}" --depth 1 # Install requirements for running polars tests rapids-logger "Install polars test requirements" rapids-pip-retry install -r polars/py-polars/requirements-dev.txt -r polars/py-polars/requirements-ci.txt -# TODO: Workaround until https://github.com/pola-rs/polars/issues/21274 is fixed. -rapids-pip-retry install connectorx==0.4.1 # shellcheck disable=SC2317 function set_exitcode() From 3117dc26b8466ac8e2c64574ab0b26cc621a44ff Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Wed, 19 Feb 2025 14:45:15 -0500 Subject: [PATCH 060/129] Bump polars version to <1.23 (#17986) The PR upgrades the Polars version to 1.22. Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - James Lamb (https://github.com/jameslamb) - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/17986 --- .../all_cuda-118_arch-x86_64.yaml | 2 +- .../all_cuda-128_arch-x86_64.yaml | 2 +- conda/recipes/cudf-polars/meta.yaml | 2 +- dependencies.yaml | 2 +- python/cudf_polars/cudf_polars/dsl/ir.py | 43 ++++++++++++++++--- .../cudf_polars/cudf_polars/dsl/translate.py | 28 +++++++++--- .../cudf_polars/cudf_polars/testing/plugin.py | 3 ++ python/cudf_polars/pyproject.toml | 2 +- python/cudf_polars/tests/test_mapfunction.py | 13 +++++- 9 files changed, 78 insertions(+), 19 deletions(-) diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index 09eb9949f1d..4ec6ef1883a 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -66,7 +66,7 @@ dependencies: - pandas - pandas>=2.0,<2.2.4dev0 - pandoc -- polars>=1.20,<1.22 +- polars>=1.20,<1.23 - pre-commit - ptxcompiler - pyarrow>=14.0.0,<20.0.0a0 diff --git a/conda/environments/all_cuda-128_arch-x86_64.yaml b/conda/environments/all_cuda-128_arch-x86_64.yaml index 56cef28ac61..dcf96a02a36 100644 --- a/conda/environments/all_cuda-128_arch-x86_64.yaml +++ b/conda/environments/all_cuda-128_arch-x86_64.yaml @@ -64,7 +64,7 @@ dependencies: - pandas - pandas>=2.0,<2.2.4dev0 - pandoc -- polars>=1.20,<1.22 +- polars>=1.20,<1.23 - pre-commit - pyarrow>=14.0.0,<20.0.0a0 - pydata-sphinx-theme>=0.15.4 diff --git a/conda/recipes/cudf-polars/meta.yaml b/conda/recipes/cudf-polars/meta.yaml index fb7ab9332d8..1d36ab2a3e4 100644 --- a/conda/recipes/cudf-polars/meta.yaml +++ b/conda/recipes/cudf-polars/meta.yaml @@ -43,7 +43,7 @@ requirements: run: - python - pylibcudf ={{ version }} - - polars >=1.20,<1.22 + - polars >=1.20,<1.23 - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} test: diff --git a/dependencies.yaml b/dependencies.yaml index 7188e10b058..c8893fc8b49 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -803,7 +803,7 @@ dependencies: common: - output_types: [conda, requirements, pyproject] packages: - - polars>=1.20,<1.22 + - polars>=1.20,<1.23 run_cudf_polars_experimental: common: - output_types: [conda, requirements, pyproject] diff --git a/python/cudf_polars/cudf_polars/dsl/ir.py b/python/cudf_polars/cudf_polars/dsl/ir.py index 8f12a4a7570..603f51e9d40 100644 --- a/python/cudf_polars/cudf_polars/dsl/ir.py +++ b/python/cudf_polars/cudf_polars/dsl/ir.py @@ -1650,6 +1650,16 @@ def do_evaluate(cls, schema: Schema, df: DataFrame) -> DataFrame: return DataFrame(columns) +class MergeSorted(IR): + """Merge sorted operation.""" + + def __init__(self, schema: Schema, left: IR, right: IR, key: str): + # libcudf merge is not stable wrt order of inputs, since + # it uses a priority queue to manage the tables it produces. + # See: https://github.com/rapidsai/cudf/issues/16010 + raise NotImplementedError("MergeSorted not yet implemented") + + class MapFunction(IR): """Apply some function to a dataframe.""" @@ -1663,13 +1673,10 @@ class MapFunction(IR): _NAMES: ClassVar[frozenset[str]] = frozenset( [ "rechunk", - # libcudf merge is not stable wrt order of inputs, since - # it uses a priority queue to manage the tables it produces. - # See: https://github.com/rapidsai/cudf/issues/16010 - # "merge_sorted", "rename", "explode", "unpivot", + "row_index", ] ) @@ -1678,8 +1685,12 @@ def __init__(self, schema: Schema, name: str, options: Any, df: IR): self.name = name self.options = options self.children = (df,) - if self.name not in MapFunction._NAMES: - raise NotImplementedError(f"Unhandled map function {self.name}") + if ( + self.name not in MapFunction._NAMES + ): # pragma: no cover; need more polars rust functions + raise NotImplementedError( + f"Unhandled map function {self.name}" + ) # pragma: no cover if self.name == "explode": (to_explode,) = self.options if len(to_explode) > 1: @@ -1716,6 +1727,9 @@ def __init__(self, schema: Schema, name: str, options: Any, df: IR): variable_name, value_name, ) + elif self.name == "row_index": + col_name, offset = options + self.options = (col_name, offset) self._non_child_args = (schema, name, self.options) @classmethod @@ -1781,6 +1795,23 @@ def do_evaluate( Column(value_column, name=value_name), ] ) + elif name == "row_index": + col_name, offset = options + dtype = schema[col_name] + step = plc.interop.from_arrow( + pa.scalar(1, type=plc.interop.to_arrow(dtype)) + ) + init = plc.interop.from_arrow( + pa.scalar(offset, type=plc.interop.to_arrow(dtype)) + ) + index_col = Column( + plc.filling.sequence(df.num_rows, init, step), + is_sorted=plc.types.Sorted.YES, + order=plc.types.Order.ASCENDING, + null_order=plc.types.NullOrder.AFTER, + name=col_name, + ) + return DataFrame([index_col, *df.columns]) else: raise AssertionError("Should never be reached") # pragma: no cover diff --git a/python/cudf_polars/cudf_polars/dsl/translate.py b/python/cudf_polars/cudf_polars/dsl/translate.py index 4ed36e463f3..22f97f2bf52 100644 --- a/python/cudf_polars/cudf_polars/dsl/translate.py +++ b/python/cudf_polars/cudf_polars/dsl/translate.py @@ -84,7 +84,7 @@ def translate_ir(self, *, n: int | None = None) -> ir.IR: # IR is versioned with major.minor, minor is bumped for backwards # compatible changes (e.g. adding new nodes), major is bumped for # incompatible changes (e.g. renaming nodes). - if (version := self.visitor.version()) >= (5, 1): + if (version := self.visitor.version()) >= (6, 1): e = NotImplementedError( f"No support for polars IR {version=}" ) # pragma: no cover; no such version for now. @@ -299,7 +299,7 @@ def _( # Join key dtypes are dependent on the schema of the left and # right inputs, so these must be translated with the relevant # input active. - def adjust_literal_dtype(literal: expr.Literal) -> expr.Literal: + def adjust_literal_dtype(literal: expr.Literal) -> expr.Literal: # pragma: no cover if literal.dtype.id() == plc.types.TypeId.INT32: plc_int64 = plc.types.DataType(plc.types.TypeId.INT64) return expr.Literal( @@ -308,7 +308,7 @@ def adjust_literal_dtype(literal: expr.Literal) -> expr.Literal: ) return literal - def maybe_adjust_binop(e) -> expr.Expr: + def maybe_adjust_binop(e) -> expr.Expr: # pragma: no cover if isinstance(e.value, expr.BinOp): left, right = e.value.children if isinstance(left, expr.Col) and isinstance(right, expr.Literal): @@ -323,10 +323,10 @@ def translate_expr_and_maybe_fix_binop_args(translator, exprs): ] with set_node(translator.visitor, node.input_left): + # TODO: There's bug in the polars type coercion phase. + # Use translate_named_expr directly once our minimum + # supported polars version is 1.22 inp_left = translator.translate_ir(n=None) - # TODO: There's bug in the polars type coercion phase. Use - # translate_named_expr directly once it is resolved. - # Tracking issue: https://github.com/pola-rs/polars/issues/20935 left_on = translate_expr_and_maybe_fix_binop_args(translator, node.left_on) with set_node(translator.visitor, node.input_right): inp_right = translator.translate_ir(n=None) @@ -463,6 +463,21 @@ def _( return ir.Projection(schema, translator.translate_ir(n=node.input)) +@_translate_ir.register +def _( + node: pl_ir.MergeSorted, translator: Translator, schema: dict[str, plc.DataType] +) -> ir.IR: + inp_left = translator.translate_ir(n=node.input_left) + inp_right = translator.translate_ir(n=node.input_right) + key = node.key + return ir.MergeSorted( + schema, + inp_left, + inp_right, + key, + ) + + @_translate_ir.register def _( node: pl_ir.MapFunction, translator: Translator, schema: dict[str, plc.DataType] @@ -472,7 +487,6 @@ def _( schema, name, options, - # TODO: merge_sorted breaks this pattern translator.translate_ir(n=node.input), ) diff --git a/python/cudf_polars/cudf_polars/testing/plugin.py b/python/cudf_polars/cudf_polars/testing/plugin.py index 48629af920d..cf1bfbe8a69 100644 --- a/python/cudf_polars/cudf_polars/testing/plugin.py +++ b/python/cudf_polars/cudf_polars/testing/plugin.py @@ -193,6 +193,9 @@ def pytest_configure(config: pytest.Config) -> None: "tests/unit/test_cse.py::test_cse_predicate_self_join": "Debug output on stderr doesn't match", "tests/unit/test_empty.py::test_empty_9137": "Mismatching dtypes, needs cudf#15852", "tests/unit/test_errors.py::test_error_on_empty_group_by": "Incorrect exception raised", + "tests/unit/io/test_multiscan.py::test_include_file_paths[scan_parquet-write_parquet]": "Need to expose include_file_paths xref: cudf#18012", + "tests/unit/io/test_multiscan.py::test_include_file_paths[scan_csv-write_csv]": "Need to expose include_file_paths xref: cudf#18012", + "tests/unit/streaming/test_streaming_io.py::test_parquet_eq_statistics[False]": "Debug output on stderr doesn't match", # Maybe flaky, order-dependent? "tests/unit/test_projections.py::test_schema_full_outer_join_projection_pd_13287": "Order-specific result check, query is correct but in different order", "tests/unit/test_queries.py::test_group_by_agg_equals_zero_3535": "libcudf sums all nulls to null, not zero", diff --git a/python/cudf_polars/pyproject.toml b/python/cudf_polars/pyproject.toml index 805d7925bb4..872c08a66f9 100644 --- a/python/cudf_polars/pyproject.toml +++ b/python/cudf_polars/pyproject.toml @@ -19,7 +19,7 @@ authors = [ license = { text = "Apache 2.0" } requires-python = ">=3.10" dependencies = [ - "polars>=1.20,<1.22", + "polars>=1.20,<1.23", "pylibcudf==25.4.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. classifiers = [ diff --git a/python/cudf_polars/tests/test_mapfunction.py b/python/cudf_polars/tests/test_mapfunction.py index 63aa1c573a9..7a9f4a56545 100644 --- a/python/cudf_polars/tests/test_mapfunction.py +++ b/python/cudf_polars/tests/test_mapfunction.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 from __future__ import annotations @@ -93,3 +93,14 @@ def test_unpivot_defaults(): ) q = df.unpivot(index="d") assert_gpu_result_equal(q) + + +def test_with_row_index_defaults(): + lf = pl.LazyFrame( + { + "a": [1, 3, 5], + "b": [2, 4, 6], + } + ) + q = lf.with_row_index() + assert_gpu_result_equal(q) From abffae8fa2bd43d3285d0ec1f684cbad9582dc9d Mon Sep 17 00:00:00 2001 From: GALI PREM SAGAR Date: Wed, 19 Feb 2025 21:09:36 -0600 Subject: [PATCH 061/129] Prevent setting custom attributes to `ColumnMethods` (#18005) Fixes: #17750 This PR disallows setting custom attributes to `ColumnMethods` Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/18005 --- python/cudf/cudf/core/column/methods.py | 8 +++++++- python/cudf/cudf/tests/test_list.py | 7 +++++++ python/cudf/cudf/tests/test_string.py | 12 ++++++++++++ 3 files changed, 26 insertions(+), 1 deletion(-) diff --git a/python/cudf/cudf/core/column/methods.py b/python/cudf/cudf/core/column/methods.py index a91c080fe21..b42e4419d72 100644 --- a/python/cudf/cudf/core/column/methods.py +++ b/python/cudf/cudf/core/column/methods.py @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from __future__ import annotations @@ -93,3 +93,9 @@ def _return_or_inplace( return cudf.Index._from_column(new_col, name=self._parent.name) else: return self._parent._mimic_inplace(new_col, inplace=False) + + def __setattr__(self, key, value): + if key in {"_parent", "_column"}: + super().__setattr__(key, value) + else: + raise AttributeError(f"You cannot add any new attribute '{key}'") diff --git a/python/cudf/cudf/tests/test_list.py b/python/cudf/cudf/tests/test_list.py index 3ffbd5ff2a8..3de733f1de2 100644 --- a/python/cudf/cudf/tests/test_list.py +++ b/python/cudf/cudf/tests/test_list.py @@ -956,6 +956,13 @@ def test_empty_nested_list_uninitialized_offsets_memory_usage(): assert ser.memory_usage() == 8 +def test_list_methods_setattr(): + ser = cudf.Series([["a", "b", "c"], ["d", "e", "f"]]) + + with pytest.raises(AttributeError): + ser.list.a = "b" + + def test_dataframe_list_round_trip(): data = [{"text": "hello", "list_col": np.asarray([1, 2], dtype="uint32")}] cudf_arrow = cudf.DataFrame(data).to_arrow() diff --git a/python/cudf/cudf/tests/test_string.py b/python/cudf/cudf/tests/test_string.py index 809fedfde7b..164fcb06624 100644 --- a/python/cudf/cudf/tests/test_string.py +++ b/python/cudf/cudf/tests/test_string.py @@ -3575,3 +3575,15 @@ def test_replace_invalid_scalar_repl(): ser = cudf.Series(["1"]) with pytest.raises(TypeError): ser.str.replace("1", 2) + + +def test_string_methods_setattr(): + ser = cudf.Series(["ab", "cd", "ef"]) + pser = ser.to_pandas() + + assert_exceptions_equal( + lfunc=ser.str.__setattr__, + rfunc=pser.str.__setattr__, + lfunc_args_and_kwargs=(("a", "b"),), + rfunc_args_and_kwargs=(("a", "b"),), + ) From 3c06da355e22162d167912a093b39c465cf4057a Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Wed, 19 Feb 2025 22:46:45 -0500 Subject: [PATCH 062/129] Expose `num_rows_per_source` (IO metadata) to pylibcudf (#18049) Closes #18048 Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/18049 --- python/pylibcudf/pylibcudf/io/types.pyi | 2 ++ python/pylibcudf/pylibcudf/io/types.pyx | 10 ++++++- .../pylibcudf/tests/io/test_types.py | 26 ++++++++++++++++++- 3 files changed, 36 insertions(+), 2 deletions(-) diff --git a/python/pylibcudf/pylibcudf/io/types.pyi b/python/pylibcudf/pylibcudf/io/types.pyi index 63fa9d1ff79..1463f4d0073 100644 --- a/python/pylibcudf/pylibcudf/io/types.pyi +++ b/python/pylibcudf/pylibcudf/io/types.pyi @@ -101,6 +101,8 @@ class TableWithMetadata: def child_names(self) -> ChildNameSpec: ... @property def per_file_user_data(self) -> list[Mapping[str, str]]: ... + @property + def num_rows_per_source(self) -> list[int]: ... class SourceInfo: def __init__( diff --git a/python/pylibcudf/pylibcudf/io/types.pyx b/python/pylibcudf/pylibcudf/io/types.pyx index 458595ca0e0..83330cf14ff 100644 --- a/python/pylibcudf/pylibcudf/io/types.pyx +++ b/python/pylibcudf/pylibcudf/io/types.pyx @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from cpython.buffer cimport PyBUF_READ from cpython.memoryview cimport PyMemoryView_FromMemory @@ -401,6 +401,14 @@ cdef class TableWithMetadata: """ return self.metadata.per_file_user_data + @property + def num_rows_per_source(self): + """ + Returns a list containing the number + of rows for each file being read in. + """ + return self.metadata.num_rows_per_source + cdef class SourceInfo: """A class containing details on a source to read from. diff --git a/python/pylibcudf/pylibcudf/tests/io/test_types.py b/python/pylibcudf/pylibcudf/tests/io/test_types.py index a7642556bf2..b14e7770e7b 100644 --- a/python/pylibcudf/pylibcudf/tests/io/test_types.py +++ b/python/pylibcudf/pylibcudf/tests/io/test_types.py @@ -1,13 +1,28 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. import gc import weakref import pyarrow as pa +import pytest import pylibcudf as plc +@pytest.fixture +def parquet_data(tmp_path): + tbl1 = pa.Table.from_pydict({"a": [3, 1, 4], "b": [1, 5, 9]}) + tbl2 = pa.Table.from_pydict({"a": [1, 6], "b": [1, 8]}) + + path1 = tmp_path / "tbl1.parquet" + path2 = tmp_path / "tbl2.parquet" + + pa.parquet.write_table(tbl1, path1) + pa.parquet.write_table(tbl2, path2) + + return [path1, path2] + + def test_gc_with_table_and_column_input_metadata(): class Foo(plc.io.types.TableInputMetadata): def __del__(self): @@ -26,3 +41,12 @@ def __del__(self): gc.collect() assert weak_tbl_meta() is None + + +def test_num_rows_per_resource(parquet_data): + source = plc.io.SourceInfo(parquet_data) + options = plc.io.parquet.ParquetReaderOptions.builder(source).build() + assert plc.io.parquet.read_parquet(options).num_rows_per_source == [3, 2] + + +# TODO: Test more IO types From eb5c309d24a9267656bb33d93ff90e4a2b12af89 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Wed, 19 Feb 2025 22:03:02 -0800 Subject: [PATCH 063/129] Pass more dtype objects to `astype` calls (#18044) Broken off from https://github.com/rapidsai/cudf/pull/17978 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Matthew Murray (https://github.com/Matt711) URL: https://github.com/rapidsai/cudf/pull/18044 --- python/cudf/cudf/core/column/categorical.py | 14 ++++-------- python/cudf/cudf/core/column/column.py | 2 +- python/cudf/cudf/core/dtypes.py | 2 +- python/cudf/cudf/core/index.py | 2 +- python/cudf/cudf/core/indexed_frame.py | 2 +- python/cudf/cudf/core/join/_join_helpers.py | 5 +++-- python/cudf/cudf/core/tools/datetimes.py | 4 ++-- python/cudf/cudf/tests/test_dataframe.py | 24 +++++++++++++++------ 8 files changed, 30 insertions(+), 25 deletions(-) diff --git a/python/cudf/cudf/core/column/categorical.py b/python/cudf/cudf/core/column/categorical.py index a789d5d5ab1..a57ff9a7817 100644 --- a/python/cudf/cudf/core/column/categorical.py +++ b/python/cudf/cudf/core/column/categorical.py @@ -811,21 +811,15 @@ def to_pandas( def to_arrow(self) -> pa.Array: """Convert to PyArrow Array.""" - # arrow doesn't support unsigned codes + # pyarrow.Table doesn't support unsigned codes signed_type = ( min_signed_type(self.codes.max()) if self.codes.size > 0 - else np.int8 + else np.dtype(np.int8) ) - codes = self.codes.astype(signed_type) - categories = self.categories - - out_indices = codes.to_arrow() - out_dictionary = categories.to_arrow() - return pa.DictionaryArray.from_arrays( - out_indices, - out_dictionary, + self.codes.astype(signed_type).to_arrow(), + self.categories.to_arrow(), ordered=self.ordered, ) diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index d281076690a..06dc4058115 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -1629,7 +1629,7 @@ def astype(self, dtype: Dtype, copy: bool = False) -> ColumnBase: elif isinstance(dtype, IntervalDtype): result = self.as_interval_column(dtype) elif isinstance(dtype, (ListDtype, StructDtype)): - if not self.dtype == dtype: + if self.dtype != dtype: raise NotImplementedError( f"Casting {self.dtype} columns not currently supported" ) diff --git a/python/cudf/cudf/core/dtypes.py b/python/cudf/cudf/core/dtypes.py index 983950580d0..12a9cce9f1c 100644 --- a/python/cudf/cudf/core/dtypes.py +++ b/python/cudf/cudf/core/dtypes.py @@ -262,7 +262,7 @@ def _init_categories( getattr(categories, "dtype", None), (cudf.IntervalDtype, pd.IntervalDtype), ): - dtype = "object" # type: Any + dtype = CUDF_STRING_DTYPE else: dtype = None diff --git a/python/cudf/cudf/core/index.py b/python/cudf/cudf/core/index.py index 8ce8dfd2198..8587bff2e32 100644 --- a/python/cudf/cudf/core/index.py +++ b/python/cudf/cudf/core/index.py @@ -3135,7 +3135,7 @@ def __init__( data = column.as_column(data) else: data = column.as_column( - data, dtype="category" if dtype is None else dtype + data, dtype=cudf.CategoricalDtype() if dtype is None else dtype ) # dtype has already been taken care dtype = None diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index ac4303394f7..9c48b31a309 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -6517,7 +6517,7 @@ def convert_dtypes( for col in self._columns: if col.dtype.kind == "f": col = col.fillna(0) - as_int = col.astype("int64") + as_int = col.astype(np.dtype(np.int64)) if cp.allclose(col, as_int): cols.append(as_int) continue diff --git a/python/cudf/cudf/core/join/_join_helpers.py b/python/cudf/cudf/core/join/_join_helpers.py index 854c44ff1a1..c329bf11d97 100644 --- a/python/cudf/cudf/core/join/_join_helpers.py +++ b/python/cudf/cudf/core/join/_join_helpers.py @@ -1,4 +1,4 @@ -# Copyright (c) 2021-2024, NVIDIA CORPORATION. +# Copyright (c) 2021-2025, NVIDIA CORPORATION. from __future__ import annotations @@ -114,7 +114,8 @@ def _match_join_keys( if how == "left" and rcol.fillna(0).can_cast_safely(ltype): return lcol, rcol.astype(ltype) - + elif common_type is None: + common_type = np.dtype(np.float64) return lcol.astype(common_type), rcol.astype(common_type) diff --git a/python/cudf/cudf/core/tools/datetimes.py b/python/cudf/cudf/core/tools/datetimes.py index 546abfc4d3d..4478be2fd04 100644 --- a/python/cudf/cudf/core/tools/datetimes.py +++ b/python/cudf/cudf/core/tools/datetimes.py @@ -369,9 +369,9 @@ def _process_col( elif col.dtype.kind == "O": if unit not in (None, "ns") or col.null_count == len(col): try: - col = col.astype(dtype="int64") + col = col.astype(np.dtype(np.int64)) except ValueError: - col = col.astype(dtype="float64") + col = col.astype(np.dtype(np.float64)) return _process_col( col=col, unit=unit, diff --git a/python/cudf/cudf/tests/test_dataframe.py b/python/cudf/cudf/tests/test_dataframe.py index 05bc221bf9d..15c11db5a84 100644 --- a/python/cudf/cudf/tests/test_dataframe.py +++ b/python/cudf/cudf/tests/test_dataframe.py @@ -4343,21 +4343,27 @@ def test_as_column_types(): assert_eq(pds, gds) - col = column.as_column(cudf.Series([], dtype="float64"), dtype="float32") + col = column.as_column( + cudf.Series([], dtype="float64"), dtype=np.dtype(np.float32) + ) assert_eq(col.dtype, np.dtype("float32")) gds = cudf.Series._from_column(col) pds = pd.Series(pd.Series([], dtype="float32")) assert_eq(pds, gds) - col = column.as_column(cudf.Series([], dtype="float64"), dtype="str") + col = column.as_column( + cudf.Series([], dtype="float64"), dtype=cudf.dtype("str") + ) assert_eq(col.dtype, np.dtype("object")) gds = cudf.Series._from_column(col) pds = pd.Series(pd.Series([], dtype="str")) assert_eq(pds, gds) - col = column.as_column(cudf.Series([], dtype="float64"), dtype="object") + col = column.as_column( + cudf.Series([], dtype="float64"), dtype=cudf.dtype("str") + ) assert_eq(col.dtype, np.dtype("object")) gds = cudf.Series._from_column(col) pds = pd.Series(pd.Series([], dtype="object")) @@ -4366,7 +4372,7 @@ def test_as_column_types(): pds = pd.Series(np.array([1, 2, 3]), dtype="float32") gds = cudf.Series._from_column( - column.as_column(np.array([1, 2, 3]), dtype="float32") + column.as_column(np.array([1, 2, 3]), dtype=np.dtype(np.float32)) ) assert_eq(pds, gds) @@ -4389,14 +4395,18 @@ def test_as_column_types(): pds = pd.Series([1.2, 18.0, 9.0], dtype="float32") gds = cudf.Series._from_column( - column.as_column(cudf.Series([1.2, 18.0, 9.0]), dtype="float32") + column.as_column( + cudf.Series([1.2, 18.0, 9.0]), dtype=np.dtype(np.float32) + ) ) assert_eq(pds, gds) pds = pd.Series([1.2, 18.0, 9.0], dtype="str") gds = cudf.Series._from_column( - column.as_column(cudf.Series([1.2, 18.0, 9.0]), dtype="str") + column.as_column( + cudf.Series([1.2, 18.0, 9.0]), dtype=cudf.dtype("str") + ) ) assert_eq(pds, gds) @@ -5228,7 +5238,7 @@ def test_empty_df_astype(dtype): ) def test_series_astype_error_handling(errors): sr = cudf.Series(["random", "words"]) - got = sr.astype("datetime64", errors=errors) + got = sr.astype("datetime64[ns]", errors=errors) assert_eq(sr, got) From da5d0a17ebfeb1e12a1a3efc4d64d96975b82378 Mon Sep 17 00:00:00 2001 From: nvdbaranec <56695930+nvdbaranec@users.noreply.github.com> Date: Thu, 20 Feb 2025 10:53:54 -0600 Subject: [PATCH 064/129] Fix 'Unexpected short subpass' exception in parquet chunked reader. (#18019) Fixes https://github.com/rapidsai/cudf/issues/18043 An incorrect computation in the subpass generation code would come to the conclusion that there weren't enough rows to decode for list columns under certain circumstances. This PR fixes the issue and I did a little bit of variable naming cleanup around the area. Ultimately the true source of the bug was poorly named variables causing them to be used incorrectly. Edit: I've disabled various checks in the chunked reader tests that expect specific chunk counts being returned from chunking operations. Changes to decompression temporary memory usage can make this unreliable. We will need a smarter solution down the road. --- cpp/src/io/parquet/reader_impl_preprocess.cu | 33 ++-- cpp/tests/io/parquet_chunked_reader_test.cu | 152 +++++++++---------- 2 files changed, 98 insertions(+), 87 deletions(-) diff --git a/cpp/src/io/parquet/reader_impl_preprocess.cu b/cpp/src/io/parquet/reader_impl_preprocess.cu index b6134947b0c..e1e9bac5a07 100644 --- a/cpp/src/io/parquet/reader_impl_preprocess.cu +++ b/cpp/src/io/parquet/reader_impl_preprocess.cu @@ -1463,7 +1463,7 @@ void reader::impl::preprocess_subpass_pages(read_mode mode, size_t chunk_read_li page_input, chunk_row_output_iter{pass.pages.device_ptr()}); - // copy chunk row into the subpass pages + // copy chunk_row into the subpass pages // only need to do this if we are not processing the whole pass in one subpass if (!subpass.single_subpass) { thrust::for_each(rmm::exec_policy_nosync(_stream), @@ -1481,31 +1481,42 @@ void reader::impl::preprocess_subpass_pages(read_mode mode, size_t chunk_read_li // able to decode for this pass. we will have selected a set of pages for each column in the // row group, but not every page will have the same number of rows. so, we can only read as many // rows as the smallest batch (by column) we have decompressed. - size_t page_index = 0; - size_t max_row = std::numeric_limits::max(); + size_t first_page_index = 0; + size_t max_row = std::numeric_limits::max(); auto const last_pass_row = _file_itm_data.input_pass_start_row_count[_file_itm_data._current_input_pass + 1]; + // for each column for (size_t idx = 0; idx < subpass.column_page_count.size(); idx++) { - auto const& last_page = subpass.pages[page_index + (subpass.column_page_count[idx] - 1)]; - auto const& chunk = pass.chunks[last_page.chunk_idx]; + // compute max row for this column in the subpass + auto const& last_page = subpass.pages[first_page_index + (subpass.column_page_count[idx] - 1)]; + auto const& last_chunk = pass.chunks[last_page.chunk_idx]; + auto max_col_row = static_cast(last_chunk.start_row) + + static_cast(last_page.chunk_row) + + static_cast(last_page.num_rows); - size_t max_col_row = - static_cast(chunk.start_row + last_page.chunk_row + last_page.num_rows); // special case. list rows can span page boundaries, but we can't tell if that is happening // here because we have not yet decoded the pages. the very last row starting in the page may // not terminate in the page. to handle this, only decode up to the second to last row in the // subpass since we know that will safely completed. - bool const is_list = chunk.max_level[level_type::REPETITION] > 0; + bool const is_list = last_chunk.max_level[level_type::REPETITION] > 0; + // corner case: only decode up to the second-to-last row, except if this is the last page in the + // entire pass. this handles the case where we only have 1 chunk, 1 page, and potentially even + // just 1 row. if (is_list && max_col_row < last_pass_row) { - auto const& first_page = subpass.pages[page_index]; - size_t const min_col_row = static_cast(chunk.start_row + first_page.chunk_row); + // compute min row for this column in the subpass + auto const& first_page = subpass.pages[first_page_index]; + auto const& first_chunk = pass.chunks[first_page.chunk_idx]; + auto const min_col_row = + static_cast(first_chunk.start_row) + static_cast(first_page.chunk_row); + + // must have at least 2 rows in the subpass. CUDF_EXPECTS((max_col_row - min_col_row) > 1, "Unexpected short subpass"); max_col_row--; } max_row = min(max_row, max_col_row); - page_index += subpass.column_page_count[idx]; + first_page_index += subpass.column_page_count[idx]; } subpass.skip_rows = pass.skip_rows + pass.processed_rows; auto const pass_end = pass.skip_rows + pass.num_rows; diff --git a/cpp/tests/io/parquet_chunked_reader_test.cu b/cpp/tests/io/parquet_chunked_reader_test.cu index 369376b6c95..04b479d719b 100644 --- a/cpp/tests/io/parquet_chunked_reader_test.cu +++ b/cpp/tests/io/parquet_chunked_reader_test.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -189,7 +189,7 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadNoData) auto const [expected, filepath] = write_file(input_columns, "chunked_read_empty", false, false); auto const [result, num_chunks] = chunked_read(filepath, 1'000); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); EXPECT_EQ(result->num_rows(), 0); EXPECT_EQ(result->num_columns(), 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); @@ -211,28 +211,28 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadSimpleData) { auto const [expected, filepath] = generate_input(false, false); auto const [result, num_chunks] = chunked_read(filepath, 240'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } { auto const [expected, filepath] = generate_input(false, true); auto const [result, num_chunks] = chunked_read(filepath, 240'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } { auto const [expected, filepath] = generate_input(true, false); auto const [result, num_chunks] = chunked_read(filepath, 240'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } { auto const [expected, filepath] = generate_input(true, true); auto const [result, num_chunks] = chunked_read(filepath, 240'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } } @@ -261,7 +261,7 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadBoundaryCases) // Test with a very small limit: 1 byte { auto const [result, num_chunks] = chunked_read(filepath, 1); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } @@ -275,49 +275,49 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadBoundaryCases) // Test with a limit slightly less than one page of data { auto const [result, num_chunks] = chunked_read(filepath, 79'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // Test with a limit exactly the size one page of data { auto const [result, num_chunks] = chunked_read(filepath, 80'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // Test with a limit slightly more the size one page of data { auto const [result, num_chunks] = chunked_read(filepath, 81'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // Test with a limit slightly less than two pages of data { auto const [result, num_chunks] = chunked_read(filepath, 159'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // Test with a limit exactly the size of two pages of data minus one byte { auto const [result, num_chunks] = chunked_read(filepath, 159'999); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // Test with a limit exactly the size of two pages of data { auto const [result, num_chunks] = chunked_read(filepath, 160'000); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // Test with a limit slightly more the size two pages of data { auto const [result, num_chunks] = chunked_read(filepath, 161'000); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } } @@ -416,22 +416,22 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithString) // Test with a very large limit { auto const [result, num_chunks] = chunked_read(filepath_no_null, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null_delta, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null_delta, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls_delta, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls_delta, *result); } @@ -439,43 +439,43 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithString) { auto const [result, num_chunks] = chunked_read(filepath_no_null, 500'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 500'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null_delta, 500'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null_delta, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls_delta, 500'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls_delta, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null, 1'000'000); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 1'000'000); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null_delta, 1'000'000); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null_delta, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls_delta, 1'000'000); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls_delta, *result); } } @@ -515,7 +515,7 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithStringPrecise) // each 1 page in size { auto const [result, num_chunks] = chunked_read(filepath_no_null, 260'007); - EXPECT_EQ(num_chunks, 3); + // EXPECT_EQ(num_chunks, 3); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } @@ -523,7 +523,7 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithStringPrecise) // pages 0-1 and page 2 { auto const [result, num_chunks] = chunked_read(filepath_no_null, 260'008); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } } @@ -567,31 +567,31 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithStructs) } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 0); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } // Test with a very small limit: 1 byte { auto const [result, num_chunks] = chunked_read(filepath_no_null, 1); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 1); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } // Test with a very large limit { auto const [result, num_chunks] = chunked_read(filepath_no_null, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } @@ -599,12 +599,12 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithStructs) { auto const [result, num_chunks] = chunked_read(filepath_no_null, 500'000); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 500'000); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } } @@ -648,42 +648,42 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithListsNoNulls) // Test with a very small limit: 1 byte { auto const [result, num_chunks] = chunked_read(filepath, 1); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // Test with a very large limit { auto const [result, num_chunks] = chunked_read(filepath, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // chunk size slightly less than 1 page (forcing it to be at least 1 page per read) { auto const [result, num_chunks] = chunked_read(filepath, 200'000); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // chunk size exactly 1 page { auto const [result, num_chunks] = chunked_read(filepath, 200'004); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // chunk size 2 pages. 3 chunks (2 pages + 2 pages + 1 page) { auto const [result, num_chunks] = chunked_read(filepath, 400'008); - EXPECT_EQ(num_chunks, 3); + // EXPECT_EQ(num_chunks, 3); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // chunk size 2 pages minus one byte: each chunk will be just one page { auto const [result, num_chunks] = chunked_read(filepath, 400'007); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } } @@ -731,42 +731,42 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithListsHavingNulls) // Test with a very small limit: 1 byte { auto const [result, num_chunks] = chunked_read(filepath, 1); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // Test with a very large limit { auto const [result, num_chunks] = chunked_read(filepath, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // chunk size slightly less than 1 page (forcing it to be at least 1 page per read) { auto const [result, num_chunks] = chunked_read(filepath, 142'500); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // chunk size exactly 1 page { auto const [result, num_chunks] = chunked_read(filepath, 142'504); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // chunk size 2 pages. 3 chunks (2 pages + 2 pages + 1 page) { auto const [result, num_chunks] = chunked_read(filepath, 285'008); - EXPECT_EQ(num_chunks, 3); + // EXPECT_EQ(num_chunks, 3); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } // chunk size 2 pages minus 1 byte: each chunk will be just one page { auto const [result, num_chunks] = chunked_read(filepath, 285'007); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); } } @@ -821,31 +821,31 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithStructsOfLists) } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 0); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } // Test with a very small limit: 1 byte { auto const [result, num_chunks] = chunked_read(filepath_no_null, 1); - EXPECT_EQ(num_chunks, 10); + // EXPECT_EQ(num_chunks, 10); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 1); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } // Test with a very large limit { auto const [result, num_chunks] = chunked_read(filepath_no_null, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } @@ -858,49 +858,49 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithStructsOfLists) { auto const [result, num_chunks] = chunked_read(filepath_no_null, 1'000'000); - EXPECT_EQ(num_chunks, 7); + // EXPECT_EQ(num_chunks, 7); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null, 1'500'000); - EXPECT_EQ(num_chunks, 4); + // EXPECT_EQ(num_chunks, 4); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null, 2'000'000); - EXPECT_EQ(num_chunks, 4); + // EXPECT_EQ(num_chunks, 4); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null, 5'000'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 1'000'000); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 1'500'000); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 2'000'000); - EXPECT_EQ(num_chunks, 3); + // EXPECT_EQ(num_chunks, 3); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 5'000'000); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } } @@ -962,31 +962,31 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithListsOfStructs) } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 0); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } // Test with a very small limit: 1 byte { auto const [result, num_chunks] = chunked_read(filepath_no_null, 1); - EXPECT_EQ(num_chunks, 10); + // EXPECT_EQ(num_chunks, 10); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 1); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } // Test with a very large limit { auto const [result, num_chunks] = chunked_read(filepath_no_null, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 2L << 40); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } @@ -996,49 +996,49 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadWithListsOfStructs) // reader_impl_preprocess.cu -> find_splits() { auto const [result, num_chunks] = chunked_read(filepath_no_null, 1'000'000); - EXPECT_EQ(num_chunks, 7); + // EXPECT_EQ(num_chunks, 7); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null, 1'500'000); - EXPECT_EQ(num_chunks, 4); + // EXPECT_EQ(num_chunks, 4); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null, 2'000'000); - EXPECT_EQ(num_chunks, 4); + // EXPECT_EQ(num_chunks, 4); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_no_null, 5'000'000); - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_no_null, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 1'000'000); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 1'500'000); - EXPECT_EQ(num_chunks, 5); + // EXPECT_EQ(num_chunks, 5); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 2'000'000); - EXPECT_EQ(num_chunks, 3); + // EXPECT_EQ(num_chunks, 3); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } { auto const [result, num_chunks] = chunked_read(filepath_with_nulls, 5'000'000); - EXPECT_EQ(num_chunks, 1); + // EXPECT_EQ(num_chunks, 1); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected_with_nulls, *result); } } @@ -1129,8 +1129,8 @@ void input_limit_test_read(std::vector const& test_filenames, for (size_t idx = 0; idx < test_filenames.size(); idx++) { auto result = chunked_read(test_filenames[idx], output_limit, input_limit); - CUDF_EXPECTS(result.second == expected_chunk_counts[idx], - "Unexpected number of chunks produced in chunk read"); + // CUDF_EXPECTS(result.second == expected_chunk_counts[idx], + // "Unexpected number of chunks produced in chunk read"); CUDF_TEST_EXPECT_TABLES_EQUIVALENT(*result.first, t); } } @@ -1509,7 +1509,7 @@ TEST_F(ParquetChunkedReaderTest, TestChunkedReadOutOfBoundChunks) auto const [result, num_chunks] = read_chunks_with_while_loop(reader); auto const out_of_bound_table_chunk = reader.read_chunk().tbl; - EXPECT_EQ(num_chunks, 2); + // EXPECT_EQ(num_chunks, 2); EXPECT_EQ(reader.has_next(), false); CUDF_TEST_EXPECT_TABLES_EQUAL(*out_of_bound_table_chunk, *empty_table); CUDF_TEST_EXPECT_TABLES_EQUAL(*expected, *result); From cc5626b2a7d37c1d82fe1be5259f3390939ad0a1 Mon Sep 17 00:00:00 2001 From: "Richard (Rick) Zamora" Date: Thu, 20 Feb 2025 12:04:21 -0600 Subject: [PATCH 065/129] Fix upstream dask `loc` test (#18045) The upstream [test_gpu_loc dask test is failing](https://github.com/rapidsai/dask-upstream-testing/actions/runs/13419658348/job/37489003426), and the culprit seems to be https://github.com/dask/dask/pull/11745 I don't see anything "wrong" with the changes in that PR. However, the divisions assertion is now sensitive to the fact that an element of divisions can end up being a 0-dimensional cupy arrays for GPU-backed data. The `test_gpu_loc` feels like a pretty strange corner case to me. So, this PR proposes a temporary fix in dask-cudf. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) URL: https://github.com/rapidsai/cudf/pull/18045 --- python/dask_cudf/dask_cudf/_expr/__init__.py | 2 ++ python/dask_cudf/dask_cudf/_expr/expr.py | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/python/dask_cudf/dask_cudf/_expr/__init__.py b/python/dask_cudf/dask_cudf/_expr/__init__.py index e8051eedafb..a7cdd873aec 100644 --- a/python/dask_cudf/dask_cudf/_expr/__init__.py +++ b/python/dask_cudf/dask_cudf/_expr/__init__.py @@ -20,6 +20,7 @@ ) from dask.dataframe.dask_expr._expr import ( Elemwise, + EnforceRuntimeDivisions, Expr, RenameAxis, VarColumns, @@ -70,6 +71,7 @@ "DXSeriesGroupBy", "DecomposableGroupbyAggregation", "Elemwise", + "EnforceRuntimeDivisions", "Expr", "FragmentWrapper", "FrameBase", diff --git a/python/dask_cudf/dask_cudf/_expr/expr.py b/python/dask_cudf/dask_cudf/_expr/expr.py index c433ab71aa1..b48fd108e4f 100644 --- a/python/dask_cudf/dask_cudf/_expr/expr.py +++ b/python/dask_cudf/dask_cudf/_expr/expr.py @@ -14,6 +14,7 @@ from dask_cudf._expr import ( CumulativeBlockwise, Elemwise, + EnforceRuntimeDivisions, Expr, Reduction, RenameAxis, @@ -202,6 +203,20 @@ def _patched_get_divisions(frame, other, *args, **kwargs): return _original_get_divisions(frame, other, *args, **kwargs) +_original_erd_divisions = EnforceRuntimeDivisions._divisions + + +def _patched_erd_divisions(self): + # This patch is needed for upstream dask testing + # (dask/dataframe/tests/test_indexing.py::test_gpu_loc). + # Without this patch, an individual element of divisions + # may end up as a 0-dim cupy array. + # TODO: Find long-term fix. + # Maybe update `LocList._layer_information`? + divs = _original_erd_divisions(self) + return tuple(div.item() if hasattr(div, "item") else div for div in divs) + + _PATCHED = False @@ -213,4 +228,5 @@ def _patch_dask_expr(): CumulativeBlockwise._kwargs = PatchCumulativeBlockwise._kwargs Expr.var = _patched_var _shuffle_module._get_divisions = _patched_get_divisions + EnforceRuntimeDivisions._divisions = _patched_erd_divisions _PATCHED = True From 8bef5423a1b16c1feeb942e91925dc24af8a4897 Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Thu, 20 Feb 2025 16:17:25 -0500 Subject: [PATCH 066/129] Fix hang on invalid UTF-8 data in string_view iterator (#18039) The `cudf::string_view::const_iterator` provides functions that navigate through UTF-8 variable-length characters appropriately. This PR fixes the iterator increment logic to prevent a possible infinite loop when the iterator wraps invalid UTF-8 encoded memory, Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Robert (Bobby) Evans (https://github.com/revans2) - Shruti Shivakumar (https://github.com/shrshi) - Vukasin Milovanovic (https://github.com/vuule) URL: https://github.com/rapidsai/cudf/pull/18039 --- cpp/include/cudf/strings/string_view.cuh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/cpp/include/cudf/strings/string_view.cuh b/cpp/include/cudf/strings/string_view.cuh index f0040e069d8..b91748cfc7d 100644 --- a/cpp/include/cudf/strings/string_view.cuh +++ b/cpp/include/cudf/strings/string_view.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -159,8 +159,11 @@ __device__ inline string_view::const_iterator::const_iterator(string_view const& __device__ inline string_view::const_iterator& string_view::const_iterator::operator++() { - if (byte_pos < bytes) - byte_pos += strings::detail::bytes_in_utf8_byte(static_cast(p[byte_pos])); + if (byte_pos < bytes) { + // max is used to prevent an infinite loop on invalid UTF-8 data + byte_pos += + cuda::std::max(1, strings::detail::bytes_in_utf8_byte(static_cast(p[byte_pos]))); + } ++char_pos; return *this; } From 163e27b4785a0890323bc0ff9c7a27657d23e0d3 Mon Sep 17 00:00:00 2001 From: Michael Schellenberger Costa Date: Thu, 20 Feb 2025 22:59:49 +0100 Subject: [PATCH 067/129] Replace deprecated CCCL features (#18036) CCCL has deprecated a set of features that will be removed in an upcoming release. Replace them with the suggested alternative. NOTE: We have some facilities like `cub::AliasTemporaries` that have been internalized in the 2.8 release, where there is still an alias available in the `cub` namespace. However that alias will be removed in the CCCL 3.0 release that we are testing in our CI. I added a conditional compilation there to ensure we are still able to build against CCCL 3.0. We can remove this once rapids switches to CCCL 2.8 Authors: - Michael Schellenberger Costa (https://github.com/miscco) - David Wendt (https://github.com/davidwendt) Approvers: - David Wendt (https://github.com/davidwendt) - Muhammad Haseeb (https://github.com/mhaseeb123) URL: https://github.com/rapidsai/cudf/pull/18036 --- cpp/src/io/fst/dispatch_dfa.cuh | 39 +++++++++++----------- cpp/src/io/fst/logical_stack.cuh | 19 +++++------ cpp/src/io/parquet/writer_impl_helpers.cpp | 3 ++ 3 files changed, 32 insertions(+), 29 deletions(-) diff --git a/cpp/src/io/fst/dispatch_dfa.cuh b/cpp/src/io/fst/dispatch_dfa.cuh index ef5e9c8a78f..d8be747d93d 100644 --- a/cpp/src/io/fst/dispatch_dfa.cuh +++ b/cpp/src/io/fst/dispatch_dfa.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -209,29 +209,25 @@ struct DispatchFSM : DeviceFSMPolicy { FstScanTileStateT fst_tile_state) { - cudaError_t error = cudaSuccess; - cub::KernelConfig dfa_simulation_config; - using PolicyT = typename ActivePolicyT::AgentDFAPolicy; - if (CubDebug(error = dfa_simulation_config.Init(dfa_kernel))) return error; // Kernel invocation uint32_t grid_size = std::max( 1u, CUB_QUOTIENT_CEILING(num_chars, PolicyT::BLOCK_THREADS * PolicyT::ITEMS_PER_THREAD)); - uint32_t block_threads = dfa_simulation_config.block_threads; - - dfa_kernel<<>>(dfa, - d_chars_in, - num_chars, - seed_state, - d_thread_state_transition, - tile_state, - fst_tile_state, - transduced_out_it, - transduced_out_idx_it, - d_num_transduced_out_it); + + dfa_kernel<<>>(dfa, + d_chars_in, + num_chars, + seed_state, + d_thread_state_transition, + tile_state, + fst_tile_state, + transduced_out_it, + transduced_out_idx_it, + d_num_transduced_out_it); // Check for errors + cudaError_t error = cudaSuccess; if (CubDebug(error = cudaPeekAtLastError())) return error; return error; @@ -394,8 +390,13 @@ struct DispatchFSM : DeviceFSMPolicy { // Alias the temporary allocations from the single storage blob (or compute the necessary size // of the blob) - error = - cub::AliasTemporaries(d_temp_storage, temp_storage_bytes, allocations, allocation_sizes); + // TODO (@miscco): remove this once rapids moves to CCCL 2.8 +#if CCCL_VERSION_MAJOR >= 3 + error = cub::detail::AliasTemporaries( +#else // ^^^ CCCL 3.x ^^^ / vvv CCCL 2.x vvv + error = cub::AliasTemporaries( +#endif // CCCL 2.x + d_temp_storage, temp_storage_bytes, allocations, allocation_sizes); if (error != cudaSuccess) return error; // Return if the caller is simply requesting the size of the storage allocation diff --git a/cpp/src/io/fst/logical_stack.cuh b/cpp/src/io/fst/logical_stack.cuh index 98641f2c893..7b217d08da3 100644 --- a/cpp/src/io/fst/logical_stack.cuh +++ b/cpp/src/io/fst/logical_stack.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -332,9 +332,8 @@ void sparse_stack_op_to_top_of_stack(StackSymbolItT d_symbols, // Transforming sequence of stack symbols to stack operations using StackSymbolToStackOpT = detail::StackSymbolToStackOp; - // TransformInputIterator converting stack symbols to stack operations - using TransformInputItT = - cub::TransformInputIterator; + // transform_iterator converting stack symbols to stack operations + using TransformInputItT = thrust::transform_iterator; constexpr bool supports_reset_op = SupportResetOperation == stack_op_support::WITH_RESET_SUPPORT; @@ -365,8 +364,8 @@ void sparse_stack_op_to_top_of_stack(StackSymbolItT d_symbols, // with the empty_stack_symbol StackOpT const empty_stack{0, empty_stack_symbol}; - cub::TransformInputIterator, StackOpT*> - kv_ops_scan_in(nullptr, detail::RemapEmptyStack{empty_stack}); + thrust::transform_iterator, StackOpT*> kv_ops_scan_in( + nullptr, detail::RemapEmptyStack{empty_stack}); StackOpT* kv_ops_scan_out = nullptr; std::size_t stack_level_scan_bytes = 0; @@ -532,7 +531,7 @@ void sparse_stack_op_to_top_of_stack(StackSymbolItT d_symbols, end_bit, stream)); - // TransformInputIterator that remaps all operations on stack level 0 to the empty stack symbol + // transform_iterator that remaps all operations on stack level 0 to the empty stack symbol kv_ops_scan_in = {reinterpret_cast(d_kv_operations_unsigned.Current()), detail::RemapEmptyStack{empty_stack}}; kv_ops_scan_out = reinterpret_cast(d_kv_operations_unsigned.Alternate()); @@ -553,9 +552,9 @@ void sparse_stack_op_to_top_of_stack(StackSymbolItT d_symbols, thrust::device_ptr{d_top_of_stack + num_symbols_out}, read_symbol); - // Transform the stack operations to the stack symbol they represent - cub::TransformInputIterator - kv_op_to_stack_sym_it(kv_ops_scan_out, detail::StackOpToStackSymbol{}); + // transform_iterator the stack operations to the stack symbol they represent + thrust::transform_iterator kv_op_to_stack_sym_it( + kv_ops_scan_out, detail::StackOpToStackSymbol{}); // Scatter the stack symbols to the output tape (spots that are not scattered to have been // pre-filled with the read-symbol) diff --git a/cpp/src/io/parquet/writer_impl_helpers.cpp b/cpp/src/io/parquet/writer_impl_helpers.cpp index ede788c97c2..dee1a3615ef 100644 --- a/cpp/src/io/parquet/writer_impl_helpers.cpp +++ b/cpp/src/io/parquet/writer_impl_helpers.cpp @@ -26,6 +26,9 @@ #include #include +#include +#include + namespace cudf::io::parquet::detail { using namespace cudf::io::detail; From 566eb9156072570179c2df92dcfdfd46e50d7c88 Mon Sep 17 00:00:00 2001 From: Bradley Dice Date: Fri, 21 Feb 2025 01:21:45 -0600 Subject: [PATCH 068/129] Test narwhals in CI (#17884) Contributes to #17662. @MarcoGorelli provided very helpful instructions for running the narwhals test suite. We should examine and correct the failing tests. We are down to 39 failures, shown in [this Kaggle notebook](https://www.kaggle.com/code/marcogorelli/testing-cudf-in-narwhals?scriptVersionId=220053914). Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Matthew Murray (https://github.com/Matt711) - Gil Forsyth (https://github.com/gforsyth) URL: https://github.com/rapidsai/cudf/pull/17884 --- .github/workflows/pr.yaml | 15 +++++++++++++ ci/test_narwhals.sh | 44 +++++++++++++++++++++++++++++++++++++++ dependencies.yaml | 10 +++++++++ 3 files changed, 69 insertions(+) create mode 100755 ci/test_narwhals.sh diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index e7a37a477b7..38b890893d0 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -40,6 +40,7 @@ jobs: - unit-tests-cudf-pandas - pandas-tests - pandas-tests-diff + - narwhals-tests - telemetry-setup - third-party-integration-tests-cudf-pandas secrets: inherit @@ -358,6 +359,20 @@ jobs: node_type: "cpu4" build_type: pull-request run_script: "ci/cudf_pandas_scripts/pandas-tests/diff.sh" + narwhals-tests: + needs: [conda-python-build, changed-files] + secrets: inherit + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 + if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python + with: + build_type: pull-request + branch: ${{ inputs.branch }} + date: ${{ inputs.date }} + sha: ${{ inputs.sha }} + node_type: "gpu-l4-latest-1" + continue-on-error: true + container_image: "rapidsai/ci-conda:latest" + run_script: ci/test_narwhals.sh spark-rapids-jni: needs: changed-files uses: ./.github/workflows/spark-rapids-jni.yaml diff --git a/ci/test_narwhals.sh b/ci/test_narwhals.sh new file mode 100755 index 00000000000..4a32ff0b0fd --- /dev/null +++ b/ci/test_narwhals.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# Copyright (c) 2025, NVIDIA CORPORATION. + +# Support invoking test_python_cudf.sh outside the script directory +cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../ || exit 1 + +# Common setup steps shared by Python test jobs +source ./ci/test_python_common.sh test_python_narwhals + +rapids-logger "Check GPU usage" +nvidia-smi +rapids-print-env +EXITCODE=0 +trap "EXITCODE=1" ERR +set +e + +rapids-logger "pytest narwhals" +git clone https://github.com/narwhals-dev/narwhals --depth=1 +pushd narwhals || exit 1 +rapids-pip-retry install -U -e ".[dev]" + +rapids-logger "Check narwhals versions" +python -c "import narwhals; print(narwhals.show_versions())" + +rapids-logger "Run narwhals tests for cuDF" +python -m pytest \ + --cache-clear \ + --junitxml="${RAPIDS_TESTS_DIR}/junit-cudf-narwhals.xml" \ + --numprocesses=8 \ + --dist=worksteal \ + --constructors=cudf + +rapids-logger "Run narwhals tests for cuDF Polars" +NARWHALS_POLARS_GPU=1 python -m pytest \ + --cache-clear \ + --junitxml="${RAPIDS_TESTS_DIR}/junit-cudf-polars-narwhals.xml" \ + --numprocesses=8 \ + --dist=worksteal \ + --constructors=polars[lazy] + +popd || exit 1 + +rapids-logger "Test script exiting with value: $EXITCODE" +exit ${EXITCODE} diff --git a/dependencies.yaml b/dependencies.yaml index c8893fc8b49..585144300af 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -379,6 +379,16 @@ files: includes: - test_python_common - test_python_cudf_common + test_python_narwhals: + output: none + includes: + - cuda_version + - py_version + - test_python_common + - test_python_cudf_common + - test_python_cudf + - depends_on_cudf + - depends_on_cudf_polars channels: - rapidsai - rapidsai-nightly From 1fe744fb43044e9beb728c244f3f4a0beac3588f Mon Sep 17 00:00:00 2001 From: Bradley Dice Date: Fri, 21 Feb 2025 07:46:39 -0600 Subject: [PATCH 069/129] Update to nvcomp 4.2.0.11 (#18042) This updates to use nvcomp 4.2.0.11. The version is updated in rapids-cmake in https://github.com/rapidsai/rapids-cmake/pull/780. --- conda/environments/all_cuda-118_arch-x86_64.yaml | 2 +- conda/environments/all_cuda-128_arch-x86_64.yaml | 2 +- conda/recipes/libcudf/conda_build_config.yaml | 2 +- dependencies.yaml | 8 ++++---- python/libcudf/pyproject.toml | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index cc01f5286ef..4df04b61bae 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -60,7 +60,7 @@ dependencies: - numpy>=1.23,<3.0a0 - numpydoc - nvcc_linux-64=11.8 -- nvcomp==4.1.0.6 +- nvcomp==4.2.0.11 - nvtx>=0.2.1 - openpyxl - packaging diff --git a/conda/environments/all_cuda-128_arch-x86_64.yaml b/conda/environments/all_cuda-128_arch-x86_64.yaml index f4cdbed9be6..a32f8b2ee25 100644 --- a/conda/environments/all_cuda-128_arch-x86_64.yaml +++ b/conda/environments/all_cuda-128_arch-x86_64.yaml @@ -58,7 +58,7 @@ dependencies: - numba>=0.59.1,<0.61.0a0 - numpy>=1.23,<3.0a0 - numpydoc -- nvcomp==4.1.0.6 +- nvcomp==4.2.0.11 - nvtx>=0.2.1 - openpyxl - packaging diff --git a/conda/recipes/libcudf/conda_build_config.yaml b/conda/recipes/libcudf/conda_build_config.yaml index 181064465ef..7c3f5777782 100644 --- a/conda/recipes/libcudf/conda_build_config.yaml +++ b/conda/recipes/libcudf/conda_build_config.yaml @@ -32,7 +32,7 @@ flatbuffers_version: - "=24.3.25" nvcomp_version: - - "=4.1.0.6" + - "=4.2.0.11" zlib_version: - ">=1.2.13" diff --git a/dependencies.yaml b/dependencies.yaml index 501128d278e..c59643273aa 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -432,7 +432,7 @@ dependencies: - output_types: conda packages: # Align nvcomp version with rapids-cmake - - nvcomp==4.1.0.6 + - nvcomp==4.2.0.11 specific: - output_types: [requirements, pyproject] matrices: @@ -440,12 +440,12 @@ dependencies: cuda: "12.*" use_cuda_wheels: "true" packages: - - nvidia-nvcomp-cu12==4.1.0.6 + - nvidia-nvcomp-cu12==4.2.0.11 - matrix: cuda: "11.*" use_cuda_wheels: "true" packages: - - nvidia-nvcomp-cu11==4.1.0.6 + - nvidia-nvcomp-cu11==4.2.0.11 # if use_cuda_wheels=false is provided, do not add dependencies on any CUDA wheels # (e.g. for DLFW and pip devcontainers) - matrix: @@ -455,7 +455,7 @@ dependencies: # (just as a source of documentation, as this populates pyproject.toml in source control) - matrix: packages: - - nvidia-nvcomp==4.1.0.6 + - nvidia-nvcomp==4.2.0.11 rapids_build_skbuild: common: - output_types: [conda, requirements, pyproject] diff --git a/python/libcudf/pyproject.toml b/python/libcudf/pyproject.toml index d16ad97ec54..8c2d5a8288d 100644 --- a/python/libcudf/pyproject.toml +++ b/python/libcudf/pyproject.toml @@ -39,7 +39,7 @@ classifiers = [ ] dependencies = [ "libkvikio==25.2.*,>=0.0.0a0", - "nvidia-nvcomp==4.1.0.6", + "nvidia-nvcomp==4.2.0.11", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. [project.urls] From 024f52bfec467acdeea2741b7081ca9cf856334f Mon Sep 17 00:00:00 2001 From: Eyal Soha <69258779+esoha-nvidia@users.noreply.github.com> Date: Fri, 21 Feb 2025 10:35:51 -0700 Subject: [PATCH 070/129] Allow cudf::type_to_id() (#17831) Currently, `cudf::type_to_id` where X is a valid type for converting to a `cudf::id` only works if X is neither `const` nor `volatile`. However, if it has either of those "cv" qualifiers, the function returns `type_id::EMPTY`. With this change, return the correct type id no matter the cv-qualifiers. Authors: - Eyal Soha (https://github.com/esoha-nvidia) - David Wendt (https://github.com/davidwendt) Approvers: - Karthikeyan (https://github.com/karthikeyann) - Bradley Dice (https://github.com/bdice) - David Wendt (https://github.com/davidwendt) URL: https://github.com/rapidsai/cudf/pull/17831 --- .../cudf/utilities/type_dispatcher.hpp | 51 +++++++++++++------ cpp/tests/types/type_dispatcher_test.cu | 8 ++- 2 files changed, 43 insertions(+), 16 deletions(-) diff --git a/cpp/include/cudf/utilities/type_dispatcher.hpp b/cpp/include/cudf/utilities/type_dispatcher.hpp index c1dd79ef14f..d0aabee6344 100644 --- a/cpp/include/cudf/utilities/type_dispatcher.hpp +++ b/cpp/include/cudf/utilities/type_dispatcher.hpp @@ -46,14 +46,14 @@ namespace CUDF_EXPORT cudf { * For example: * * ``` - * return cudf::type_to_id(); // Returns INT32 + * return cudf::base_type_to_id(); // Returns INT32 * ``` * - * @tparam T The type to map to a `cudf::type_id` + * @tparam T The non-cv type to map to a `cudf::type_id` * @return The `cudf::type_id` corresponding to the specified type */ template -CUDF_HOST_DEVICE inline constexpr type_id type_to_id() +CUDF_HOST_DEVICE inline constexpr type_id base_type_to_id() { return type_id::EMPTY; }; @@ -114,20 +114,24 @@ using device_storage_type_t = // clang-format on /** - * @brief Checks if `fixed_point`-like types have template type `T` matching the column's - * stored type id + * @brief Maps a C++ type to its corresponding `cudf::type_id` * - * @tparam T The type that is stored on the device - * @param id The `data_type::id` of the column - * @return `true` If T matches the stored column `type_id` - * @return `false` If T does not match the stored column `type_id` + * When explicitly passed a template argument of a given type, returns the + * appropriate `type_id` enum for the specified C++ type. + * + * For example: + * + * ``` + * return cudf::type_to_id(); // Returns INT32 + * ``` + * + * @tparam T The type to map to a `cudf::type_id` + * @return The `cudf::type_id` corresponding to the specified type */ template -constexpr bool type_id_matches_device_storage_type(type_id id) +constexpr inline type_id type_to_id() { - return (id == type_id::DECIMAL32 && std::is_same_v) || - (id == type_id::DECIMAL64 && std::is_same_v) || - (id == type_id::DECIMAL128 && std::is_same_v) || id == type_to_id(); + return base_type_to_id>(); } /** @@ -140,7 +144,7 @@ constexpr bool type_id_matches_device_storage_type(type_id id) #ifndef CUDF_TYPE_MAPPING #define CUDF_TYPE_MAPPING(Type, Id) \ template <> \ - constexpr inline type_id type_to_id() \ + constexpr inline type_id base_type_to_id() \ { \ return Id; \ } \ @@ -194,11 +198,28 @@ CUDF_TYPE_MAPPING(cudf::struct_view, type_id::STRUCT) * @return id for 'char' type */ template <> // CUDF_TYPE_MAPPING(char,INT8) causes duplicate id_to_type_impl definition -constexpr inline type_id type_to_id() +constexpr inline type_id base_type_to_id() { return type_id::INT8; } +/** + * @brief Checks if `fixed_point`-like types have template type `T` matching the column's + * stored type id + * + * @tparam T The type that is stored on the device + * @param id The `data_type::id` of the column + * @return `true` If T matches the stored column `type_id` + * @return `false` If T does not match the stored column `type_id` + */ +template +constexpr bool type_id_matches_device_storage_type(type_id id) +{ + return (id == type_id::DECIMAL32 && std::is_same_v) || + (id == type_id::DECIMAL64 && std::is_same_v) || + (id == type_id::DECIMAL128 && std::is_same_v) || id == type_to_id(); +} + /** * @brief Use this specialization on `type_dispatcher` whenever you only need to operate on the * underlying stored type. diff --git a/cpp/tests/types/type_dispatcher_test.cu b/cpp/tests/types/type_dispatcher_test.cu index f18e9afc09c..ddd318710a4 100644 --- a/cpp/tests/types/type_dispatcher_test.cu +++ b/cpp/tests/types/type_dispatcher_test.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -50,6 +50,12 @@ TYPED_TEST(TypedDispatcherTest, TypeToId) { EXPECT_TRUE(cudf::type_dispatcher(cudf::data_type{cudf::type_to_id()}, type_tester{})); + EXPECT_TRUE(cudf::type_dispatcher(cudf::data_type{cudf::type_to_id()}, + type_tester{})); + EXPECT_TRUE(cudf::type_dispatcher(cudf::data_type{cudf::type_to_id()}, + type_tester{})); + EXPECT_TRUE(cudf::type_dispatcher(cudf::data_type{cudf::type_to_id()}, + type_tester{})); } namespace { From 90dc38c84a5e712f60c5253c85f58ef508c0adcd Mon Sep 17 00:00:00 2001 From: Tanmay Gujar Date: Fri, 21 Feb 2025 10:28:58 -0800 Subject: [PATCH 071/129] Fix memcopy direction for concatenate (#18058) `cudaMemcpyAsync` can return a `cudaError_t` value which should be checked for runtime errors. We should preserve the behavior of `thrust::copy` which got replaced with the `cudaMemcpyAsync` call in #17584. The driver may do the right thing and infer the source and destination pointer location instead of using the `cudaMemcpyKind`, but this still leads to weird circumstances where the copy type in code is DtoD while the actual copy at runtime is HtoD Authors: - Tanmay Gujar (https://github.com/tgujar) Approvers: - Bradley Dice (https://github.com/bdice) - Karthikeyan (https://github.com/karthikeyann) - David Wendt (https://github.com/davidwendt) URL: https://github.com/rapidsai/cudf/pull/18058 --- cpp/src/copying/concatenate.cu | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/cpp/src/copying/concatenate.cu b/cpp/src/copying/concatenate.cu index 6fc49afd7ac..4237e3f0954 100644 --- a/cpp/src/copying/concatenate.cu +++ b/cpp/src/copying/concatenate.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -308,11 +308,11 @@ std::unique_ptr for_each_concatenate(host_span views, auto count = 0; for (auto& v : views) { - cudaMemcpyAsync(m_view.begin() + count, - v.begin(), - v.size() * sizeof(T), - cudaMemcpyDeviceToDevice, - stream.value()); + CUDF_CUDA_TRY(cudaMemcpyAsync(m_view.begin() + count, + v.begin(), + v.size() * sizeof(T), + cudaMemcpyDefault, + stream.value())); count += v.size(); } From 1a95a0c8bbdc3164db8e76d0c1fc4b68424f41b2 Mon Sep 17 00:00:00 2001 From: Vukasin Milovanovic Date: Fri, 21 Feb 2025 13:42:28 -0800 Subject: [PATCH 072/129] Host Snappy compression (#17824) Issue #17641 Simple implementation of the algorithm based on https://github.com/google/snappy/blob/main/snappy.cc. Other changes: - Limited the number of threads in the compression thread pool to 32, based on benchmark results; - Submit tasks to the thread pool starting from the largest buffers to improve CPU utilization (impact verified with benchmark results); - Changed the environment variable from `LIBCUDF_USE_HOST_COMPRESSION` to `LIBCUDF_HOST_COMPRESSION`, and its type to string, to allow potential future settings "OFF", "AUTO" and "ON". Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Basit Ayantunde (https://github.com/lamarrr) - Shruti Shivakumar (https://github.com/shrshi) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: https://github.com/rapidsai/cudf/pull/17824 --- cpp/src/io/comp/comp.cpp | 247 +++++++++++++++--- cpp/src/io/utilities/getenv_or.hpp | 2 +- cpp/tests/CMakeLists.txt | 2 +- .../comp/{decomp_test.cpp => comp_test.cpp} | 40 ++- 4 files changed, 247 insertions(+), 44 deletions(-) rename cpp/tests/io/comp/{decomp_test.cpp => comp_test.cpp} (86%) diff --git a/cpp/src/io/comp/comp.cpp b/cpp/src/io/comp/comp.cpp index 3800835eaf1..280c07a4ff1 100644 --- a/cpp/src/io/comp/comp.cpp +++ b/cpp/src/io/comp/comp.cpp @@ -18,7 +18,6 @@ #include "gpuinflate.hpp" #include "io/utilities/getenv_or.hpp" -#include "io/utilities/hostdevice_vector.hpp" #include "nvcomp_adapter.hpp" #include @@ -32,14 +31,17 @@ #include #include // GZIP compression +#include + namespace cudf::io::detail { namespace { auto& h_comp_pool() { - static std::size_t pool_size = - getenv_or("LIBCUDF_HOST_COMPRESSION_NUM_THREADS", std::thread::hardware_concurrency()); + static const std::size_t default_pool_size = std::min(32u, std::thread::hardware_concurrency()); + static const std::size_t pool_size = + getenv_or("LIBCUDF_HOST_COMPRESSION_NUM_THREADS", default_pool_size); static BS::thread_pool pool(pool_size); return pool; } @@ -92,35 +94,199 @@ std::vector compress_gzip(host_span src) return dst; } -/** - * @brief SNAPPY device compressor - */ -std::vector compress_snappy(host_span src, - rmm::cuda_stream_view stream) +namespace snappy { + +template +[[nodiscard]] T load(uint8_t const* ptr) +{ + T value; + std::memcpy(&value, ptr, sizeof(T)); + return value; +} + +class hash_table { + std::vector tbl; + static constexpr int hash_table_bits = 15; + + public: + hash_table() : tbl(1 << hash_table_bits, 0) {} + + void clear() { std::fill(tbl.begin(), tbl.end(), 0); } + + [[nodiscard]] uint16_t* entry(uint32_t bytes) + { + constexpr uint32_t multiplier = 0x1e35a7bd; + auto const hash = (bytes * multiplier) >> (31 - hash_table_bits); + return tbl.data() + hash / sizeof(uint16_t); + } +}; + +uint8_t* emit_literal(uint8_t* out_begin, uint8_t const* literal_begin, uint8_t const* literal_end) +{ + auto const literal_size = literal_end - literal_begin; + if (literal_size == 0) { return out_begin; } + auto const n = literal_size - 1; + + auto out_it = out_begin; + if (n < 60) { + // Fits into a single tag byte + *out_it++ = n << 2; + } else { + auto const log2_n = 31 - __builtin_clz(n); + auto const count = (log2_n >> 3) + 1; + *out_it++ = (59 + count) << 2; + std::memcpy(out_it, &n, count); + out_it += count; + } + std::memcpy(out_it, literal_begin, literal_size); + return out_it + literal_size; +} + +uint8_t* emit_copy(uint8_t* out_begin, size_t offset, size_t len) +{ + while (len > 0) { + auto const copy_len = std::min(len, 64ul); + auto const out_val = 2 + ((copy_len - 1) << 2) + (offset << 8); + std::memcpy(out_begin, &out_val, 3); + + out_begin += 3; + len -= copy_len; + } + return out_begin; +} + +size_t compress_block(host_span input, hash_table& table, host_span output) +{ + auto const [in_remain, out_remain] = [&]() -> std::pair { + auto in_it = input.begin(); + auto out_it = output.begin(); + + // The algorithm reads 8 bytes at a time, so we need to ensure there are at least 8 bytes + auto const input_max = input.end() - sizeof(uint64_t); + while (in_it < input_max) { + auto const next_emit = in_it++; + auto data = load(in_it); + uint32_t stride = 1; + uint8_t const* candidate = nullptr; + + auto word_match_found = [&]() { + if (input_max - in_it < 16) { return false; } + for (size_t word_idx = 0; word_idx < 4; ++word_idx) { + for (size_t byte_idx = 0; byte_idx < sizeof(uint32_t); ++byte_idx) { + auto const offset = sizeof(uint32_t) * word_idx + byte_idx; + auto* const entry = table.entry(static_cast(data)); + candidate = input.begin() + *entry; + *entry = in_it - input.data() + offset; + + if (load(candidate) == static_cast(data)) { + *(out_it++) = offset * sizeof(uint32_t); + std::memcpy(out_it, next_emit, offset + 1); + in_it += offset; + out_it += offset + 1; + stride = 1; + return true; + } + data >>= 8; + } + // Fetch the next eight bytes + data = load(in_it + sizeof(uint32_t) * (word_idx + 1)); + } + in_it += 16; + return false; + }(); + + if (not word_match_found) { + // keep looking for a match with increasing stride + while (true) { + auto* const entry = table.entry(static_cast(data)); + candidate = input.begin() + *entry; + *entry = in_it - input.begin(); + if (static_cast(data) == load(candidate)) { + stride = 1; + break; + } + + auto const next_input = in_it + stride; + if (next_input > input_max) { + // Reached the end of the input without finding a match + return {next_emit, out_it}; + } + + data = load(next_input); + in_it = next_input; + stride += 1; + } + + // Emit data prior to the match as literal + out_it = emit_literal(out_it, next_emit, in_it); + } + + // Emit match(es) + do { + auto const match_len = std::mismatch(in_it, input.end(), candidate).first - in_it; + out_it = emit_copy(out_it, in_it - candidate, match_len); + + in_it += match_len; + if (in_it >= input_max) { + // Reached the end of the input, no more matches to look for + return {in_it, out_it}; + } + data = load(in_it); + *table.entry(load(in_it - 1)) = in_it - input.begin() - 1; + auto* const entry = table.entry(data); + candidate = input.begin() + *entry; + *entry = in_it - input.begin(); + + } while (static_cast(data) == load(candidate)); + } + + return {in_it, out_it}; + }(); + + // Emit the remaining data as a literal + return emit_literal(out_remain, in_remain, input.end()) - output.begin(); +} + +void append_varint(std::vector& output, size_t v) +{ + while (v > 127) { + output.push_back((v & 0x7F) | 0x80); + v >>= 7; + } + output.push_back(v); +} + +[[nodiscard]] std::vector compress(host_span src) { - auto const d_src = - cudf::detail::make_device_uvector_async(src, stream, cudf::get_current_device_resource_ref()); - cudf::detail::hostdevice_vector> inputs(1, stream); - inputs[0] = d_src; - inputs.host_to_device_async(stream); - - auto dst_size = compress_max_output_chunk_size(nvcomp::compression_type::SNAPPY, src.size()); - rmm::device_uvector d_dst(dst_size, stream); - cudf::detail::hostdevice_vector> outputs(1, stream); - outputs[0] = d_dst; - outputs.host_to_device_async(stream); - - cudf::detail::hostdevice_vector hd_status(1, stream); - hd_status[0] = {}; - hd_status.host_to_device_async(stream); - - nvcomp::batched_compress(nvcomp::compression_type::SNAPPY, inputs, outputs, hd_status, stream); - - hd_status.device_to_host_sync(stream); - CUDF_EXPECTS(hd_status[0].status == compression_status::SUCCESS, "snappy compression failed"); - return cudf::detail::make_std_vector_sync(d_dst, stream); + std::vector dst; + append_varint(dst, src.size()); + dst.reserve(dst.size() + max_compressed_size(compression_type::SNAPPY, src.size())); + + hash_table table; // reuse hash table across blocks + constexpr size_t block_size = 1 << 16; + auto const block_max_compressed_size = max_compressed_size(compression_type::SNAPPY, block_size); + for (std::size_t src_offset = 0; src_offset < src.size(); src_offset += block_size) { + // Compress data in blocks of limited size + auto const block = src.subspan(src_offset, std::min(src.size() - src_offset, block_size)); + + auto const previous_size = dst.size(); + auto const curr_block_max_comp_size = + (block.size() == block_size) ? block_max_compressed_size + : max_compressed_size(compression_type::SNAPPY, block.size()); + dst.resize(previous_size + curr_block_max_comp_size); + auto const block_dst = + host_span{dst.data() + previous_size, dst.size() - previous_size}; + + table.clear(); + auto const comp_block_size = compress_block(block, table, block_dst); + dst.resize(previous_size + comp_block_size); + } + + return dst; } +} // namespace snappy + void device_compress(compression_type compression, device_span const> inputs, device_span const> outputs, @@ -156,6 +322,13 @@ void host_compress(compression_type compression, auto const h_outputs = cudf::detail::make_host_vector_async(outputs, stream); stream.synchronize(); + // Generate order vector to submit largest tasks first + std::vector task_order(num_chunks); + std::iota(task_order.begin(), task_order.end(), 0); + std::sort(task_order.begin(), task_order.end(), [&](size_t a, size_t b) { + return h_inputs[a].size() > h_inputs[b].size(); + }); + std::vector> tasks; auto const num_streams = std::min({num_chunks, @@ -163,9 +336,12 @@ void host_compress(compression_type compression, h_comp_pool().get_thread_count()}); auto const streams = cudf::detail::fork_streams(stream, num_streams); for (size_t i = 0; i < num_chunks; ++i) { + auto const idx = task_order[i]; auto const cur_stream = streams[i % streams.size()]; - auto task = [d_in = h_inputs[i], d_out = h_outputs[i], cur_stream, compression]() -> size_t { - auto const h_in = cudf::detail::make_host_vector_sync(d_in, cur_stream); + auto task = + [d_in = h_inputs[idx], d_out = h_outputs[idx], cur_stream, compression]() -> size_t { + auto h_in = cudf::detail::make_pinned_vector_async(d_in.size(), cur_stream); + cudf::detail::cuda_memcpy(h_in, d_in, cur_stream); auto const h_out = compress(compression, h_in, cur_stream); cudf::detail::cuda_memcpy(d_out.subspan(0, h_out.size()), h_out, cur_stream); return h_out.size(); @@ -174,7 +350,7 @@ void host_compress(compression_type compression, } for (auto i = 0ul; i < num_chunks; ++i) { - h_results[i] = {tasks[i].get(), compression_status::SUCCESS}; + h_results[task_order[i]] = {tasks[i].get(), compression_status::SUCCESS}; } cudf::detail::cuda_memcpy_async(results, h_results, stream); } @@ -183,6 +359,7 @@ void host_compress(compression_type compression, { switch (compression) { case compression_type::GZIP: + case compression_type::SNAPPY: case compression_type::NONE: return true; default: return false; } @@ -212,7 +389,7 @@ void host_compress(compression_type compression, if (not host_compression_supported(compression)) { return false; } if (not device_compression_supported(compression)) { return true; } // If both host and device compression are supported, use the host if the env var is set - return getenv_or("LIBCUDF_USE_HOST_COMPRESSION", 0); + return getenv_or("LIBCUDF_HOST_COMPRESSION", std::string{"OFF"}) == "ON"; } } // namespace @@ -249,12 +426,12 @@ std::optional compress_max_allowed_chunk_size(compression_type compressi std::vector compress(compression_type compression, host_span src, - rmm::cuda_stream_view stream) + rmm::cuda_stream_view) { CUDF_FUNC_RANGE(); switch (compression) { case compression_type::GZIP: return compress_gzip(src); - case compression_type::SNAPPY: return compress_snappy(src, stream); + case compression_type::SNAPPY: return snappy::compress(src); default: CUDF_FAIL("Unsupported compression type"); } } diff --git a/cpp/src/io/utilities/getenv_or.hpp b/cpp/src/io/utilities/getenv_or.hpp index acfd2221797..4d5c3ec6d22 100644 --- a/cpp/src/io/utilities/getenv_or.hpp +++ b/cpp/src/io/utilities/getenv_or.hpp @@ -45,7 +45,7 @@ T getenv_or(std::string_view env_var_name, T default_val) ss.str()); } - if (env_val == nullptr) { return default_val; } + if (env_val == nullptr) { return std::move(default_val); } std::stringstream sstream(env_val); T converted_val; diff --git a/cpp/tests/CMakeLists.txt b/cpp/tests/CMakeLists.txt index fd8cb3f22f2..cfc6a0dc425 100644 --- a/cpp/tests/CMakeLists.txt +++ b/cpp/tests/CMakeLists.txt @@ -298,7 +298,7 @@ ConfigureTest( # ################################################################################################## # * io tests -------------------------------------------------------------------------------------- -ConfigureTest(DECOMPRESSION_TEST io/comp/decomp_test.cpp) +ConfigureTest(COMPRESSION_TEST io/comp/comp_test.cpp) ConfigureTest(ROW_SELECTION_TEST io/row_selection_test.cpp) ConfigureTest( diff --git a/cpp/tests/io/comp/decomp_test.cpp b/cpp/tests/io/comp/comp_test.cpp similarity index 86% rename from cpp/tests/io/comp/decomp_test.cpp rename to cpp/tests/io/comp/comp_test.cpp index 5bbe8b63c47..e3bee708485 100644 --- a/cpp/tests/io/comp/decomp_test.cpp +++ b/cpp/tests/io/comp/comp_test.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -14,7 +14,9 @@ * limitations under the License. */ +#include "io/comp/comp.hpp" #include "io/comp/gpuinflate.hpp" +#include "io/comp/io_uncomp.hpp" #include "io/utilities/hostdevice_vector.hpp" #include @@ -34,6 +36,12 @@ using cudf::io::detail::compression_result; using cudf::io::detail::compression_status; namespace nvcomp = cudf::io::detail::nvcomp; +[[nodiscard]] std::vector vector_from_string(std::string const& str) +{ + return {reinterpret_cast(str.data()), + reinterpret_cast(str.data() + str.size())}; +} + /** * @brief Base test fixture for decompression * @@ -42,12 +50,6 @@ namespace nvcomp = cudf::io::detail::nvcomp; */ template struct DecompressTest : public cudf::test::BaseFixture { - [[nodiscard]] std::vector vector_from_string(std::string const str) const - { - return {reinterpret_cast(str.c_str()), - reinterpret_cast(str.c_str()) + strlen(str.c_str())}; - } - void Decompress(std::vector& decompressed, uint8_t const* compressed, size_t compressed_size) @@ -76,6 +78,11 @@ struct DecompressTest : public cudf::test::BaseFixture { } }; +struct HostCompressTest : public cudf::test::BaseFixture { + HostCompressTest() { setenv("LIBCUDF_HOST_COMPRESSION", "ON", 1); } + ~HostCompressTest() override { unsetenv("LIBCUDF_HOST_COMPRESSION"); } +}; + /** * @brief Derived fixture for GZIP decompression */ @@ -222,4 +229,23 @@ TEST_F(NvcompConfigTest, Decompression) EXPECT_TRUE(decomp_disabled(compression_type::SNAPPY, {false, false})); } +TEST_F(HostCompressTest, SnappyCompression) +{ + std::vector expected; + expected.reserve(8 * (32 << 20)); + for (size_t size = 1; size < 32 << 20; size *= 2) { + // Using number strings to generate data that is compressible, but not trivially so + for (size_t i = size / 2; i < size; ++i) { + auto const num_string = std::to_string(i); + // Keep adding to the test data + expected.insert(expected.end(), num_string.begin(), num_string.end()); + } + auto const compressed = cudf::io::detail::compress( + cudf::io::compression_type::SNAPPY, expected, cudf::get_default_stream()); + auto const decompressed = + cudf::io::detail::decompress(cudf::io::compression_type::SNAPPY, compressed); + EXPECT_EQ(expected, decompressed); + } +} + CUDF_TEST_PROGRAM_MAIN() From 4be30a1104b57726f50a35182f23de6285a245bf Mon Sep 17 00:00:00 2001 From: Robert Maynard Date: Sat, 22 Feb 2025 00:16:33 -0500 Subject: [PATCH 073/129] Require CMake 3.30.4 (#18007) Update CMake minimum required to 3.30.4 across all of RAPIDS Authors: - Robert Maynard (https://github.com/robertmaynard) - Nghia Truong (https://github.com/ttnghia) Approvers: - Nghia Truong (https://github.com/ttnghia) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18007 --- conda/environments/all_cuda-118_arch-x86_64.yaml | 2 +- conda/environments/all_cuda-128_arch-x86_64.yaml | 2 +- conda/recipes/cudf/conda_build_config.yaml | 2 +- conda/recipes/cudf_kafka/conda_build_config.yaml | 2 +- conda/recipes/libcudf/conda_build_config.yaml | 2 +- conda/recipes/pylibcudf/conda_build_config.yaml | 2 +- cpp/CMakeLists.txt | 2 +- cpp/examples/basic/CMakeLists.txt | 2 +- cpp/examples/billion_rows/CMakeLists.txt | 2 +- cpp/examples/interop/CMakeLists.txt | 2 +- cpp/examples/nested_types/CMakeLists.txt | 2 +- cpp/examples/parquet_io/CMakeLists.txt | 2 +- cpp/examples/strings/CMakeLists.txt | 2 +- cpp/libcudf_kafka/CMakeLists.txt | 4 ++-- dependencies.yaml | 2 +- java/ci/Dockerfile.rocky | 2 +- java/src/main/native/CMakeLists.txt | 2 +- python/cudf/CMakeLists.txt | 4 ++-- python/cudf/pyproject.toml | 2 +- python/cudf/udf_cpp/CMakeLists.txt | 4 ++-- python/cudf_kafka/CMakeLists.txt | 4 ++-- python/cudf_kafka/pyproject.toml | 2 +- python/libcudf/CMakeLists.txt | 2 +- python/libcudf/pyproject.toml | 2 +- python/pylibcudf/CMakeLists.txt | 4 ++-- python/pylibcudf/pyproject.toml | 2 +- 26 files changed, 31 insertions(+), 31 deletions(-) diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index 21cc9b8c9e2..cc674732ba4 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -15,7 +15,7 @@ dependencies: - cachetools - clang-tools=16.0.6 - clang==16.0.6 -- cmake>=3.26.4,!=3.30.0 +- cmake>=3.30.4 - cramjam - cubinlinker - cuda-nvtx=11.8 diff --git a/conda/environments/all_cuda-128_arch-x86_64.yaml b/conda/environments/all_cuda-128_arch-x86_64.yaml index 939d6ff9eb9..7593a72cc68 100644 --- a/conda/environments/all_cuda-128_arch-x86_64.yaml +++ b/conda/environments/all_cuda-128_arch-x86_64.yaml @@ -15,7 +15,7 @@ dependencies: - cachetools - clang-tools=16.0.6 - clang==16.0.6 -- cmake>=3.26.4,!=3.30.0 +- cmake>=3.30.4 - cramjam - cuda-cudart-dev - cuda-nvcc diff --git a/conda/recipes/cudf/conda_build_config.yaml b/conda/recipes/cudf/conda_build_config.yaml index a4a6a0910ce..bab277b8f60 100644 --- a/conda/recipes/cudf/conda_build_config.yaml +++ b/conda/recipes/cudf/conda_build_config.yaml @@ -13,7 +13,7 @@ c_stdlib_version: - "2.28" cmake_version: - - ">=3.26.4,!=3.30.0" + - ">=3.30.4" cuda_compiler: - cuda-nvcc # [not os.environ.get("RAPIDS_CUDA_VERSION", "").startswith("11")] diff --git a/conda/recipes/cudf_kafka/conda_build_config.yaml b/conda/recipes/cudf_kafka/conda_build_config.yaml index a4a6a0910ce..bab277b8f60 100644 --- a/conda/recipes/cudf_kafka/conda_build_config.yaml +++ b/conda/recipes/cudf_kafka/conda_build_config.yaml @@ -13,7 +13,7 @@ c_stdlib_version: - "2.28" cmake_version: - - ">=3.26.4,!=3.30.0" + - ">=3.30.4" cuda_compiler: - cuda-nvcc # [not os.environ.get("RAPIDS_CUDA_VERSION", "").startswith("11")] diff --git a/conda/recipes/libcudf/conda_build_config.yaml b/conda/recipes/libcudf/conda_build_config.yaml index 4d75646da78..48b2acf3a02 100644 --- a/conda/recipes/libcudf/conda_build_config.yaml +++ b/conda/recipes/libcudf/conda_build_config.yaml @@ -17,7 +17,7 @@ c_stdlib_version: - "2.28" cmake_version: - - ">=3.26.4,!=3.30.0" + - ">=3.30.4" dlpack_version: - ">=0.8,<1.0" diff --git a/conda/recipes/pylibcudf/conda_build_config.yaml b/conda/recipes/pylibcudf/conda_build_config.yaml index a4a6a0910ce..bab277b8f60 100644 --- a/conda/recipes/pylibcudf/conda_build_config.yaml +++ b/conda/recipes/pylibcudf/conda_build_config.yaml @@ -13,7 +13,7 @@ c_stdlib_version: - "2.28" cmake_version: - - ">=3.26.4,!=3.30.0" + - ">=3.30.4" cuda_compiler: - cuda-nvcc # [not os.environ.get("RAPIDS_CUDA_VERSION", "").startswith("11")] diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index 2e4dd21667e..bb4d20f837c 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -12,7 +12,7 @@ # the License. # ============================================================================= -cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../rapids_config.cmake) include(rapids-cmake) diff --git a/cpp/examples/basic/CMakeLists.txt b/cpp/examples/basic/CMakeLists.txt index 8e89b461e30..455494a40eb 100644 --- a/cpp/examples/basic/CMakeLists.txt +++ b/cpp/examples/basic/CMakeLists.txt @@ -1,6 +1,6 @@ # Copyright (c) 2020-2025, NVIDIA CORPORATION. -cmake_minimum_required(VERSION 3.26.4) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../set_cuda_architecture.cmake) diff --git a/cpp/examples/billion_rows/CMakeLists.txt b/cpp/examples/billion_rows/CMakeLists.txt index 603c8d0b457..f7dbd3e79b1 100644 --- a/cpp/examples/billion_rows/CMakeLists.txt +++ b/cpp/examples/billion_rows/CMakeLists.txt @@ -1,6 +1,6 @@ # Copyright (c) 2024-2025, NVIDIA CORPORATION. -cmake_minimum_required(VERSION 3.26.4) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../set_cuda_architecture.cmake) diff --git a/cpp/examples/interop/CMakeLists.txt b/cpp/examples/interop/CMakeLists.txt index 6f1249beaaa..37a55b98093 100644 --- a/cpp/examples/interop/CMakeLists.txt +++ b/cpp/examples/interop/CMakeLists.txt @@ -1,6 +1,6 @@ # Copyright (c) 2024-2025, NVIDIA CORPORATION. -cmake_minimum_required(VERSION 3.26.4) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../set_cuda_architecture.cmake) diff --git a/cpp/examples/nested_types/CMakeLists.txt b/cpp/examples/nested_types/CMakeLists.txt index e7972d1531b..4df41f2acd6 100644 --- a/cpp/examples/nested_types/CMakeLists.txt +++ b/cpp/examples/nested_types/CMakeLists.txt @@ -1,6 +1,6 @@ # Copyright (c) 2023-2025, NVIDIA CORPORATION. -cmake_minimum_required(VERSION 3.26.4) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../set_cuda_architecture.cmake) diff --git a/cpp/examples/parquet_io/CMakeLists.txt b/cpp/examples/parquet_io/CMakeLists.txt index 17f86fdf5e0..da12b7056fb 100644 --- a/cpp/examples/parquet_io/CMakeLists.txt +++ b/cpp/examples/parquet_io/CMakeLists.txt @@ -1,6 +1,6 @@ # Copyright (c) 2024-2025, NVIDIA CORPORATION. -cmake_minimum_required(VERSION 3.26.4) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../set_cuda_architecture.cmake) diff --git a/cpp/examples/strings/CMakeLists.txt b/cpp/examples/strings/CMakeLists.txt index 9010d495715..a0831488d60 100644 --- a/cpp/examples/strings/CMakeLists.txt +++ b/cpp/examples/strings/CMakeLists.txt @@ -1,6 +1,6 @@ # Copyright (c) 2022-2025, NVIDIA CORPORATION. -cmake_minimum_required(VERSION 3.26.4) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../set_cuda_architecture.cmake) diff --git a/cpp/libcudf_kafka/CMakeLists.txt b/cpp/libcudf_kafka/CMakeLists.txt index 9760ecfe067..26c81e7fd2f 100644 --- a/cpp/libcudf_kafka/CMakeLists.txt +++ b/cpp/libcudf_kafka/CMakeLists.txt @@ -1,5 +1,5 @@ # ============================================================================= -# Copyright (c) 2018-2024, NVIDIA CORPORATION. +# Copyright (c) 2018-2025, NVIDIA CORPORATION. # # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except # in compliance with the License. You may obtain a copy of the License at @@ -11,7 +11,7 @@ # or implied. See the License for the specific language governing permissions and limitations under # the License. # ============================================================================= -cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../../rapids_config.cmake) include(rapids-cmake) diff --git a/dependencies.yaml b/dependencies.yaml index 83f4e96c748..e7840d56880 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -400,7 +400,7 @@ dependencies: common: - output_types: [conda, requirements, pyproject] packages: - - &cmake_ver cmake>=3.26.4,!=3.30.0 + - &cmake_ver cmake>=3.30.4 - &ninja ninja build_all: common: diff --git a/java/ci/Dockerfile.rocky b/java/ci/Dockerfile.rocky index 9f3305278cb..277e33bb8eb 100644 --- a/java/ci/Dockerfile.rocky +++ b/java/ci/Dockerfile.rocky @@ -33,7 +33,7 @@ RUN dnf --enablerepo=powertools install -y scl-utils gcc-toolset-${TOOLSET_VERS RUN mkdir /usr/local/rapids /rapids && chmod 777 /usr/local/rapids /rapids # 3.22.3+: CUDA architecture 'native' support + flexible CMAKE__*_LAUNCHER for ccache -ARG CMAKE_VERSION=3.28.6 +ARG CMAKE_VERSION=3.30.7 # default x86_64 from x86 build, aarch64 cmake for arm build ARG CMAKE_ARCH=x86_64 RUN cd /usr/local && wget --quiet https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-${CMAKE_ARCH}.tar.gz && \ diff --git a/java/src/main/native/CMakeLists.txt b/java/src/main/native/CMakeLists.txt index 3923d8b45e3..1fa6f6d561f 100644 --- a/java/src/main/native/CMakeLists.txt +++ b/java/src/main/native/CMakeLists.txt @@ -11,7 +11,7 @@ # or implied. See the License for the specific language governing permissions and limitations under # the License. # ============================================================================= -cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../../../../rapids_config.cmake) include(rapids-cmake) diff --git a/python/cudf/CMakeLists.txt b/python/cudf/CMakeLists.txt index 7193ada5b93..2a17bc5dbb7 100644 --- a/python/cudf/CMakeLists.txt +++ b/python/cudf/CMakeLists.txt @@ -1,5 +1,5 @@ # ============================================================================= -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. # # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except # in compliance with the License. You may obtain a copy of the License at @@ -12,7 +12,7 @@ # the License. # ============================================================================= -cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../../rapids_config.cmake) include(rapids-cuda) diff --git a/python/cudf/pyproject.toml b/python/cudf/pyproject.toml index d716114cf7e..16cd97677ef 100644 --- a/python/cudf/pyproject.toml +++ b/python/cudf/pyproject.toml @@ -118,7 +118,7 @@ build-backend = "scikit_build_core.build" dependencies-file = "../../dependencies.yaml" matrix-entry = "cuda_suffixed=true" requires = [ - "cmake>=3.26.4,!=3.30.0", + "cmake>=3.30.4", "cython>=3.0.3", "libcudf==25.4.*,>=0.0.0a0", "librmm==25.4.*,>=0.0.0a0", diff --git a/python/cudf/udf_cpp/CMakeLists.txt b/python/cudf/udf_cpp/CMakeLists.txt index fa7855cfc65..9f6b67d0cdc 100644 --- a/python/cudf/udf_cpp/CMakeLists.txt +++ b/python/cudf/udf_cpp/CMakeLists.txt @@ -1,5 +1,5 @@ # ============================================================================= -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. # # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except # in compliance with the License. You may obtain a copy of the License at @@ -12,7 +12,7 @@ # the License. # ============================================================================= -cmake_minimum_required(VERSION 3.26.4) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(rapids-cmake) include(rapids-cpm) diff --git a/python/cudf_kafka/CMakeLists.txt b/python/cudf_kafka/CMakeLists.txt index fd835010c4e..3e12eb6aa41 100644 --- a/python/cudf_kafka/CMakeLists.txt +++ b/python/cudf_kafka/CMakeLists.txt @@ -1,5 +1,5 @@ # ============================================================================= -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. # # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except # in compliance with the License. You may obtain a copy of the License at @@ -12,7 +12,7 @@ # the License. # ============================================================================= -cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../../rapids_config.cmake) diff --git a/python/cudf_kafka/pyproject.toml b/python/cudf_kafka/pyproject.toml index 4a7143e1134..424010e632c 100644 --- a/python/cudf_kafka/pyproject.toml +++ b/python/cudf_kafka/pyproject.toml @@ -83,7 +83,7 @@ build-backend = "scikit_build_core.build" dependencies-file = "../../dependencies.yaml" matrix-entry = "cuda_suffixed=true" requires = [ - "cmake>=3.26.4,!=3.30.0", + "cmake>=3.30.4", "cython>=3.0.3", "ninja", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. diff --git a/python/libcudf/CMakeLists.txt b/python/libcudf/CMakeLists.txt index 259492b98d1..d5450639471 100644 --- a/python/libcudf/CMakeLists.txt +++ b/python/libcudf/CMakeLists.txt @@ -12,7 +12,7 @@ # the License. # ============================================================================= -cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../../rapids_config.cmake) diff --git a/python/libcudf/pyproject.toml b/python/libcudf/pyproject.toml index 18aa824c6df..01fe6097936 100644 --- a/python/libcudf/pyproject.toml +++ b/python/libcudf/pyproject.toml @@ -79,7 +79,7 @@ build-backend = "scikit_build_core.build" dependencies-file = "../../dependencies.yaml" matrix-entry = "cuda_suffixed=true;use_cuda_wheels=true" requires = [ - "cmake>=3.26.4,!=3.30.0", + "cmake>=3.30.4", "libkvikio==25.4.*,>=0.0.0a0", "librmm==25.4.*,>=0.0.0a0", "ninja", diff --git a/python/pylibcudf/CMakeLists.txt b/python/pylibcudf/CMakeLists.txt index a4b831790fb..fe6e73a3f14 100644 --- a/python/pylibcudf/CMakeLists.txt +++ b/python/pylibcudf/CMakeLists.txt @@ -1,5 +1,5 @@ # ============================================================================= -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. # # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except # in compliance with the License. You may obtain a copy of the License at @@ -12,7 +12,7 @@ # the License. # ============================================================================= -cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR) +cmake_minimum_required(VERSION 3.30.4 FATAL_ERROR) include(../../rapids_config.cmake) include(rapids-cuda) diff --git a/python/pylibcudf/pyproject.toml b/python/pylibcudf/pyproject.toml index 2f846b5f0b9..939da65c1ec 100644 --- a/python/pylibcudf/pyproject.toml +++ b/python/pylibcudf/pyproject.toml @@ -109,7 +109,7 @@ build-backend = "scikit_build_core.build" dependencies-file = "../../dependencies.yaml" matrix-entry = "cuda_suffixed=true" requires = [ - "cmake>=3.26.4,!=3.30.0", + "cmake>=3.30.4", "cython>=3.0.3", "libcudf==25.4.*,>=0.0.0a0", "librmm==25.4.*,>=0.0.0a0", From d0e219ef36e51e07573c7650819a26808b60c9a4 Mon Sep 17 00:00:00 2001 From: ustcfy <96854327+ustcfy@users.noreply.github.com> Date: Mon, 24 Feb 2025 13:07:30 +0800 Subject: [PATCH 074/129] [FEA] Expose `stripe_size_rows` setting for `ORCWriterOptions` (#17927) closes https://github.com/rapidsai/cudf/issues/17785 This PR exposes the `stripe_size_rows` setting for the `ORCWriterOptions` Java interface. Exposing this interface is solely for the convenience of conducting some tests. Authors: - https://github.com/ustcfy - Nghia Truong (https://github.com/ttnghia) Approvers: - Nghia Truong (https://github.com/ttnghia) - Chong Gao (https://github.com/res-life) URL: https://github.com/rapidsai/cudf/pull/17927 --- .../java/ai/rapids/cudf/ORCWriterOptions.java | 19 ++++++++++++++++++- java/src/main/java/ai/rapids/cudf/Table.java | 4 ++++ java/src/main/native/src/TableJni.cpp | 4 ++++ 3 files changed, 26 insertions(+), 1 deletion(-) diff --git a/java/src/main/java/ai/rapids/cudf/ORCWriterOptions.java b/java/src/main/java/ai/rapids/cudf/ORCWriterOptions.java index 372f919532e..009f5e12815 100644 --- a/java/src/main/java/ai/rapids/cudf/ORCWriterOptions.java +++ b/java/src/main/java/ai/rapids/cudf/ORCWriterOptions.java @@ -1,6 +1,6 @@ /* * - * Copyright (c) 2019-2021, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -23,17 +23,34 @@ * that will be used by the ORC writer to write the file. */ public class ORCWriterOptions extends CompressionMetadataWriterOptions { + private int stripeSizeRows; private ORCWriterOptions(Builder builder) { super(builder); + this.stripeSizeRows = builder.stripeSizeRows; } public static Builder builder() { return new Builder(); } + public int getStripeSizeRows() { + return stripeSizeRows; + } + public static class Builder extends CompressionMetadataWriterOptions.Builder { + // < 1M rows default orc stripe rows, defined in cudf/cpp/include/cudf/io/orc.hpp + private int stripeSizeRows = 1000000; + + public Builder withStripeSizeRows(int stripeSizeRows) { + // maximum stripe size cannot be smaller than 512 + if (stripeSizeRows < 512) { + throw new IllegalArgumentException("Maximum stripe size cannot be smaller than 512"); + } + this.stripeSizeRows = stripeSizeRows; + return this; + } public ORCWriterOptions build() { return new ORCWriterOptions(this); diff --git a/java/src/main/java/ai/rapids/cudf/Table.java b/java/src/main/java/ai/rapids/cudf/Table.java index 298f2cff6f3..422989143c7 100644 --- a/java/src/main/java/ai/rapids/cudf/Table.java +++ b/java/src/main/java/ai/rapids/cudf/Table.java @@ -475,6 +475,7 @@ private static native long writeORCFileBegin(String[] columnNames, int compression, int[] precisions, boolean[] isMapValues, + int stripeSizeRows, String filename) throws CudfException; /** @@ -501,6 +502,7 @@ private static native long writeORCBufferBegin(String[] columnNames, int compression, int[] precisions, boolean[] isMapValues, + int stripeSizeRows, HostBufferConsumer consumer, HostMemoryAllocator hostMemoryAllocator ) throws CudfException; @@ -1823,6 +1825,7 @@ private ORCTableWriter(ORCWriterOptions options, File outputFile) { options.getCompressionType().nativeId, options.getFlatPrecision(), options.getFlatIsMap(), + options.getStripeSizeRows(), outputFile.getAbsolutePath())); this.consumer = null; } @@ -1838,6 +1841,7 @@ private ORCTableWriter(ORCWriterOptions options, HostBufferConsumer consumer, options.getCompressionType().nativeId, options.getFlatPrecision(), options.getFlatIsMap(), + options.getStripeSizeRows(), consumer, hostMemoryAllocator)); this.consumer = consumer; } diff --git a/java/src/main/native/src/TableJni.cpp b/java/src/main/native/src/TableJni.cpp index 50c6ae842f4..e1b487b1f7c 100644 --- a/java/src/main/native/src/TableJni.cpp +++ b/java/src/main/native/src/TableJni.cpp @@ -2480,6 +2480,7 @@ Java_ai_rapids_cudf_Table_writeORCBufferBegin(JNIEnv* env, jint j_compression, jintArray j_precisions, jbooleanArray j_is_map, + jint j_stripe_size_rows, jobject consumer, jobject host_memory_allocator) { @@ -2535,6 +2536,7 @@ Java_ai_rapids_cudf_Table_writeORCBufferBegin(JNIEnv* env, .enable_statistics(ORC_STATISTICS_ROW_GROUP) .key_value_metadata(kv_metadata) .compression_statistics(stats) + .stripe_size_rows(j_stripe_size_rows) .build(); auto writer_ptr = std::make_unique(opts); cudf::jni::native_orc_writer_handle* ret = new cudf::jni::native_orc_writer_handle( @@ -2555,6 +2557,7 @@ JNIEXPORT long JNICALL Java_ai_rapids_cudf_Table_writeORCFileBegin(JNIEnv* env, jint j_compression, jintArray j_precisions, jbooleanArray j_is_map, + jint j_stripe_size_rows, jstring j_output_path) { JNI_NULL_CHECK(env, j_col_names, "null columns", 0); @@ -2606,6 +2609,7 @@ JNIEXPORT long JNICALL Java_ai_rapids_cudf_Table_writeORCFileBegin(JNIEnv* env, .enable_statistics(ORC_STATISTICS_ROW_GROUP) .key_value_metadata(kv_metadata) .compression_statistics(stats) + .stripe_size_rows(j_stripe_size_rows) .build(); auto writer_ptr = std::make_unique(opts); cudf::jni::native_orc_writer_handle* ret = From d4c62efb3d326a71c1cd2504ee46817ba2d3975f Mon Sep 17 00:00:00 2001 From: "Richard (Rick) Zamora" Date: Mon, 24 Feb 2025 08:46:10 -0600 Subject: [PATCH 075/129] Fix `dask_cudf.to_orc` deprecation (#18038) **TLDR**: This PR adds a trivial change to avoid multiple deprecation-warning stages when `import dask_cudf.to_orc` is called. Dask cuDF uses a `_deprecated_api` utility to discourage users from importing IO functions from `dask_cudf.io`. We also encourage users to leverage `DataFrame` methods in lieu of stand-alone functions for output IO. For example, we want people to call `df.to_orc(...)` (and to avoid patterns like `from dask_cudf import to_orc`). It turns out that the calling `import dask_cudf.to_orc` currently results in **two** deprecation warnings being raised before the "real" `dask_cudf.io.orc.to_orc` implementation is reached. This is because the first `dask_cudf.to_orc` import is incorrectly redirected to `dask_cudf.io.to_orc` (rather than `dask_cudf.io.orc.to_orc`). This subtle bug doesn't seem to cause problems in CI right now, but I do see local failures in `test_orc.py::test_deprecated_api_paths` with a slightly different environment (not sure of the specific differences atm). Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Murray (https://github.com/Matt711) - Tom Augspurger (https://github.com/TomAugspurger) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/18038 --- python/dask_cudf/dask_cudf/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/dask_cudf/dask_cudf/__init__.py b/python/dask_cudf/dask_cudf/__init__.py index 9afe93a6e80..0cdb4525207 100644 --- a/python/dask_cudf/dask_cudf/__init__.py +++ b/python/dask_cudf/dask_cudf/__init__.py @@ -37,7 +37,7 @@ def read_parquet(*args, **kwargs): read_text = DataFrame.read_text to_orc = _deprecated_api( "dask_cudf.to_orc", - new_api="dask_cudf.io.to_orc", + new_api="dask_cudf.io.orc.to_orc", rec="Please use DataFrame.to_orc instead.", ) From 3c698d00c9e8b8b02f2ad596bcc757430a9a6916 Mon Sep 17 00:00:00 2001 From: Michael Schellenberger Costa Date: Mon, 24 Feb 2025 19:01:16 +0100 Subject: [PATCH 076/129] Use the right version macro `CCCL_MAJOR_VERSION` (#18073) I accidentally used `CCCL_VERSION_MAJOR`, which is used in our cmake scripts Authors: - Michael Schellenberger Costa (https://github.com/miscco) Approvers: - David Wendt (https://github.com/davidwendt) - Bradley Dice (https://github.com/bdice) - Yunsong Wang (https://github.com/PointKernel) URL: https://github.com/rapidsai/cudf/pull/18073 --- cpp/src/io/fst/dispatch_dfa.cuh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cpp/src/io/fst/dispatch_dfa.cuh b/cpp/src/io/fst/dispatch_dfa.cuh index d8be747d93d..e8709b0d7bb 100644 --- a/cpp/src/io/fst/dispatch_dfa.cuh +++ b/cpp/src/io/fst/dispatch_dfa.cuh @@ -391,7 +391,7 @@ struct DispatchFSM : DeviceFSMPolicy { // Alias the temporary allocations from the single storage blob (or compute the necessary size // of the blob) // TODO (@miscco): remove this once rapids moves to CCCL 2.8 -#if CCCL_VERSION_MAJOR >= 3 +#if CCCL_MAJOR_VERSION >= 3 error = cub::detail::AliasTemporaries( #else // ^^^ CCCL 3.x ^^^ / vvv CCCL 2.x vvv error = cub::AliasTemporaries( From 2b6dcb0faa28a51989e32da6dd78378778b72198 Mon Sep 17 00:00:00 2001 From: GALI PREM SAGAR Date: Mon, 24 Feb 2025 12:05:26 -0600 Subject: [PATCH 077/129] Enable pytest-xdist runs for py-polars tests (#18016) This PR reduces CI time by parallelizing pytests execution for `py-polars` tests. This PR: ``` ========== 14748 passed, 28 skipped, 140 xfailed in 112.37s (0:01:52) ========== ``` On `branch-25.04`: ``` == 14748 passed, 28 skipped, 58 deselected, 140 xfailed in 529.08s (0:08:49) === ``` Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - Gil Forsyth (https://github.com/gforsyth) URL: https://github.com/rapidsai/cudf/pull/18016 --- ci/run_cudf_polars_polars_tests.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ci/run_cudf_polars_polars_tests.sh b/ci/run_cudf_polars_polars_tests.sh index dfabe6093a9..757f4eb94c4 100755 --- a/ci/run_cudf_polars_polars_tests.sh +++ b/ci/run_cudf_polars_polars_tests.sh @@ -48,7 +48,9 @@ python -m pytest \ --cache-clear \ -m "" \ -p cudf_polars.testing.plugin \ - -v \ + -n 8 \ + --dist=worksteal \ + -vv \ --tb=native \ $DESELECTED_TESTS_STR \ "$@" \ From bcff1f7c55eb7077fa42e8e1ef231d0542fadd46 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Mon, 24 Feb 2025 14:44:56 -0500 Subject: [PATCH 078/129] Add a slice expression to polars IR (#18050) Closes https://github.com/rapidsai/cudf/issues/18051 Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/18050 --- .../cudf_polars/containers/dataframe.py | 4 +- python/cudf_polars/cudf_polars/dsl/expr.py | 4 +- .../cudf_polars/dsl/expressions/slicing.py | 51 +++++++++++++++++++ .../cudf_polars/cudf_polars/dsl/translate.py | 14 +++++ .../tests/expressions/test_slice.py | 24 +++++++++ 5 files changed, 95 insertions(+), 2 deletions(-) create mode 100644 python/cudf_polars/cudf_polars/dsl/expressions/slicing.py create mode 100644 python/cudf_polars/tests/expressions/test_slice.py diff --git a/python/cudf_polars/cudf_polars/containers/dataframe.py b/python/cudf_polars/cudf_polars/containers/dataframe.py index a605b476197..a2b496b8cfe 100644 --- a/python/cudf_polars/cudf_polars/containers/dataframe.py +++ b/python/cudf_polars/cudf_polars/containers/dataframe.py @@ -295,7 +295,7 @@ def filter(self, mask: Column) -> Self: table = plc.stream_compaction.apply_boolean_mask(self.table, mask.obj) return type(self).from_table(table, self.column_names).sorted_like(self) - def slice(self, zlice: tuple[int, int] | None) -> Self: + def slice(self, zlice: tuple[int, int | None] | None) -> Self: """ Slice a dataframe. @@ -312,6 +312,8 @@ def slice(self, zlice: tuple[int, int] | None) -> Self: if zlice is None: return self start, length = zlice + if length is None: + length = self.num_rows if start < 0: start += self.num_rows # Polars implementation wraps negative start by num_rows, then diff --git a/python/cudf_polars/cudf_polars/dsl/expr.py b/python/cudf_polars/cudf_polars/dsl/expr.py index 98d49e36fb1..3ba54543a3e 100644 --- a/python/cudf_polars/cudf_polars/dsl/expr.py +++ b/python/cudf_polars/cudf_polars/dsl/expr.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 # TODO: remove need for this # ruff: noqa: D101 @@ -30,6 +30,7 @@ from cudf_polars.dsl.expressions.literal import Literal, LiteralColumn from cudf_polars.dsl.expressions.rolling import GroupedRollingWindow, RollingWindow from cudf_polars.dsl.expressions.selection import Filter, Gather +from cudf_polars.dsl.expressions.slicing import Slice from cudf_polars.dsl.expressions.sorting import Sort, SortBy from cudf_polars.dsl.expressions.string import StringFunction from cudf_polars.dsl.expressions.ternary import Ternary @@ -53,6 +54,7 @@ "LiteralColumn", "NamedExpr", "RollingWindow", + "Slice", "Sort", "SortBy", "StringFunction", diff --git a/python/cudf_polars/cudf_polars/dsl/expressions/slicing.py b/python/cudf_polars/cudf_polars/dsl/expressions/slicing.py new file mode 100644 index 00000000000..2d3640cce86 --- /dev/null +++ b/python/cudf_polars/cudf_polars/dsl/expressions/slicing.py @@ -0,0 +1,51 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. +# SPDX-License-Identifier: Apache-2.0 +# TODO: remove need for this +# ruff: noqa: D101 +"""Slicing DSL nodes.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from cudf_polars.dsl.expressions.base import ( + ExecutionContext, + Expr, +) + +if TYPE_CHECKING: + from collections.abc import Mapping + + import pylibcudf as plc + + from cudf_polars.containers import Column, DataFrame + + +__all__ = ["Slice"] + + +class Slice(Expr): + __slots__ = ("length", "offset") + _non_child = ("dtype", "offset", "length") + + def __init__( + self, + dtype: plc.DataType, + offset: int, + length: int, + column: Expr, + ) -> None: + self.dtype = dtype + self.offset = offset + self.length = length + self.children = (column,) + + def do_evaluate( + self, + df: DataFrame, + *, + context: ExecutionContext = ExecutionContext.FRAME, + mapping: Mapping[Expr, Column] | None = None, + ) -> Column: + """Evaluate this expression given a dataframe for context.""" + return df.slice((self.offset, self.length)).columns[0] diff --git a/python/cudf_polars/cudf_polars/dsl/translate.py b/python/cudf_polars/cudf_polars/dsl/translate.py index 22f97f2bf52..369328d3a8c 100644 --- a/python/cudf_polars/cudf_polars/dsl/translate.py +++ b/python/cudf_polars/cudf_polars/dsl/translate.py @@ -690,6 +690,20 @@ def _(node: pl_expr.SortBy, translator: Translator, dtype: plc.DataType) -> expr ) +@_translate_expr.register +def _(node: pl_expr.Slice, translator: Translator, dtype: plc.DataType) -> expr.Expr: + offset = translator.translate_expr(n=node.offset) + length = translator.translate_expr(n=node.length) + assert isinstance(offset, expr.Literal) + assert isinstance(length, expr.Literal) + return expr.Slice( + dtype, + offset.value.as_py(), + length.value.as_py(), + translator.translate_expr(n=node.input), + ) + + @_translate_expr.register def _(node: pl_expr.Gather, translator: Translator, dtype: plc.DataType) -> expr.Expr: return expr.Gather( diff --git a/python/cudf_polars/tests/expressions/test_slice.py b/python/cudf_polars/tests/expressions/test_slice.py new file mode 100644 index 00000000000..9873be2455f --- /dev/null +++ b/python/cudf_polars/tests/expressions/test_slice.py @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. +# SPDX-License-Identifier: Apache-2.0 +from __future__ import annotations + +import pytest + +import polars as pl + +from cudf_polars.testing.asserts import assert_gpu_result_equal + + +@pytest.mark.parametrize( + "zlice", + [ + (1,), + (1, 3), + (-1,), + ], +) +def test_slice(zlice): + df = pl.LazyFrame({"a": [0, 1, 2, 3], "b": [1, 2, 3, 4]}) + q = df.select(pl.col("a").slice(*zlice)) + + assert_gpu_result_equal(q) From 6590cc200652ef813dfdc8025e674a3570af6ae1 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Mon, 24 Feb 2025 13:09:58 -0800 Subject: [PATCH 079/129] Support melt(ignore_index=False) (#18080) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Additionally, refactors `melt` to avoid Series/DataFrame constructors by operating on columns and passing the result to `_from_data` ```python In [1]: import numpy as np, cudf In [2]: df = cudf.DataFrame(np.ones((1000, 1000))) In [3]: %timeit df.melt(id_vars=range(50, 300)) # this PR 1.35 s ± 12.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In [3]: %timeit df.melt(id_vars=range(50, 300)) # branch-25.04 24.8 s ± 47 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ``` Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18080 --- python/cudf/cudf/core/reshape.py | 54 +++++++++++++++----------- python/cudf/cudf/tests/test_reshape.py | 25 +++++++++--- 2 files changed, 51 insertions(+), 28 deletions(-) diff --git a/python/cudf/cudf/core/reshape.py b/python/cudf/cudf/core/reshape.py index 21f8dc9bb8a..c5d2fd349e9 100644 --- a/python/cudf/cudf/core/reshape.py +++ b/python/cudf/cudf/core/reshape.py @@ -14,11 +14,18 @@ from cudf.api.extensions import no_default from cudf.api.types import is_scalar from cudf.core._compat import PANDAS_LT_300 -from cudf.core.column import ColumnBase, as_column, column_empty +from cudf.core.column import ( + ColumnBase, + as_column, + column_empty, + concat_columns, +) from cudf.core.column_accessor import ColumnAccessor from cudf.utils.dtypes import SIZE_TYPE_DTYPE, min_unsigned_type if TYPE_CHECKING: + from collections.abc import Hashable + from cudf._typing import DtypeObj _AXIS_MAP = {0: 0, 1: 1, "index": 0, "columns": 1} @@ -534,14 +541,14 @@ def concat( def melt( - frame, + frame: cudf.DataFrame, id_vars=None, value_vars=None, var_name=None, - value_name="value", + value_name: Hashable = "value", col_level=None, ignore_index: bool = True, -): +) -> cudf.DataFrame: """Unpivots a DataFrame from wide format to long format, optionally leaving identifier variables set. @@ -605,14 +612,12 @@ def melt( """ if col_level is not None: raise NotImplementedError("col_level != None is not supported yet.") - if ignore_index is not True: - raise NotImplementedError("ignore_index is currently not supported.") # Arg cleaning # id_vars if id_vars is not None: - if cudf.api.types.is_scalar(id_vars): + if is_scalar(id_vars): id_vars = [id_vars] id_vars = list(id_vars) missing = set(id_vars) - set(frame._column_names) @@ -626,7 +631,7 @@ def melt( # value_vars if value_vars is not None: - if cudf.api.types.is_scalar(value_vars): + if is_scalar(value_vars): value_vars = [value_vars] value_vars = list(value_vars) missing = set(value_vars) - set(frame._column_names) @@ -643,7 +648,7 @@ def melt( # Error for unimplemented support for datatype if any( isinstance(frame[col].dtype, cudf.CategoricalDtype) - for col in id_vars + value_vars + for col in itertools.chain(id_vars, value_vars) ): raise NotImplementedError( "Categorical columns are not yet supported for function" @@ -668,15 +673,14 @@ def melt( N = len(frame) K = len(value_vars) - def _tile(A, reps): - series_list = [A] * reps + def _tile(base_col: ColumnBase, reps: int) -> ColumnBase: if reps > 0: - return cudf.Series._concat(objs=series_list, index=False) + return concat_columns([base_col] * reps) else: - return cudf.Series([], dtype=A.dtype) + return column_empty(0, dtype=base_col.dtype) # Step 1: tile id_vars - mdata = {col: _tile(frame[col], K) for col in id_vars} + mdata = {col: _tile(frame[col]._column, K) for col in id_vars} # Step 2: add variable nval = len(value_vars) @@ -687,23 +691,27 @@ def _tile(A, reps): if not value_vars: # TODO: Use frame._data.label_dtype when it's more consistently set - var_data = cudf.Series( - value_vars, dtype=frame._data.to_pandas_index.dtype + var_data = column_empty( + 0, dtype=cudf.dtype(frame._data.to_pandas_index.dtype) ) else: - var_data = ( - cudf.Series(value_vars) - .take(np.repeat(np.arange(nval, dtype=dtype), N)) - .reset_index(drop=True) + var_data = as_column(value_vars).take( + as_column(np.repeat(np.arange(nval, dtype=dtype), N)), + check_bounds=False, ) mdata[var_name] = var_data # Step 3: add values - mdata[value_name] = cudf.Series._concat( - objs=[frame[val] for val in value_vars], index=False + mdata[value_name] = concat_columns( + [frame[val]._column for val in value_vars] ) - return cudf.DataFrame(mdata) + result = cudf.DataFrame._from_data(mdata) + if not ignore_index: + taker = np.tile(np.arange(len(frame)), frame.shape[1] - len(id_vars)) + result.index = frame.index.take(taker) + + return result def get_dummies( diff --git a/python/cudf/cudf/tests/test_reshape.py b/python/cudf/cudf/tests/test_reshape.py index 5cebdf37c9f..7fbe072dde7 100644 --- a/python/cudf/cudf/tests/test_reshape.py +++ b/python/cudf/cudf/tests/test_reshape.py @@ -1,4 +1,4 @@ -# Copyright (c) 2021-2024, NVIDIA CORPORATION. +# Copyright (c) 2021-2025, NVIDIA CORPORATION. import re from itertools import chain @@ -40,7 +40,10 @@ @pytest.mark.parametrize("num_rows", [1, 2, 100]) @pytest.mark.parametrize("dtype", NUMERIC_TYPES + DATETIME_TYPES) @pytest.mark.parametrize("nulls", ["none", "some", "all"]) -def test_melt(nulls, num_id_vars, num_value_vars, num_rows, dtype): +@pytest.mark.parametrize("ignore_index", [True, False]) +def test_melt( + nulls, num_id_vars, num_value_vars, num_rows, dtype, ignore_index +): if dtype not in ["float32", "float64"] and nulls in ["some", "all"]: pytest.skip(reason="nulls not supported in dtype: " + dtype) @@ -72,10 +75,22 @@ def test_melt(nulls, num_id_vars, num_value_vars, num_rows, dtype): gdf = cudf.from_pandas(pdf) - got = cudf.melt(frame=gdf, id_vars=id_vars, value_vars=value_vars) - got_from_melt_method = gdf.melt(id_vars=id_vars, value_vars=value_vars) + got = cudf.melt( + frame=gdf, + id_vars=id_vars, + value_vars=value_vars, + ignore_index=ignore_index, + ) + got_from_melt_method = gdf.melt( + id_vars=id_vars, value_vars=value_vars, ignore_index=ignore_index + ) - expect = pd.melt(frame=pdf, id_vars=id_vars, value_vars=value_vars) + expect = pd.melt( + frame=pdf, + id_vars=id_vars, + value_vars=value_vars, + ignore_index=ignore_index, + ) assert_eq(expect, got) From 5c82909aa4f4f5030627b62e0d0587b0df7c1e44 Mon Sep 17 00:00:00 2001 From: Kyle Edwards Date: Mon, 24 Feb 2025 17:04:44 -0500 Subject: [PATCH 080/129] Remove `FindCUDAToolkit` backport (#18081) In 81cd4a00, `FindCUDAToolkit.cmake` was backported from CMake 3.31 to get https://gitlab.kitware.com/cmake/cmake/-/commit/b38a8e77cb3c8401b3022a68f07a4fd77b290524, which was introduced in CMake 3.28. Since we now require CMake 3.30, we no longer need the backport. Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18081 --- cpp/cmake/Modules/FindCUDAToolkit.cmake | 1437 ----------------------- 1 file changed, 1437 deletions(-) delete mode 100644 cpp/cmake/Modules/FindCUDAToolkit.cmake diff --git a/cpp/cmake/Modules/FindCUDAToolkit.cmake b/cpp/cmake/Modules/FindCUDAToolkit.cmake deleted file mode 100644 index 6f0272aa2d7..00000000000 --- a/cpp/cmake/Modules/FindCUDAToolkit.cmake +++ /dev/null @@ -1,1437 +0,0 @@ -# CMake - Cross Platform Makefile Generator -# Copyright 2000-2024 Kitware, Inc. and Contributors -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions -# are met: -# -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# -# * Neither the name of Kitware, Inc. nor the names of Contributors -# may be used to endorse or promote products derived from this -# software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -#[=======================================================================[.rst: -FindCUDAToolkit ---------------- - -.. versionadded:: 3.17 - -This script locates the NVIDIA CUDA toolkit and the associated libraries, but -does not require the ``CUDA`` language be enabled for a given project. This -module does not search for the NVIDIA CUDA Samples. - -.. versionadded:: 3.19 - QNX support. - -Search Behavior -^^^^^^^^^^^^^^^ - -The CUDA Toolkit search behavior uses the following order: - -1. If the ``CUDA`` language has been enabled we will use the directory - containing the compiler as the first search location for ``nvcc``. - -2. If the variable :variable:`CMAKE_CUDA_COMPILER _COMPILER>` or - the environment variable :envvar:`CUDACXX` is defined, it will be used - as the path to the ``nvcc`` executable. - -3. If the ``CUDAToolkit_ROOT`` cmake configuration variable (e.g., - ``-DCUDAToolkit_ROOT=/some/path``) *or* environment variable is defined, it - will be searched. If both an environment variable **and** a - configuration variable are specified, the *configuration* variable takes - precedence. - - The directory specified here must be such that the executable ``nvcc`` or - the appropriate ``version.txt`` or ``version.json`` file can be found - underneath the specified directory. - -4. If the CUDA_PATH environment variable is defined, it will be searched - for ``nvcc``. - -5. The user's path is searched for ``nvcc`` using :command:`find_program`. If - this is found, no subsequent search attempts are performed. Users are - responsible for ensuring that the first ``nvcc`` to show up in the path is - the desired path in the event that multiple CUDA Toolkits are installed. - -6. On Unix systems, if the symbolic link ``/usr/local/cuda`` exists, this is - used. No subsequent search attempts are performed. No default symbolic link - location exists for the Windows platform. - -7. The platform specific default install locations are searched. If exactly one - candidate is found, this is used. The default CUDA Toolkit install locations - searched are: - - +-------------+-------------------------------------------------------------+ - | Platform | Search Pattern | - +=============+=============================================================+ - | macOS | ``/Developer/NVIDIA/CUDA-X.Y`` | - +-------------+-------------------------------------------------------------+ - | Other Unix | ``/usr/local/cuda-X.Y`` | - +-------------+-------------------------------------------------------------+ - | Windows | ``C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\vX.Y`` | - +-------------+-------------------------------------------------------------+ - - Where ``X.Y`` would be a specific version of the CUDA Toolkit, such as - ``/usr/local/cuda-9.0`` or - ``C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0`` - - .. note:: - - When multiple CUDA Toolkits are installed in the default location of a - system (e.g., both ``/usr/local/cuda-9.0`` and ``/usr/local/cuda-10.0`` - exist but the ``/usr/local/cuda`` symbolic link does **not** exist), this - package is marked as **not** found. - - There are too many factors involved in making an automatic decision in - the presence of multiple CUDA Toolkits being installed. In this - situation, users are encouraged to either (1) set ``CUDAToolkit_ROOT`` or - (2) ensure that the correct ``nvcc`` executable shows up in ``$PATH`` for - :command:`find_program` to find. - -Arguments -^^^^^^^^^ - -``[]`` - The ``[]`` argument requests a version with which the package found - should be compatible. See :ref:`find_package version format ` - for more details. - -Options -^^^^^^^ - -``REQUIRED`` - If specified, configuration will error if a suitable CUDA Toolkit is not - found. - -``QUIET`` - If specified, the search for a suitable CUDA Toolkit will not produce any - messages. - -``EXACT`` - If specified, the CUDA Toolkit is considered found only if the exact - ``VERSION`` specified is recovered. - -Imported targets -^^^^^^^^^^^^^^^^ - -An :ref:`imported target ` named ``CUDA::toolkit`` is provided. - -This module defines :prop_tgt:`IMPORTED` targets for each -of the following libraries that are part of the CUDAToolkit: - -- :ref:`CUDA Runtime Library` -- :ref:`CUDA Driver Library` -- :ref:`cuBLAS` -- :ref:`cuDLA` -- :ref:`cuFile` -- :ref:`cuFFT` -- :ref:`cuRAND` -- :ref:`cuSOLVER` -- :ref:`cuSPARSE` -- :ref:`cuPTI` -- :ref:`NPP` -- :ref:`nvBLAS` -- :ref:`nvGRAPH` -- :ref:`nvJPEG` -- :ref:`nvidia-ML` -- :ref:`nvPTX Compiler` -- :ref:`nvRTC` -- :ref:`nvJitLink` -- :ref:`nvFatBin` -- :ref:`nvToolsExt` -- :ref:`nvtx3` -- :ref:`OpenCL` -- :ref:`cuLIBOS` - -.. _`cuda_toolkit_rt_lib`: - -CUDA Runtime Library -"""""""""""""""""""" - -The CUDA Runtime library (cudart) are what most applications will typically -need to link against to make any calls such as `cudaMalloc`, and `cudaFree`. - -Targets Created: - -- ``CUDA::cudart`` -- ``CUDA::cudart_static`` - -.. _`cuda_toolkit_driver_lib`: - -CUDA Driver Library -"""""""""""""""""""" - -The CUDA Driver library (cuda) are used by applications that use calls -such as `cuMemAlloc`, and `cuMemFree`. - -Targets Created: - -- ``CUDA::cuda_driver`` - -.. _`cuda_toolkit_cuBLAS`: - -cuBLAS -"""""" - -The `cuBLAS `_ library. - -Targets Created: - -- ``CUDA::cublas`` -- ``CUDA::cublas_static`` -- ``CUDA::cublasLt`` starting in CUDA 10.1 -- ``CUDA::cublasLt_static`` starting in CUDA 10.1 - -.. _`cuda_toolkit_cuDLA`: - -cuDLA -"""""" - -.. versionadded:: 3.27 - -The NVIDIA Tegra Deep Learning Accelerator `cuDLA `_ library. - -Targets Created: - -- ``CUDA::cudla`` starting in CUDA 11.6 - -.. _`cuda_toolkit_cuFile`: - -cuFile -"""""" - -.. versionadded:: 3.25 - -The NVIDIA GPUDirect Storage `cuFile `_ library. - -Targets Created: - -- ``CUDA::cuFile`` starting in CUDA 11.4 -- ``CUDA::cuFile_static`` starting in CUDA 11.4 -- ``CUDA::cuFile_rdma`` starting in CUDA 11.4 -- ``CUDA::cuFile_rdma_static`` starting in CUDA 11.4 - -.. _`cuda_toolkit_cuFFT`: - -cuFFT -""""" - -The `cuFFT `_ library. - -Targets Created: - -- ``CUDA::cufft`` -- ``CUDA::cufftw`` -- ``CUDA::cufft_static`` -- ``CUDA::cufft_static_nocallback`` starting in CUDA 9.2, requires CMake 3.23+ -- ``CUDA::cufftw_static`` - -cuRAND -"""""" - -The `cuRAND `_ library. - -Targets Created: - -- ``CUDA::curand`` -- ``CUDA::curand_static`` - -.. _`cuda_toolkit_cuSOLVER`: - -cuSOLVER -"""""""" - -The `cuSOLVER `_ library. - -Targets Created: - -- ``CUDA::cusolver`` -- ``CUDA::cusolver_static`` - -.. _`cuda_toolkit_cuSPARSE`: - -cuSPARSE -"""""""" - -The `cuSPARSE `_ library. - -Targets Created: - -- ``CUDA::cusparse`` -- ``CUDA::cusparse_static`` - -.. _`cuda_toolkit_cupti`: - -cupti -""""" - -The `NVIDIA CUDA Profiling Tools Interface `_. - -Targets Created: - -- ``CUDA::cupti`` -- ``CUDA::cupti_static`` - -.. versionadded:: 3.27 - - - ``CUDA::nvperf_host`` starting in CUDA 10.2 - - ``CUDA::nvperf_host_static`` starting in CUDA 10.2 - - ``CUDA::nvperf_target`` starting in CUDA 10.2 - - ``CUDA::pcsamplingutil`` starting in CUDA 11.3 - -.. _`cuda_toolkit_NPP`: - -NPP -""" - -The `NPP `_ libraries. - -Targets Created: - -- `nppc`: - - - ``CUDA::nppc`` - - ``CUDA::nppc_static`` - -- `nppial`: Arithmetic and logical operation functions in `nppi_arithmetic_and_logical_operations.h` - - - ``CUDA::nppial`` - - ``CUDA::nppial_static`` - -- `nppicc`: Color conversion and sampling functions in `nppi_color_conversion.h` - - - ``CUDA::nppicc`` - - ``CUDA::nppicc_static`` - -- `nppicom`: JPEG compression and decompression functions in `nppi_compression_functions.h` - Removed starting in CUDA 11.0, use :ref:`nvJPEG` instead. - - - ``CUDA::nppicom`` - - ``CUDA::nppicom_static`` - -- `nppidei`: Data exchange and initialization functions in `nppi_data_exchange_and_initialization.h` - - - ``CUDA::nppidei`` - - ``CUDA::nppidei_static`` - -- `nppif`: Filtering and computer vision functions in `nppi_filter_functions.h` - - - ``CUDA::nppif`` - - ``CUDA::nppif_static`` - -- `nppig`: Geometry transformation functions found in `nppi_geometry_transforms.h` - - - ``CUDA::nppig`` - - ``CUDA::nppig_static`` - -- `nppim`: Morphological operation functions found in `nppi_morphological_operations.h` - - - ``CUDA::nppim`` - - ``CUDA::nppim_static`` - -- `nppist`: Statistics and linear transform in `nppi_statistics_functions.h` and `nppi_linear_transforms.h` - - - ``CUDA::nppist`` - - ``CUDA::nppist_static`` - -- `nppisu`: Memory support functions in `nppi_support_functions.h` - - - ``CUDA::nppisu`` - - ``CUDA::nppisu_static`` - -- `nppitc`: Threshold and compare operation functions in `nppi_threshold_and_compare_operations.h` - - - ``CUDA::nppitc`` - - ``CUDA::nppitc_static`` - -- `npps`: - - - ``CUDA::npps`` - - ``CUDA::npps_static`` - -.. _`cuda_toolkit_nvBLAS`: - -nvBLAS -"""""" - -The `nvBLAS `_ libraries. -This is a shared library only. - -Targets Created: - -- ``CUDA::nvblas`` - -.. _`cuda_toolkit_nvGRAPH`: - -nvGRAPH -""""""" - -The `nvGRAPH `_ library. -Removed starting in CUDA 11.0 - -Targets Created: - -- ``CUDA::nvgraph`` -- ``CUDA::nvgraph_static`` - - -.. _`cuda_toolkit_nvJPEG`: - -nvJPEG -"""""" - -The `nvJPEG `_ library. -Introduced in CUDA 10. - -Targets Created: - -- ``CUDA::nvjpeg`` -- ``CUDA::nvjpeg_static`` - -.. _`cuda_toolkit_nvPTX`: - -nvPTX Compiler -"""""""""""""" - -.. versionadded:: 3.25 - -The `nvPTX `_ (PTX Compilation) library. -The PTX Compiler APIs are a set of APIs which can be used to compile a PTX program into GPU assembly code. -Introduced in CUDA 11.1 -This is a static library only. - -Targets Created: - -- ``CUDA::nvptxcompiler_static`` starting in CUDA 11.1 - -.. _`cuda_toolkit_nvRTC`: - -nvRTC -""""" - -The `nvRTC `_ (Runtime Compilation) library. - -Targets Created: - -- ``CUDA::nvrtc`` - -.. versionadded:: 3.26 - - - ``CUDA::nvrtc_builtins`` - - ``CUDA::nvrtc_static`` starting in CUDA 11.5 - - ``CUDA::nvrtc_builtins_static`` starting in CUDA 11.5 - -.. _`cuda_toolkit_nvjitlink`: - -nvJitLink -""""""""" - -The `nvJItLink `_ (Runtime LTO Linking) library. - -Targets Created: - -- ``CUDA::nvJitLink`` starting in CUDA 12.0 -- ``CUDA::nvJitLink_static`` starting in CUDA 12.0 - -.. _`cuda_toolkit_nvfatbin`: - -nvFatBin -""""""""" - -.. versionadded:: 3.30 - -The `nvFatBin `_ (Runtime fatbin creation) library. - -Targets Created: - -- ``CUDA::nvfatbin`` starting in CUDA 12.4 -- ``CUDA::nvfatbin_static`` starting in CUDA 12.4 - -.. _`cuda_toolkit_nvml`: - -nvidia-ML -""""""""" - -The `NVIDIA Management Library `_. - -Targets Created: - -- ``CUDA::nvml`` -- ``CUDA::nvml_static`` starting in CUDA 12.4 - -.. versionadded:: 3.31 - Added ``CUDA::nvml_static``. - -.. _`cuda_toolkit_nvToolsExt`: - -nvToolsExt -"""""""""" - -.. deprecated:: 3.25 With CUDA 10.0+, use :ref:`nvtx3 `. - -The `NVIDIA Tools Extension `_. -This is a shared library only. - -Targets Created: - -- ``CUDA::nvToolsExt`` - -.. _`cuda_toolkit_nvtx3`: - -nvtx3 -""""" - -.. versionadded:: 3.25 - -The header-only `NVIDIA Tools Extension Library `_. -Introduced in CUDA 10.0. - -Targets created: - -- ``CUDA::nvtx3`` - -.. _`cuda_toolkit_opencl`: - -OpenCL -"""""" - -The `NVIDIA OpenCL Library `_. -This is a shared library only. - -Targets Created: - -- ``CUDA::OpenCL`` - -.. _`cuda_toolkit_cuLIBOS`: - -cuLIBOS -""""""" - -The cuLIBOS library is a backend thread abstraction layer library which is -static only. The ``CUDA::cublas_static``, ``CUDA::cusparse_static``, -``CUDA::cufft_static``, ``CUDA::curand_static``, and (when implemented) NPP -libraries all automatically have this dependency linked. - -Target Created: - -- ``CUDA::culibos`` - -**Note**: direct usage of this target by consumers should not be necessary. - -.. _`cuda_toolkit_cuRAND`: - - - -Result variables -^^^^^^^^^^^^^^^^ - -``CUDAToolkit_FOUND`` - A boolean specifying whether or not the CUDA Toolkit was found. - -``CUDAToolkit_VERSION`` - The exact version of the CUDA Toolkit found (as reported by - ``nvcc --version``, ``version.txt``, or ``version.json``). - -``CUDAToolkit_VERSION_MAJOR`` - The major version of the CUDA Toolkit. - -``CUDAToolkit_VERSION_MINOR`` - The minor version of the CUDA Toolkit. - -``CUDAToolkit_VERSION_PATCH`` - The patch version of the CUDA Toolkit. - -``CUDAToolkit_BIN_DIR`` - The path to the CUDA Toolkit library directory that contains the CUDA - executable ``nvcc``. - -``CUDAToolkit_INCLUDE_DIRS`` - List of paths to all the CUDA Toolkit folders containing header files - required to compile a project linking against CUDA. - -``CUDAToolkit_LIBRARY_DIR`` - The path to the CUDA Toolkit library directory that contains the CUDA - Runtime library ``cudart``. - -``CUDAToolkit_LIBRARY_ROOT`` - .. versionadded:: 3.18 - - The path to the CUDA Toolkit directory containing the nvvm directory and - either version.txt or version.json. - -``CUDAToolkit_TARGET_DIR`` - The path to the CUDA Toolkit directory including the target architecture - when cross-compiling. When not cross-compiling this will be equivalent to - the parent directory of ``CUDAToolkit_BIN_DIR``. - -``CUDAToolkit_NVCC_EXECUTABLE`` - The path to the NVIDIA CUDA compiler ``nvcc``. Note that this path may - **not** be the same as - :variable:`CMAKE_CUDA_COMPILER _COMPILER>`. ``nvcc`` must be - found to determine the CUDA Toolkit version as well as determining other - features of the Toolkit. This variable is set for the convenience of - modules that depend on this one. - - -#]=======================================================================] - -# NOTE: much of this was simply extracted from FindCUDA.cmake. - -# James Bigler, NVIDIA Corp (nvidia.com - jbigler) -# Abe Stephens, SCI Institute -- http://www.sci.utah.edu/~abe/FindCuda.html -# -# Copyright (c) 2008 - 2009 NVIDIA Corporation. All rights reserved. -# -# Copyright (c) 2007-2009 -# Scientific Computing and Imaging Institute, University of Utah -# -# This code is licensed under the MIT License. See the FindCUDA.cmake script -# for the text of the license. - -# The MIT License -# -# License for the specific language governing rights and limitations under -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS -# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -# DEALINGS IN THE SOFTWARE. -# -############################################################################### - -function(_CUDAToolkit_build_include_dirs result_variable default_paths_variable) - set(content "${${default_paths_variable}}") - set(${result_variable} "${content}" PARENT_SCOPE) -endfunction() - -function(_CUDAToolkit_build_library_dirs result_variable default_paths_variable) - set(content "${${default_paths_variable}}") - set(${result_variable} "${content}" PARENT_SCOPE) -endfunction() - -# The toolkit is located during compiler detection for CUDA and stored in CMakeCUDACompiler.cmake as -# - CMAKE_CUDA_COMPILER_TOOLKIT_ROOT -# - CMAKE_CUDA_COMPILER_LIBRARY_ROOT -# - CMAKE_CUDA_COMPILER_LIBRARY_DIRECTORIES_FROM_IMPLICIT_LIBRARIES -# - CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES -# We compute the rest based on those here to avoid re-searching and to avoid finding a possibly -# different installation. -if(CMAKE_CUDA_COMPILER_TOOLKIT_ROOT) - set(CUDAToolkit_ROOT_DIR "${CMAKE_CUDA_COMPILER_TOOLKIT_ROOT}") - set(CUDAToolkit_LIBRARY_ROOT "${CMAKE_CUDA_COMPILER_LIBRARY_ROOT}") - _CUDAToolkit_build_library_dirs(CUDAToolkit_IMPLICIT_LIBRARY_DIRECTORIES CMAKE_CUDA_HOST_IMPLICIT_LINK_DIRECTORIES) - _CUDAToolkit_build_include_dirs(CUDAToolkit_INCLUDE_DIRECTORIES CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES) - set(CUDAToolkit_BIN_DIR "${CUDAToolkit_ROOT_DIR}/bin") - set(CUDAToolkit_NVCC_EXECUTABLE "${CUDAToolkit_BIN_DIR}/nvcc${CMAKE_EXECUTABLE_SUFFIX}") - set(CUDAToolkit_VERSION "${CMAKE_CUDA_COMPILER_TOOLKIT_VERSION}") - - if(CUDAToolkit_VERSION MATCHES [=[([0-9]+)\.([0-9]+)\.([0-9]+)]=]) - set(CUDAToolkit_VERSION_MAJOR "${CMAKE_MATCH_1}") - set(CUDAToolkit_VERSION_MINOR "${CMAKE_MATCH_2}") - set(CUDAToolkit_VERSION_PATCH "${CMAKE_MATCH_3}") - endif() -else() - function(_CUDAToolkit_find_root_dir ) - cmake_parse_arguments(arg "COMPILER_PATHS" "" "SEARCH_PATHS;FIND_FLAGS" ${ARGN}) - - if(NOT CUDAToolkit_BIN_DIR) - if(arg_COMPILER_PATHS) - # need to find parent dir, since this could clang and not nvcc - if(EXISTS "${CMAKE_CUDA_COMPILER}") - get_filename_component(possible_nvcc_path "${CMAKE_CUDA_COMPILER}" PROGRAM PROGRAM_ARGS CUDAToolkit_compiler_args) - get_filename_component(possible_nvcc_path "${possible_nvcc_path}" DIRECTORY) - elseif(EXISTS "$ENV{CUDACXX}") - get_filename_component(possible_nvcc_path "$ENV{CUDACXX}" PROGRAM PROGRAM_ARGS CUDAToolkit_compiler_args) - get_filename_component(possible_nvcc_path "${possible_nvcc_path}" DIRECTORY) - endif() - if(possible_nvcc_path) - find_program(CUDAToolkit_NVCC_EXECUTABLE - NAMES nvcc nvcc.exe - NO_DEFAULT_PATH - PATHS ${possible_nvcc_path} - ) - endif() - endif() - - if(NOT CUDAToolkit_SENTINEL_FILE) - find_program(CUDAToolkit_NVCC_EXECUTABLE - NAMES nvcc nvcc.exe - PATHS ${arg_SEARCH_PATHS} - ${arg_FIND_FLAGS} - ) - endif() - - if(NOT CUDAToolkit_NVCC_EXECUTABLE) - find_file(CUDAToolkit_SENTINEL_FILE - NAMES version.txt version.json - PATHS ${arg_SEARCH_PATHS} - NO_DEFAULT_PATH - ) - endif() - - if(EXISTS "${CUDAToolkit_NVCC_EXECUTABLE}") - # If NVCC exists then invoke it to find the toolkit location. - # This allows us to support wrapper scripts (e.g. ccache or colornvcc), CUDA Toolkit, - # NVIDIA HPC SDK, and distro's splayed layouts - execute_process(COMMAND ${CUDAToolkit_NVCC_EXECUTABLE} "-v" "__cmake_determine_cuda" - OUTPUT_VARIABLE _CUDA_NVCC_OUT ERROR_VARIABLE _CUDA_NVCC_OUT) - message(CONFIGURE_LOG - "Executed nvcc to extract CUDAToolkit information:\n${_CUDA_NVCC_OUT}\n\n") - if(_CUDA_NVCC_OUT MATCHES "\\#\\$ TOP=([^\r\n]*)") - get_filename_component(CUDAToolkit_BIN_DIR "${CMAKE_MATCH_1}/bin" ABSOLUTE) - message(CONFIGURE_LOG - "Parsed CUDAToolkit nvcc location:\n${CUDAToolkit_BIN_DIR}\n\n") - else() - get_filename_component(CUDAToolkit_BIN_DIR "${CUDAToolkit_NVCC_EXECUTABLE}" DIRECTORY) - endif() - if(_CUDA_NVCC_OUT MATCHES "\\#\\$ INCLUDES=([^\r\n]*)") - separate_arguments(_nvcc_output NATIVE_COMMAND "${CMAKE_MATCH_1}") - foreach(line IN LISTS _nvcc_output) - string(REGEX REPLACE "^-I" "" line "${line}") - get_filename_component(line "${line}" ABSOLUTE) - list(APPEND _cmake_CUDAToolkit_include_directories "${line}") - endforeach() - message(CONFIGURE_LOG - "Parsed CUDAToolkit nvcc implicit include information:\n${_cmake_CUDAToolkit_include_directories}\n\n") - - set(_cmake_CUDAToolkit_include_directories "${_cmake_CUDAToolkit_include_directories}" CACHE INTERNAL "CUDAToolkit internal list of include directories") - endif() - if(_CUDA_NVCC_OUT MATCHES "\\#\\$ LIBRARIES=([^\r\n]*)") - include(${CMAKE_ROOT}/Modules/CMakeParseImplicitLinkInfo.cmake) - set(_nvcc_link_line "cuda-fake-ld ${CMAKE_MATCH_1}") - CMAKE_PARSE_IMPLICIT_LINK_INFO("${_nvcc_link_line}" - _cmake_CUDAToolkit_implicit_link_libs - _cmake_CUDAToolkit_implicit_link_directories - _cmake_CUDAToolkit_implicit_frameworks - _nvcc_log - "${CMAKE_CUDA_IMPLICIT_OBJECT_REGEX}" - LANGUAGE CUDA) - message(CONFIGURE_LOG - "Parsed CUDAToolkit nvcc implicit link information:\n${_nvcc_log}\n${_cmake_CUDAToolkit_implicit_link_directories}\n\n") - unset(_nvcc_link_line) - unset(_cmake_CUDAToolkit_implicit_link_libs) - unset(_cmake_CUDAToolkit_implicit_frameworks) - - set(_cmake_CUDAToolkit_implicit_link_directories "${_cmake_CUDAToolkit_implicit_link_directories}" CACHE INTERNAL "CUDAToolkit internal list of implicit link directories") - endif() - unset(_CUDA_NVCC_OUT) - - set(CUDAToolkit_BIN_DIR "${CUDAToolkit_BIN_DIR}" CACHE PATH "" FORCE) - mark_as_advanced(CUDAToolkit_BIN_DIR) - endif() - - if(CUDAToolkit_SENTINEL_FILE) - get_filename_component(CUDAToolkit_BIN_DIR ${CUDAToolkit_SENTINEL_FILE} DIRECTORY ABSOLUTE) - set(CUDAToolkit_BIN_DIR "${CUDAToolkit_BIN_DIR}/bin") - - set(CUDAToolkit_BIN_DIR "${CUDAToolkit_BIN_DIR}" CACHE PATH "" FORCE) - mark_as_advanced(CUDAToolkit_BIN_DIR) - endif() - endif() - - if(DEFINED _cmake_CUDAToolkit_include_directories) - _CUDAToolkit_build_include_dirs(_cmake_CUDAToolkit_contents _cmake_CUDAToolkit_include_directories) - set(CUDAToolkit_INCLUDE_DIRECTORIES "${_cmake_CUDAToolkit_contents}" PARENT_SCOPE) - endif() - if(DEFINED _cmake_CUDAToolkit_implicit_link_directories) - _CUDAToolkit_build_library_dirs(_cmake_CUDAToolkit_contents _cmake_CUDAToolkit_implicit_link_directories) - set(CUDAToolkit_IMPLICIT_LIBRARY_DIRECTORIES "${_cmake_CUDAToolkit_contents}" PARENT_SCOPE) - endif() - - if(CUDAToolkit_BIN_DIR) - get_filename_component(CUDAToolkit_ROOT_DIR ${CUDAToolkit_BIN_DIR} DIRECTORY ABSOLUTE) - set(CUDAToolkit_ROOT_DIR "${CUDAToolkit_ROOT_DIR}" PARENT_SCOPE) - endif() - - endfunction() - - function(_CUDAToolkit_find_version_file result_variable) - # We first check for a non-scattered installation to prefer it over a scattered installation. - set(version_files version.txt version.json) - foreach(vf IN LISTS version_files) - if(CUDAToolkit_ROOT AND EXISTS "${CUDAToolkit_ROOT}/${vf}") - set(${result_variable} "${CUDAToolkit_ROOT}/${vf}" PARENT_SCOPE) - break() - elseif(CUDAToolkit_ROOT_DIR AND EXISTS "${CUDAToolkit_ROOT_DIR}/${vf}") - set(${result_variable} "${CUDAToolkit_ROOT_DIR}/${vf}" PARENT_SCOPE) - break() - elseif(CMAKE_SYSROOT_LINK AND EXISTS "${CMAKE_SYSROOT_LINK}/usr/lib/cuda/${vf}") - set(${result_variable} "${CMAKE_SYSROOT_LINK}/usr/lib/cuda/${vf}" PARENT_SCOPE) - break() - elseif(EXISTS "${CMAKE_SYSROOT}/usr/lib/cuda/${vf}") - set(${result_variable} "${CMAKE_SYSROOT}/usr/lib/cuda/${vf}" PARENT_SCOPE) - break() - endif() - endforeach() - endfunction() - - function(_CUDAToolkit_parse_version_file version_file) - if(version_file) - file(READ "${version_file}" file_conents) - cmake_path(GET version_file EXTENSION LAST_ONLY version_ext) - if(version_ext STREQUAL ".json") - string(JSON cuda_version_info GET "${file_conents}" "cuda" "version") - set(cuda_version_match_regex [=[([0-9]+)\.([0-9]+)\.([0-9]+)]=]) - elseif(version_ext STREQUAL ".txt") - set(cuda_version_info "${file_conents}") - set(cuda_version_match_regex [=[CUDA Version ([0-9]+)\.([0-9]+)\.([0-9]+)]=]) - endif() - - if(cuda_version_info MATCHES "${cuda_version_match_regex}") - set(CUDAToolkit_VERSION_MAJOR "${CMAKE_MATCH_1}" PARENT_SCOPE) - set(CUDAToolkit_VERSION_MINOR "${CMAKE_MATCH_2}" PARENT_SCOPE) - set(CUDAToolkit_VERSION_PATCH "${CMAKE_MATCH_3}" PARENT_SCOPE) - set(CUDAToolkit_VERSION "${CMAKE_MATCH_1}.${CMAKE_MATCH_2}.${CMAKE_MATCH_3}" PARENT_SCOPE) - endif() - endif() - endfunction() - - # For NVCC we can easily deduce the SDK binary directory from the compiler path. - if(CMAKE_CUDA_COMPILER_LOADED AND NOT CUDAToolkit_BIN_DIR AND CMAKE_CUDA_COMPILER_ID STREQUAL "NVIDIA") - get_filename_component(CUDAToolkit_BIN_DIR "${CMAKE_CUDA_COMPILER}" DIRECTORY) - set(CUDAToolkit_BIN_DIR "${CUDAToolkit_BIN_DIR}" CACHE PATH "") - # Try language provided path first. - _CUDAToolkit_find_root_dir(SEARCH_PATHS "${CUDAToolkit_BIN_DIR}" FIND_FLAGS NO_DEFAULT_PATH) - mark_as_advanced(CUDAToolkit_BIN_DIR) - endif() - - # Try user provided path - _CUDAToolkit_find_root_dir(COMPILER_PATHS) - if(NOT CUDAToolkit_ROOT_DIR AND CUDAToolkit_ROOT) - _CUDAToolkit_find_root_dir(SEARCH_PATHS "${CUDAToolkit_ROOT}" FIND_FLAGS PATH_SUFFIXES bin NO_DEFAULT_PATH) - endif() - if(NOT CUDAToolkit_ROOT_DIR) - _CUDAToolkit_find_root_dir(FIND_FLAGS PATHS ENV CUDA_PATH PATH_SUFFIXES bin) - endif() - - # If the user specified CUDAToolkit_ROOT but the toolkit could not be found, this is an error. - if(NOT CUDAToolkit_ROOT_DIR AND (DEFINED CUDAToolkit_ROOT OR DEFINED ENV{CUDAToolkit_ROOT})) - # Declare error messages now, print later depending on find_package args. - set(fail_base "Could not find nvcc executable in path specified by") - set(cuda_root_fail "${fail_base} CUDAToolkit_ROOT=${CUDAToolkit_ROOT}") - set(env_cuda_root_fail "${fail_base} environment variable CUDAToolkit_ROOT=$ENV{CUDAToolkit_ROOT}") - - if(CUDAToolkit_FIND_REQUIRED) - if(DEFINED CUDAToolkit_ROOT) - message(FATAL_ERROR ${cuda_root_fail}) - elseif(DEFINED ENV{CUDAToolkit_ROOT}) - message(FATAL_ERROR ${env_cuda_root_fail}) - endif() - else() - if(NOT CUDAToolkit_FIND_QUIETLY) - if(DEFINED CUDAToolkit_ROOT) - message(STATUS ${cuda_root_fail}) - elseif(DEFINED ENV{CUDAToolkit_ROOT}) - message(STATUS ${env_cuda_root_fail}) - endif() - endif() - set(CUDAToolkit_FOUND FALSE) - unset(fail_base) - unset(cuda_root_fail) - unset(env_cuda_root_fail) - return() - endif() - endif() - - # CUDAToolkit_ROOT cmake / env variable not specified, try platform defaults. - # - # - Linux: /usr/local/cuda-X.Y - # - macOS: /Developer/NVIDIA/CUDA-X.Y - # - Windows: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\vX.Y - # - # We will also search the default symlink location /usr/local/cuda first since - # if CUDAToolkit_ROOT is not specified, it is assumed that the symlinked - # directory is the desired location. - if(NOT CUDAToolkit_ROOT_DIR) - if(UNIX) - if(NOT APPLE) - set(platform_base "/usr/local/cuda-") - else() - set(platform_base "/Developer/NVIDIA/CUDA-") - endif() - else() - set(platform_base "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v") - endif() - - # Build out a descending list of possible cuda installations, e.g. - file(GLOB possible_paths "${platform_base}*") - # Iterate the glob results and create a descending list. - set(versions) - foreach(p ${possible_paths}) - # Extract version number from end of string - string(REGEX MATCH "[0-9][0-9]?\\.[0-9]$" p_version ${p}) - if(IS_DIRECTORY ${p} AND p_version) - list(APPEND versions ${p_version}) - endif() - endforeach() - - # Sort numerically in descending order, so we try the newest versions first. - list(SORT versions COMPARE NATURAL ORDER DESCENDING) - - # With a descending list of versions, populate possible paths to search. - set(search_paths) - foreach(v ${versions}) - list(APPEND search_paths "${platform_base}${v}") - endforeach() - - # Force the global default /usr/local/cuda to the front on Unix. - if(UNIX) - list(INSERT search_paths 0 "/usr/local/cuda") - endif() - - # Now search for the toolkit again using the platform default search paths. - _CUDAToolkit_find_root_dir(SEARCH_PATHS "${search_paths}" FIND_FLAGS PATH_SUFFIXES bin) - - # We are done with these variables now, cleanup for caller. - unset(platform_base) - unset(possible_paths) - unset(versions) - unset(search_paths) - - if(NOT CUDAToolkit_ROOT_DIR) - if(CUDAToolkit_FIND_REQUIRED) - message(FATAL_ERROR "Could not find nvcc, please set CUDAToolkit_ROOT.") - elseif(NOT CUDAToolkit_FIND_QUIETLY) - message(STATUS "Could not find nvcc, please set CUDAToolkit_ROOT.") - endif() - - set(CUDAToolkit_FOUND FALSE) - return() - endif() - endif() - - _CUDAToolkit_find_version_file( _CUDAToolkit_version_file ) - if(_CUDAToolkit_version_file) - # CUDAToolkit_LIBRARY_ROOT contains the device library and version file. - get_filename_component(CUDAToolkit_LIBRARY_ROOT "${_CUDAToolkit_version_file}" DIRECTORY ABSOLUTE) - endif() - unset(_CUDAToolkit_version_file) - - if(CUDAToolkit_NVCC_EXECUTABLE AND - CMAKE_CUDA_COMPILER_VERSION AND - CUDAToolkit_NVCC_EXECUTABLE STREQUAL CMAKE_CUDA_COMPILER) - # Need to set these based off the already computed CMAKE_CUDA_COMPILER_VERSION value - # This if statement will always match, but is used to provide variables for MATCH 1,2,3... - if(CMAKE_CUDA_COMPILER_VERSION MATCHES [=[([0-9]+)\.([0-9]+)\.([0-9]+)]=]) - set(CUDAToolkit_VERSION_MAJOR "${CMAKE_MATCH_1}") - set(CUDAToolkit_VERSION_MINOR "${CMAKE_MATCH_2}") - set(CUDAToolkit_VERSION_PATCH "${CMAKE_MATCH_3}") - set(CUDAToolkit_VERSION "${CMAKE_CUDA_COMPILER_VERSION}") - endif() - elseif(CUDAToolkit_NVCC_EXECUTABLE) - # Compute the version by invoking nvcc - execute_process(COMMAND ${CUDAToolkit_NVCC_EXECUTABLE} "--version" OUTPUT_VARIABLE NVCC_OUT) - if(NVCC_OUT MATCHES [=[ V([0-9]+)\.([0-9]+)\.([0-9]+)]=]) - set(CUDAToolkit_VERSION_MAJOR "${CMAKE_MATCH_1}") - set(CUDAToolkit_VERSION_MINOR "${CMAKE_MATCH_2}") - set(CUDAToolkit_VERSION_PATCH "${CMAKE_MATCH_3}") - set(CUDAToolkit_VERSION "${CMAKE_MATCH_1}.${CMAKE_MATCH_2}.${CMAKE_MATCH_3}") - endif() - unset(NVCC_OUT) - else() - _CUDAToolkit_find_version_file(version_file) - _CUDAToolkit_parse_version_file("${version_file}") - endif() -endif() - -# Find target directory when crosscompiling. -if(CMAKE_CROSSCOMPILING) - if(CMAKE_SYSTEM_PROCESSOR STREQUAL "armv7-a") - # Support for NVPACK - set(CUDAToolkit_TARGET_NAMES "armv7-linux-androideabi") - elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "arm") - set(CUDAToolkit_TARGET_NAMES "armv7-linux-gnueabihf") - elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64") - if(ANDROID_ARCH_NAME STREQUAL "arm64") - set(CUDAToolkit_TARGET_NAMES "aarch64-linux-androideabi") - elseif (CMAKE_SYSTEM_NAME STREQUAL "QNX") - set(CUDAToolkit_TARGET_NAMES "aarch64-qnx") - else() - set(CUDAToolkit_TARGET_NAMES "aarch64-linux" "sbsa-linux") - endif() - elseif(CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64") - set(CUDAToolkit_TARGET_NAMES "x86_64-linux") - endif() - - foreach(CUDAToolkit_TARGET_NAME IN LISTS CUDAToolkit_TARGET_NAMES) - if(EXISTS "${CUDAToolkit_ROOT_DIR}/targets/${CUDAToolkit_TARGET_NAME}") - set(CUDAToolkit_TARGET_DIR "${CUDAToolkit_ROOT_DIR}/targets/${CUDAToolkit_TARGET_NAME}") - # add known CUDA target root path to the set of directories we search for programs, libraries and headers - list(PREPEND CMAKE_FIND_ROOT_PATH "${CUDAToolkit_TARGET_DIR}") - - # Mark that we need to pop the root search path changes after we have - # found all cuda libraries so that searches for our cross-compilation - # libraries work when another cuda sdk is in CMAKE_PREFIX_PATH or - # PATh - set(_CUDAToolkit_Pop_ROOT_PATH True) - break() - endif() - endforeach() -endif() - -# Determine windows search path suffix for libraries -if(CMAKE_HOST_SYSTEM_NAME STREQUAL "Windows") - if(CMAKE_HOST_SYSTEM_PROCESSOR STREQUAL "AMD64") - set(_CUDAToolkit_win_search_dirs lib/x64) - set(_CUDAToolkit_win_stub_search_dirs lib/x64/stubs) - endif() -endif() - -# If not already set we can simply use the toolkit root or it's a scattered installation. -if(NOT CUDAToolkit_TARGET_DIR) - # Not cross compiling - set(CUDAToolkit_TARGET_DIR "${CUDAToolkit_ROOT_DIR}") - # Now that we have the real ROOT_DIR, find components inside it. - list(APPEND CMAKE_PREFIX_PATH ${CUDAToolkit_ROOT_DIR}) - - # Mark that we need to pop the prefix path changes after we have - # found the cudart library. - set(_CUDAToolkit_Pop_Prefix True) -endif() - - -# We don't need to verify the cuda_runtime header when we are using `nvcc` include paths -# as the compiler being enabled means the header was found -if(NOT CUDAToolkit_INCLUDE_DIRECTORIES) - # Otherwise use CUDAToolkit_TARGET_DIR to guess where the `cuda_runtime.h` is located - # On a scattered installation /usr, on a non-scattered something like /usr/local/cuda or /usr/local/cuda-10.2/targets/aarch64-linux. - if(EXISTS "${CUDAToolkit_TARGET_DIR}/include/cuda_runtime.h") - set(CUDAToolkit_INCLUDE_DIRECTORIES "${CUDAToolkit_TARGET_DIR}/include") - else() - message(STATUS "Unable to find cuda_runtime.h in \"${CUDAToolkit_TARGET_DIR}/include\" for CUDAToolkit_INCLUDE_DIRECTORIES.") - endif() -endif() - -# The NVHPC layout moves math library headers and libraries to a sibling directory and it could be nested under -# the version of the CUDA toolchain -# Create a separate variable so this directory can be selectively added to math targets. -find_path(CUDAToolkit_CUBLAS_INCLUDE_DIR cublas_v2.h PATHS - ${CUDAToolkit_INCLUDE_DIRECTORIES} - NO_DEFAULT_PATH) - -if(NOT CUDAToolkit_CUBLAS_INCLUDE_DIR) - file(REAL_PATH "${CUDAToolkit_TARGET_DIR}" CUDAToolkit_MATH_INCLUDE_DIR) - cmake_path(APPEND CUDAToolkit_MATH_INCLUDE_DIR "../../math_libs/") - if(EXISTS "${CUDAToolkit_MATH_INCLUDE_DIR}/${CUDAToolkit_VERSION_MAJOR}.${CUDAToolkit_VERSION_MINOR}/") - cmake_path(APPEND CUDAToolkit_MATH_INCLUDE_DIR "${CUDAToolkit_VERSION_MAJOR}.${CUDAToolkit_VERSION_MINOR}/") - endif() - cmake_path(APPEND CUDAToolkit_MATH_INCLUDE_DIR "include") - cmake_path(NORMAL_PATH CUDAToolkit_MATH_INCLUDE_DIR) - - find_path(CUDAToolkit_CUBLAS_INCLUDE_DIR cublas_v2.h PATHS - ${CUDAToolkit_INCLUDE_DIRECTORIES} - ) - if(CUDAToolkit_CUBLAS_INCLUDE_DIR) - list(APPEND CUDAToolkit_INCLUDE_DIRECTORIES "${CUDAToolkit_CUBLAS_INCLUDE_DIR}") - endif() -endif() -unset(CUDAToolkit_CUBLAS_INCLUDE_DIR CACHE) -unset(CUDAToolkit_CUBLAS_INCLUDE_DIR) - -# Find the CUDA Runtime Library libcudart -find_library(CUDA_CUDART - NAMES cudart - PATHS ${CUDAToolkit_IMPLICIT_LIBRARY_DIRECTORIES} - PATH_SUFFIXES lib64 ${_CUDAToolkit_win_search_dirs} -) -find_library(CUDA_CUDART - NAMES cudart - PATHS ${CUDAToolkit_IMPLICIT_LIBRARY_DIRECTORIES} - PATH_SUFFIXES lib64/stubs ${_CUDAToolkit_win_stub_search_dirs} lib/stubs stubs -) - -if(NOT CUDA_CUDART AND NOT CUDAToolkit_FIND_QUIETLY) - message(STATUS "Unable to find cudart library.") -endif() - -if(_CUDAToolkit_Pop_Prefix) - list(REMOVE_AT CMAKE_PREFIX_PATH -1) - unset(_CUDAToolkit_Pop_Prefix) -endif() - -#----------------------------------------------------------------------------- -# Perform version comparison and validate all required variables are set. -include(${CMAKE_ROOT}/Modules/FindPackageHandleStandardArgs.cmake) -find_package_handle_standard_args(CUDAToolkit - REQUIRED_VARS - CUDAToolkit_INCLUDE_DIRECTORIES - CUDA_CUDART - CUDAToolkit_BIN_DIR - VERSION_VAR - CUDAToolkit_VERSION -) - -unset(CUDAToolkit_ROOT_DIR) -mark_as_advanced(CUDA_CUDART - CUDAToolkit_NVCC_EXECUTABLE - CUDAToolkit_SENTINEL_FILE - ) - -#----------------------------------------------------------------------------- -# Construct result variables -if(CUDAToolkit_FOUND) - set(CUDAToolkit_INCLUDE_DIRS "${CUDAToolkit_INCLUDE_DIRECTORIES}") - get_filename_component(CUDAToolkit_LIBRARY_DIR ${CUDA_CUDART} DIRECTORY ABSOLUTE) - - # Build search paths without any symlinks - file(REAL_PATH "${CUDAToolkit_LIBRARY_DIR}" _cmake_search_dir) - set(CUDAToolkit_LIBRARY_SEARCH_DIRS "${_cmake_search_dir}") - - # Detect we are in a splayed nvhpc toolkit layout and add extra - # search paths without symlinks - if(CUDAToolkit_LIBRARY_DIR MATCHES ".*/cuda/${CUDAToolkit_VERSION_MAJOR}.${CUDAToolkit_VERSION_MINOR}/lib64$") - # Search location for math_libs/ - block(SCOPE_FOR POLICIES) - cmake_policy(SET CMP0152 NEW) - file(REAL_PATH "${CUDAToolkit_LIBRARY_DIR}/../../../../../" _cmake_search_dir) - list(APPEND CUDAToolkit_LIBRARY_SEARCH_DIRS "${_cmake_search_dir}") - - # Search location for extras like cupti - file(REAL_PATH "${CUDAToolkit_LIBRARY_DIR}/../../../" _cmake_search_dir) - list(APPEND CUDAToolkit_LIBRARY_SEARCH_DIRS "${_cmake_search_dir}") - endblock() - endif() - - if(DEFINED CUDAToolkit_IMPLICIT_LIBRARY_DIRECTORIES) - list(APPEND CUDAToolkit_LIBRARY_SEARCH_DIRS "${CUDAToolkit_IMPLICIT_LIBRARY_DIRECTORIES}") - endif() - - # If no `CUDAToolkit_LIBRARY_ROOT` exists set it based on CUDAToolkit_LIBRARY_DIR - if(NOT DEFINED CUDAToolkit_LIBRARY_ROOT) - foreach(CUDAToolkit_search_loc IN LISTS CUDAToolkit_LIBRARY_DIR CUDAToolkit_BIN_DIR) - get_filename_component(CUDAToolkit_possible_lib_root "${CUDAToolkit_search_loc}" DIRECTORY ABSOLUTE) - if(EXISTS "${CUDAToolkit_possible_lib_root}/nvvm/") - set(CUDAToolkit_LIBRARY_ROOT "${CUDAToolkit_possible_lib_root}") - break() - endif() - endforeach() - unset(CUDAToolkit_search_loc) - unset(CUDAToolkit_possible_lib_root) - endif() -else() - # clear cache results when we fail - unset(_cmake_CUDAToolkit_implicit_link_directories CACHE) - unset(_cmake_CUDAToolkit_include_directories CACHE) - unset(CUDA_CUDART CACHE) - unset(CUDAToolkit_BIN_DIR CACHE) - unset(CUDAToolkit_NVCC_EXECUTABLE CACHE) - unset(CUDAToolkit_SENTINEL_FILE CACHE) -endif() -unset(CUDAToolkit_IMPLICIT_LIBRARY_DIRECTORIES) -unset(CUDAToolkit_INCLUDE_DIRECTORIES) - -#----------------------------------------------------------------------------- -# Construct import targets -if(CUDAToolkit_FOUND) - - function(_CUDAToolkit_find_and_add_import_lib lib_name) - cmake_parse_arguments(arg "" "" "ALT;DEPS;EXTRA_PATH_SUFFIXES;EXTRA_INCLUDE_DIRS;ONLY_SEARCH_FOR" ${ARGN}) - - if(arg_ONLY_SEARCH_FOR) - set(search_names ${arg_ONLY_SEARCH_FOR}) - else() - set(search_names ${lib_name} ${arg_ALT}) - endif() - - find_library(CUDA_${lib_name}_LIBRARY - NAMES ${search_names} - HINTS ${CUDAToolkit_LIBRARY_SEARCH_DIRS} - ENV CUDA_PATH - PATH_SUFFIXES nvidia/current lib64 ${_CUDAToolkit_win_search_dirs} lib - # Support NVHPC splayed math library layout - math_libs/${CUDAToolkit_VERSION_MAJOR}.${CUDAToolkit_VERSION_MINOR}/lib64 - math_libs/lib64 - ${arg_EXTRA_PATH_SUFFIXES} - ) - # Don't try any stub directories until we have exhausted all other - # search locations. - set(CUDA_IMPORT_PROPERTY IMPORTED_LOCATION) - set(CUDA_IMPORT_TYPE UNKNOWN) - if(NOT CUDA_${lib_name}_LIBRARY) - find_library(CUDA_${lib_name}_LIBRARY - NAMES ${search_names} - HINTS ${CUDAToolkit_LIBRARY_SEARCH_DIRS} - ENV CUDA_PATH - PATH_SUFFIXES lib64/stubs ${_CUDAToolkit_win_stub_search_dirs} lib/stubs stubs - ) - endif() - if(CUDA_${lib_name}_LIBRARY MATCHES "/stubs/" AND NOT CUDA_${lib_name}_LIBRARY MATCHES "\\.a$" AND NOT WIN32) - # Use a SHARED library with IMPORTED_IMPLIB, but not IMPORTED_LOCATION, - # to indicate that the stub is for linkers but not dynamic loaders. - # It will not contribute any RPATH entry. When encountered as - # a private transitive dependency of another shared library, - # it will be passed explicitly to linkers so they can find it - # even when the runtime library file does not exist on disk. - set(CUDA_IMPORT_PROPERTY IMPORTED_IMPLIB) - set(CUDA_IMPORT_TYPE SHARED) - endif() - - mark_as_advanced(CUDA_${lib_name}_LIBRARY) - - if (NOT TARGET CUDA::${lib_name} AND CUDA_${lib_name}_LIBRARY) - add_library(CUDA::${lib_name} ${CUDA_IMPORT_TYPE} IMPORTED) - target_include_directories(CUDA::${lib_name} SYSTEM INTERFACE "${CUDAToolkit_INCLUDE_DIRS}") - if(DEFINED CUDAToolkit_MATH_INCLUDE_DIR) - string(FIND ${CUDA_${lib_name}_LIBRARY} "math_libs" math_libs) - if(NOT ${math_libs} EQUAL -1) - target_include_directories(CUDA::${lib_name} SYSTEM INTERFACE "${CUDAToolkit_MATH_INCLUDE_DIR}") - endif() - endif() - set_property(TARGET CUDA::${lib_name} PROPERTY ${CUDA_IMPORT_PROPERTY} "${CUDA_${lib_name}_LIBRARY}") - foreach(dep ${arg_DEPS}) - if(TARGET CUDA::${dep}) - target_link_libraries(CUDA::${lib_name} INTERFACE CUDA::${dep}) - endif() - endforeach() - if(arg_EXTRA_INCLUDE_DIRS) - target_include_directories(CUDA::${lib_name} SYSTEM INTERFACE "${arg_EXTRA_INCLUDE_DIRS}") - endif() - endif() - endfunction() - - if(NOT TARGET CUDA::toolkit) - add_library(CUDA::toolkit IMPORTED INTERFACE) - target_include_directories(CUDA::toolkit SYSTEM INTERFACE "${CUDAToolkit_INCLUDE_DIRS}") - target_link_directories(CUDA::toolkit INTERFACE "${CUDAToolkit_LIBRARY_DIR}") - endif() - - # setup dependencies that are required for cudart/cudart_static when building - # on linux. These are generally only required when using the CUDA toolkit - # when CUDA language is disabled - if(NOT TARGET CUDA::cudart_static_deps) - add_library(CUDA::cudart_static_deps IMPORTED INTERFACE) - if(UNIX AND (CMAKE_C_COMPILER OR CMAKE_CXX_COMPILER)) - find_package(Threads REQUIRED) - target_link_libraries(CUDA::cudart_static_deps INTERFACE Threads::Threads ${CMAKE_DL_LIBS}) - endif() - - if(UNIX AND NOT APPLE AND NOT (CMAKE_SYSTEM_NAME STREQUAL "QNX")) - # On Linux, you must link against librt when using the static cuda runtime. - find_library(CUDAToolkit_rt_LIBRARY rt) - mark_as_advanced(CUDAToolkit_rt_LIBRARY) - if(NOT CUDAToolkit_rt_LIBRARY) - message(WARNING "Could not find librt library, needed by CUDA::cudart_static") - else() - target_link_libraries(CUDA::cudart_static_deps INTERFACE ${CUDAToolkit_rt_LIBRARY}) - endif() - endif() - endif() - - _CUDAToolkit_find_and_add_import_lib(cuda_driver ALT cuda DEPS cudart_static_deps) - _CUDAToolkit_find_and_add_import_lib(cudart DEPS cudart_static_deps) - _CUDAToolkit_find_and_add_import_lib(cudart_static DEPS cudart_static_deps) - - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 12.0.0) - _CUDAToolkit_find_and_add_import_lib(nvJitLink) - _CUDAToolkit_find_and_add_import_lib(nvJitLink_static DEPS cudart_static_deps) - endif() - - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 12.4.0) - _CUDAToolkit_find_and_add_import_lib(nvfatbin DEPS cudart_static_deps) - _CUDAToolkit_find_and_add_import_lib(nvfatbin_static DEPS cudart_static_deps) - endif() - - _CUDAToolkit_find_and_add_import_lib(culibos) # it's a static library - foreach (cuda_lib cublasLt cufft nvjpeg) - _CUDAToolkit_find_and_add_import_lib(${cuda_lib}) - _CUDAToolkit_find_and_add_import_lib(${cuda_lib}_static DEPS cudart_static_deps culibos) - endforeach() - foreach (cuda_lib curand nppc) - _CUDAToolkit_find_and_add_import_lib(${cuda_lib}) - _CUDAToolkit_find_and_add_import_lib(${cuda_lib}_static DEPS culibos) - endforeach() - - _CUDAToolkit_find_and_add_import_lib(cusparse DEPS nvJitLink) - _CUDAToolkit_find_and_add_import_lib(cusparse_static DEPS nvJitLink_static culibos) - - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 11.0.0) - # cublas depends on cublasLt - # https://docs.nvidia.com/cuda/archive/11.0/cublas#static-library - _CUDAToolkit_find_and_add_import_lib(cublas DEPS cublasLt culibos) - _CUDAToolkit_find_and_add_import_lib(cublas_static DEPS cublasLt_static culibos) - else() - _CUDAToolkit_find_and_add_import_lib(cublas DEPS culibos) - _CUDAToolkit_find_and_add_import_lib(cublas_static DEPS culibos) - endif() - - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 11.4) - _CUDAToolkit_find_and_add_import_lib(cuFile ALT cufile DEPS culibos) - _CUDAToolkit_find_and_add_import_lib(cuFile_static ALT cufile_static DEPS culibos) - - _CUDAToolkit_find_and_add_import_lib(cuFile_rdma ALT cufile_rdma DEPS cuFile culibos) - _CUDAToolkit_find_and_add_import_lib(cuFile_rdma_static ALT cufile_rdma_static DEPS cuFile_static culibos) - endif() - - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 11.6) - _CUDAToolkit_find_and_add_import_lib(cudla) - endif() - - - # cuFFTW depends on cuFFT - _CUDAToolkit_find_and_add_import_lib(cufftw DEPS cufft) - _CUDAToolkit_find_and_add_import_lib(cufftw_static DEPS cufft_static) - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 9.2) - _CUDAToolkit_find_and_add_import_lib(cufft_static_nocallback DEPS culibos) - endif() - - # cuSOLVER depends on cuBLAS, and cuSPARSE - set(cusolver_deps cublas cusparse) - set(cusolver_static_deps cublas_static cusparse_static culibos) - if(CUDAToolkit_VERSION VERSION_GREATER 11.2.1) - # cusolver depends on libcusolver_metis and cublasLt - # https://docs.nvidia.com/cuda/archive/11.2.2/cusolver#link-dependency - list(APPEND cusolver_deps cublasLt) - _CUDAToolkit_find_and_add_import_lib(cusolver_metis_static ALT metis_static) # implementation detail static lib - list(APPEND cusolver_static_deps cusolver_metis_static cublasLt_static) - endif() - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 10.1.2) - # cusolver depends on liblapack_static.a starting with CUDA 10.1 update 2, - # https://docs.nvidia.com/cuda/archive/11.5.0/cusolver#static-link-lapack - _CUDAToolkit_find_and_add_import_lib(cusolver_lapack_static ALT lapack_static) # implementation detail static lib - list(APPEND cusolver_static_deps cusolver_lapack_static) - endif() - _CUDAToolkit_find_and_add_import_lib(cusolver DEPS ${cusolver_deps}) - _CUDAToolkit_find_and_add_import_lib(cusolver_static DEPS ${cusolver_static_deps}) - unset(cusolver_deps) - unset(cusolver_static_deps) - - # nvGRAPH depends on cuRAND, and cuSOLVER. - _CUDAToolkit_find_and_add_import_lib(nvgraph DEPS curand cusolver) - _CUDAToolkit_find_and_add_import_lib(nvgraph_static DEPS curand_static cusolver_static) - - # Process the majority of the NPP libraries. - foreach (cuda_lib nppial nppicc nppidei nppif nppig nppim nppist nppitc npps nppicom nppisu) - _CUDAToolkit_find_and_add_import_lib(${cuda_lib} DEPS nppc) - _CUDAToolkit_find_and_add_import_lib(${cuda_lib}_static DEPS nppc_static) - endforeach() - - find_path(CUDAToolkit_CUPTI_INCLUDE_DIR cupti.h PATHS - "${CUDAToolkit_ROOT_DIR}/extras/CUPTI/include" - ${CUDAToolkit_INCLUDE_DIRS} - PATH_SUFFIXES "../extras/CUPTI/include" - "../../../extras/CUPTI/include" - NO_DEFAULT_PATH) - mark_as_advanced(CUDAToolkit_CUPTI_INCLUDE_DIR) - - if(CUDAToolkit_CUPTI_INCLUDE_DIR) - set(_cmake_cupti_extra_paths extras/CUPTI/lib64/ - extras/CUPTI/lib/ - ../extras/CUPTI/lib64/ - ../extras/CUPTI/lib/) - _CUDAToolkit_find_and_add_import_lib(cupti - EXTRA_PATH_SUFFIXES ${_cmake_cupti_extra_paths} - EXTRA_INCLUDE_DIRS "${CUDAToolkit_CUPTI_INCLUDE_DIR}") - _CUDAToolkit_find_and_add_import_lib(cupti_static - EXTRA_PATH_SUFFIXES ${_cmake_cupti_extra_paths} - EXTRA_INCLUDE_DIRS "${CUDAToolkit_CUPTI_INCLUDE_DIR}") - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 10.2.0) - _CUDAToolkit_find_and_add_import_lib(nvperf_host - EXTRA_PATH_SUFFIXES ${_cmake_cupti_extra_paths} - EXTRA_INCLUDE_DIRS "${CUDAToolkit_CUPTI_INCLUDE_DIR}") - _CUDAToolkit_find_and_add_import_lib(nvperf_host_static - EXTRA_PATH_SUFFIXES ${_cmake_cupti_extra_paths} - EXTRA_INCLUDE_DIRS "${CUDAToolkit_CUPTI_INCLUDE_DIR}") - _CUDAToolkit_find_and_add_import_lib(nvperf_target - EXTRA_PATH_SUFFIXES ${_cmake_cupti_extra_paths} - EXTRA_INCLUDE_DIRS "${CUDAToolkit_CUPTI_INCLUDE_DIR}") - endif() - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 11.3.0) - _CUDAToolkit_find_and_add_import_lib(pcsamplingutil - EXTRA_PATH_SUFFIXES ${_cmake_cupti_extra_paths} - EXTRA_INCLUDE_DIRS "${CUDAToolkit_CUPTI_INCLUDE_DIR}") - endif() - endif() - - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 11.1.0) - if(NOT TARGET CUDA::nvptxcompiler_static) - _CUDAToolkit_find_and_add_import_lib(nvptxcompiler_static) - if(TARGET CUDA::nvptxcompiler_static) - target_link_libraries(CUDA::nvptxcompiler_static INTERFACE CUDA::cudart_static_deps) - endif() - endif() - endif() - - _CUDAToolkit_find_and_add_import_lib(nvrtc_builtins ALT nvrtc-builtins) - _CUDAToolkit_find_and_add_import_lib(nvrtc) - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 11.5.0) - _CUDAToolkit_find_and_add_import_lib(nvrtc_builtins_static ALT nvrtc-builtins_static) - if(NOT TARGET CUDA::nvrtc_static) - _CUDAToolkit_find_and_add_import_lib(nvrtc_static DEPS nvrtc_builtins_static nvptxcompiler_static) - if(TARGET CUDA::nvrtc_static AND WIN32 AND NOT (BORLAND OR MINGW OR CYGWIN)) - target_link_libraries(CUDA::nvrtc_static INTERFACE Ws2_32.lib) - endif() - endif() - endif() - - _CUDAToolkit_find_and_add_import_lib(nvml ALT nvidia-ml nvml) - _CUDAToolkit_find_and_add_import_lib(nvml_static ONLY_SEARCH_FOR libnvidia-ml.a libnvml.a) - - if(WIN32) - # nvtools can be installed outside the CUDA toolkit directory - # so prefer the NVTOOLSEXT_PATH windows only environment variable - # In addition on windows the most common name is nvToolsExt64_1 - find_library(CUDA_nvToolsExt_LIBRARY - NAMES nvToolsExt64_1 nvToolsExt64 nvToolsExt - PATHS ENV NVTOOLSEXT_PATH - ENV CUDA_PATH - PATH_SUFFIXES lib/x64 lib - ) - endif() - _CUDAToolkit_find_and_add_import_lib(nvToolsExt ALT nvToolsExt64) - - if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL 10.0) - # nvToolsExt is deprecated since nvtx3 introduction. - # Warn only if the project requires a sufficiently new CMake to make migration possible. - if(TARGET CUDA::nvToolsExt AND CMAKE_MINIMUM_REQUIRED_VERSION VERSION_GREATER_EQUAL 3.25) - set_property(TARGET CUDA::nvToolsExt PROPERTY DEPRECATION "nvToolsExt has been superseded by nvtx3 since CUDA 10.0 and CMake 3.25. Use CUDA::nvtx3 and include instead.") - endif() - - # Header-only variant. Uses dlopen(). - if(NOT TARGET CUDA::nvtx3) - add_library(CUDA::nvtx3 INTERFACE IMPORTED) - target_include_directories(CUDA::nvtx3 SYSTEM INTERFACE "${CUDAToolkit_INCLUDE_DIRS}") - target_link_libraries(CUDA::nvtx3 INTERFACE ${CMAKE_DL_LIBS}) - endif() - endif() - - _CUDAToolkit_find_and_add_import_lib(OpenCL) -endif() - -if(_CUDAToolkit_Pop_ROOT_PATH) - list(REMOVE_AT CMAKE_FIND_ROOT_PATH 0) - unset(_CUDAToolkit_Pop_ROOT_PATH) -endif() - -unset(_CUDAToolkit_win_search_dirs) -unset(_CUDAToolkit_win_stub_search_dirs) From 8c7eecfc55d8e3ed91ef327c75cce368ad5375b6 Mon Sep 17 00:00:00 2001 From: Vukasin Milovanovic Date: Mon, 24 Feb 2025 15:58:59 -0800 Subject: [PATCH 081/129] Read the footers in parallel when reading multiple Parquet files (#17957) Depends on https://github.com/rapidsai/cudf/pull/18018 When reading multiple files, all data(i.e. pages) IO is performed in the same "batch", allowing parallel IO operations (provided by kvikIO). However, footers are read serially, leading to poor performance when reading many files. This is especially pronounced for IO that benefits from high level of parallelism. This PR performs footer reading/parsing asynchronously using an internal thread pool. The pool size can be controlled with an environment variable `LIBCUDF_NUM_HOST_WORKERS`. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Nghia Truong (https://github.com/ttnghia) - Paul Mattione (https://github.com/pmattione-nvidia) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/17957 --- cpp/CMakeLists.txt | 1 + .../detail/utilities/host_worker_pool.hpp | 32 +++++++++++++++++++ cpp/src/io/parquet/reader_impl_helpers.cpp | 19 ++++++++--- cpp/src/utilities/host_worker_pool.cpp | 32 +++++++++++++++++++ 4 files changed, 80 insertions(+), 4 deletions(-) create mode 100644 cpp/include/cudf/detail/utilities/host_worker_pool.hpp create mode 100644 cpp/src/utilities/host_worker_pool.cpp diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index bb4d20f837c..0282282b5f3 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -773,6 +773,7 @@ add_library( src/utilities/cuda_memcpy.cu src/utilities/default_stream.cpp src/utilities/host_memory.cpp + src/utilities/host_worker_pool.cpp src/utilities/linked_column.cpp src/utilities/logger.cpp src/utilities/prefetch.cpp diff --git a/cpp/include/cudf/detail/utilities/host_worker_pool.hpp b/cpp/include/cudf/detail/utilities/host_worker_pool.hpp new file mode 100644 index 00000000000..7bd0cab76bc --- /dev/null +++ b/cpp/include/cudf/detail/utilities/host_worker_pool.hpp @@ -0,0 +1,32 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +namespace cudf::detail { + +/** + * @brief Retrieves a reference to the global host worker thread pool. + * + * This function returns a reference to a thread pool that can be used for executing host-only + * tasks. The pool size is potentially not optimal for tasks that include device operations, like + * copies between host and device and kernel calls. + * + * @return A reference to the host worker thread pool. + */ +BS::thread_pool& host_worker_pool(); + +} // namespace cudf::detail diff --git a/cpp/src/io/parquet/reader_impl_helpers.cpp b/cpp/src/io/parquet/reader_impl_helpers.cpp index 768ca384352..ffc164964a5 100644 --- a/cpp/src/io/parquet/reader_impl_helpers.cpp +++ b/cpp/src/io/parquet/reader_impl_helpers.cpp @@ -23,6 +23,7 @@ #include "ipc/Message_generated.h" #include "ipc/Schema_generated.h" +#include #include #include @@ -352,11 +353,21 @@ metadata::metadata(datasource* source) std::vector aggregate_reader_metadata::metadatas_from_sources( host_span const> sources) { + // Avoid using the thread pool for a single source + if (sources.size() == 1) { return {metadata{sources[0].get()}}; } + + std::vector> metadata_ctor_tasks; + metadata_ctor_tasks.reserve(sources.size()); + for (auto const& source : sources) { + metadata_ctor_tasks.emplace_back(cudf::detail::host_worker_pool().submit_task( + [source = source.get()] { return metadata{source}; })); + } std::vector metadatas; - std::transform( - sources.begin(), sources.end(), std::back_inserter(metadatas), [](auto const& source) { - return metadata(source.get()); - }); + metadatas.reserve(sources.size()); + std::transform(metadata_ctor_tasks.begin(), + metadata_ctor_tasks.end(), + std::back_inserter(metadatas), + [](std::future& task) { return std::move(task).get(); }); return metadatas; } diff --git a/cpp/src/utilities/host_worker_pool.cpp b/cpp/src/utilities/host_worker_pool.cpp new file mode 100644 index 00000000000..fa0b8b6620d --- /dev/null +++ b/cpp/src/utilities/host_worker_pool.cpp @@ -0,0 +1,32 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "io/utilities/getenv_or.hpp" + +#include + +namespace cudf::detail { + +BS::thread_pool& host_worker_pool() +{ + static const std::size_t default_pool_size = + std::min(32u, std::thread::hardware_concurrency() / 2); + static const std::size_t pool_size = getenv_or("LIBCUDF_NUM_HOST_WORKERS", default_pool_size); + static BS::thread_pool pool(pool_size); + return pool; +} + +} // namespace cudf::detail From 5f71f7665977d3c976e37d30cc64adea49235776 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Mon, 24 Feb 2025 17:50:35 -0800 Subject: [PATCH 082/129] Support IntervalDtype(subtype=None) (#18017) closes https://github.com/rapidsai/cudf/issues/17997 Will help unblock https://github.com/rapidsai/cudf/pull/17978 where we will need to interpret `dtype="interval"` as "interval without a subtype" instead of "interval with float64 subtype" Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18017 --- docs/cudf/source/conf.py | 2 + python/cudf/cudf/core/column/interval.py | 8 +- python/cudf/cudf/core/dtypes.py | 131 ++++++++++++++--------- python/cudf/cudf/core/index.py | 2 +- python/cudf/cudf/tests/test_interval.py | 11 +- 5 files changed, 95 insertions(+), 59 deletions(-) diff --git a/docs/cudf/source/conf.py b/docs/cudf/source/conf.py index ac34c10d22f..c74da8d0ca9 100644 --- a/docs/cudf/source/conf.py +++ b/docs/cudf/source/conf.py @@ -593,6 +593,8 @@ def on_missing_reference(app, env, node, contnode): ("py:class", "pyarrow.lib.ChunkedArray"), ("py:class", "pyarrow.lib.Array"), ("py:class", "ColumnLike"), + ("py:class", "DtypeObj"), + ("py:class", "pa.StructType"), # TODO: Remove this when we figure out why typing_extensions doesn't seem # to map types correctly for intersphinx ("py:class", "typing_extensions.Self"), diff --git a/python/cudf/cudf/core/column/interval.py b/python/cudf/cudf/core/column/interval.py index dd8f58a118e..2be85fcaa83 100644 --- a/python/cudf/cudf/core/column/interval.py +++ b/python/cudf/cudf/core/column/interval.py @@ -1,4 +1,4 @@ -# Copyright (c) 2018-2024, NVIDIA CORPORATION. +# Copyright (c) 2018-2025, NVIDIA CORPORATION. from __future__ import annotations from typing import TYPE_CHECKING, Literal @@ -105,9 +105,7 @@ def copy(self, deep: bool = True) -> Self: return IntervalColumn( # type: ignore[return-value] data=None, size=struct_copy.size, - dtype=IntervalDtype( - struct_copy.dtype.fields["left"], self.dtype.closed - ), + dtype=IntervalDtype(self.dtype.subtype, self.dtype.closed), mask=struct_copy.base_mask, offset=struct_copy.offset, null_count=struct_copy.null_count, @@ -163,7 +161,7 @@ def set_closed( return IntervalColumn( # type: ignore[return-value] data=None, size=self.size, - dtype=IntervalDtype(self.dtype.fields["left"], closed), + dtype=IntervalDtype(self.dtype.subtype, closed), mask=self.base_mask, offset=self.offset, null_count=self.null_count, diff --git a/python/cudf/cudf/core/dtypes.py b/python/cudf/cudf/core/dtypes.py index 12a9cce9f1c..977208f5eb4 100644 --- a/python/cudf/cudf/core/dtypes.py +++ b/python/cudf/cudf/core/dtypes.py @@ -6,7 +6,7 @@ import textwrap import warnings from functools import cached_property -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, Literal import numpy as np import pandas as pd @@ -19,7 +19,11 @@ from cudf.core._compat import PANDAS_GE_210, PANDAS_LT_300 from cudf.core.abc import Serializable from cudf.utils.docutils import doc_apply -from cudf.utils.dtypes import CUDF_STRING_DTYPE, cudf_dtype_from_pa_type +from cudf.utils.dtypes import ( + CUDF_STRING_DTYPE, + cudf_dtype_from_pa_type, + cudf_dtype_to_pa_type, +) if PANDAS_GE_210: PANDAS_NUMPY_DTYPE = pd.core.dtypes.dtypes.NumpyEADtype @@ -29,7 +33,9 @@ if TYPE_CHECKING: from collections.abc import Callable - from cudf._typing import Dtype + from typing_extension import Self + + from cudf._typing import Dtype, DtypeObj from cudf.core.buffer import Buffer @@ -573,15 +579,11 @@ class StructDtype(_BaseDtype): name = "struct" - def __init__(self, fields): - pa_fields = { - k: cudf.utils.dtypes.cudf_dtype_to_pa_type(cudf.dtype(v)) - for k, v in fields.items() - } - self._typ = pa.struct(pa_fields) + def __init__(self, fields: dict[str, Dtype]) -> None: + self._fields = {k: cudf.dtype(v) for k, v in fields.items()} @property - def fields(self): + def fields(self) -> dict[str, DtypeObj]: """ Returns an ordered dict of column name and dtype key-value. @@ -594,10 +596,7 @@ def fields(self): >>> struct_dtype.fields {'a': dtype('int64'), 'b': dtype('O')} """ - return { - field.name: cudf.utils.dtypes.cudf_dtype_from_pa_type(field.type) - for field in self._typ - } + return self._fields @property def type(self): @@ -606,7 +605,7 @@ def type(self): return dict @classmethod - def from_arrow(cls, typ): + def from_arrow(cls, typ: pa.StructType) -> Self: """ Convert a ``pyarrow.StructType`` to ``StructDtype``. @@ -620,11 +619,19 @@ def from_arrow(cls, typ): >>> cudf.StructDtype.from_arrow(pa_struct_type) StructDtype({'x': dtype('int32'), 'y': dtype('O')}) """ - obj = object.__new__(cls) - obj._typ = typ - return obj + return cls( + { + typ.field(i).name: cudf_dtype_from_pa_type(typ.field(i).type) + for i in range(typ.num_fields) + } + # Once pyarrow 18 is the min version, replace with this version + # { + # field.name: cudf_dtype_from_pa_type(field.type) + # for field in typ.fields + # } + ) - def to_arrow(self): + def to_arrow(self) -> pa.StructType: """ Convert a ``StructDtype`` to a ``pyarrow.StructType``. @@ -637,20 +644,25 @@ def to_arrow(self): >>> struct_type.to_arrow() StructType(struct) """ - return self._typ + return pa.struct( + { + k: cudf_dtype_to_pa_type(dtype) + for k, dtype in self.fields.items() + } + ) - def __eq__(self, other): + def __eq__(self, other) -> bool: if isinstance(other, str): return other == self.name if not isinstance(other, StructDtype): return False - return self._typ.equals(other._typ) + return self.to_arrow().equals(other.to_arrow()) - def __repr__(self): + def __repr__(self) -> str: return f"{type(self).__name__}({self.fields})" - def __hash__(self): - return hash(self._typ) + def __hash__(self) -> int: + return hash(self.to_arrow()) def serialize(self) -> tuple[dict, list]: header: dict[str, Any] = {} @@ -674,7 +686,7 @@ def serialize(self) -> tuple[dict, list]: return header, frames @classmethod - def deserialize(cls, header: dict, frames: list): + def deserialize(cls, header: dict, frames: list) -> Self: _check_type(cls, header, frames) fields = {} for k, dtype in header["fields"].items(): @@ -689,11 +701,8 @@ def deserialize(cls, header: dict, frames: list): return cls(fields) @cached_property - def itemsize(self): - return sum( - cudf.utils.dtypes.cudf_dtype_from_pa_type(field.type).itemsize - for field in self._typ - ) + def itemsize(self) -> int: + return sum(field.itemsize for field in self.fields.values()) def _recursively_replace_fields(self, result: dict) -> dict: """ @@ -926,6 +935,10 @@ class Decimal128Dtype(DecimalDtype): class IntervalDtype(StructDtype): """ + A data type for Interval data. + + Parameters + ---------- subtype: str, np.dtype The dtype of the Interval bounds. closed: {'right', 'left', 'both', 'neither'}, default 'right' @@ -935,43 +948,55 @@ class IntervalDtype(StructDtype): name = "interval" - def __init__(self, subtype, closed="right"): - super().__init__(fields={"left": subtype, "right": subtype}) - - if closed is None: - closed = "right" - if closed in ["left", "right", "neither", "both"]: + def __init__( + self, + subtype: None | Dtype = None, + closed: Literal["left", "right", "neither", "both"] = "right", + ) -> None: + if closed in {"left", "right", "neither", "both"}: self.closed = closed else: - raise ValueError("closed value is not valid") + raise ValueError(f"{closed=} is not valid") + if subtype is None: + self._subtype = None + dtypes = {} + else: + self._subtype = cudf.dtype(subtype) + dtypes = {"left": self._subtype, "right": self._subtype} + super().__init__(dtypes) @property - def subtype(self): - return self.fields["left"] + def subtype(self) -> DtypeObj | None: + return self._subtype def __repr__(self) -> str: + if self.subtype is None: + return "interval" return f"interval[{self.subtype}, {self.closed}]" def __str__(self) -> str: - return self.__repr__() + return repr(self) @classmethod - def from_arrow(cls, typ): - return IntervalDtype(typ.subtype.to_pandas_dtype(), typ.closed) + def from_arrow(cls, typ: ArrowIntervalType) -> Self: + return cls(typ.subtype.to_pandas_dtype(), typ.closed) - def to_arrow(self): + def to_arrow(self) -> ArrowIntervalType: return ArrowIntervalType( - pa.from_numpy_dtype(self.subtype), self.closed + cudf_dtype_to_pa_type(self.subtype), self.closed ) @classmethod - def from_pandas(cls, pd_dtype: pd.IntervalDtype) -> "IntervalDtype": - return cls(subtype=pd_dtype.subtype, closed=pd_dtype.closed) + def from_pandas(cls, pd_dtype: pd.IntervalDtype) -> Self: + return cls( + subtype=pd_dtype.subtype, + closed="right" if pd_dtype.closed is None else pd_dtype.closed, + ) def to_pandas(self) -> pd.IntervalDtype: return pd.IntervalDtype(subtype=self.subtype, closed=self.closed) - def __eq__(self, other): + def __eq__(self, other) -> bool: if isinstance(other, str): # This means equality isn't transitive but mimics pandas return other in (self.name, str(self)) @@ -981,21 +1006,23 @@ def __eq__(self, other): and self.closed == other.closed ) - def __hash__(self): + def __hash__(self) -> int: return hash((self.subtype, self.closed)) def serialize(self) -> tuple[dict, list]: header = { - "fields": (self.subtype.str, self.closed), + "fields": ( + self.subtype.str if self.subtype is not None else self.subtype, + self.closed, + ), "frame_count": 0, } return header, [] @classmethod - def deserialize(cls, header: dict, frames: list): + def deserialize(cls, header: dict, frames: list) -> Self: _check_type(cls, header, frames) subtype, closed = header["fields"] - subtype = np.dtype(subtype) return cls(subtype, closed=closed) diff --git a/python/cudf/cudf/core/index.py b/python/cudf/cudf/core/index.py index 8587bff2e32..1730a692dc1 100644 --- a/python/cudf/cudf/core/index.py +++ b/python/cudf/cudf/core/index.py @@ -3517,7 +3517,7 @@ def _from_column( def from_breaks( cls, breaks, - closed: Literal["left", "right", "neither", "both"] | None = "right", + closed: Literal["left", "right", "neither", "both"] = "right", name=None, copy: bool = False, dtype=None, diff --git a/python/cudf/cudf/tests/test_interval.py b/python/cudf/cudf/tests/test_interval.py index 5e1dd33fbf1..757eed0c9e3 100644 --- a/python/cudf/cudf/tests/test_interval.py +++ b/python/cudf/cudf/tests/test_interval.py @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. import numpy as np @@ -210,3 +210,12 @@ def test_reduction_return_interval_pandas_compatible(): result = cudf_ii.min() expected = ii.min() assert result == expected + + +def test_empty_intervaldtype(): + # "older pandas" supported closed=None, cudf chooses not to support that + pd_id = pd.IntervalDtype(closed="right") + cudf_id = cudf.IntervalDtype() + + assert str(pd_id) == str(cudf_id) + assert pd_id.subtype == cudf_id.subtype From 59d8f2697440caf79f90a332acbcde54ab4d4b3b Mon Sep 17 00:00:00 2001 From: Peter Andreas Entschev Date: Tue, 25 Feb 2025 16:30:28 +0100 Subject: [PATCH 083/129] Support Distributed in cudf-polars tests and IR evaluation (#17364) Support testing cudf-polars with the Distribute scheduler, and IR evaluation through serialization. Authors: - Peter Andreas Entschev (https://github.com/pentschev) - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Lawrence Mitchell (https://github.com/wence-) - Richard (Rick) Zamora (https://github.com/rjzamora) - Bradley Dice (https://github.com/bdice) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: https://github.com/rapidsai/cudf/pull/17364 --- ci/run_cudf_polars_pytests.sh | 8 ++- .../cudf_polars/experimental/parallel.py | 54 +++++++++++++++++-- python/cudf_polars/tests/conftest.py | 46 +++++++++++++++- 3 files changed, 103 insertions(+), 5 deletions(-) diff --git a/ci/run_cudf_polars_pytests.sh b/ci/run_cudf_polars_pytests.sh index bf5a3ccee8e..e881055e9e3 100755 --- a/ci/run_cudf_polars_pytests.sh +++ b/ci/run_cudf_polars_pytests.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. set -euo pipefail @@ -13,3 +13,9 @@ python -m pytest --cache-clear "$@" tests # Test the "dask-experimental" executor python -m pytest --cache-clear "$@" tests --executor dask-experimental + +# Test the "dask-experimental" executor with Distributed cluster +# Not all tests pass yet, deselecting by name those that are failing. +python -m pytest --cache-clear "$@" tests --executor dask-experimental --dask-cluster \ + -k "not test_groupby_maintain_order_random and not test_scan_csv_multi and not test_select_literal_series" \ + --cov-fail-under=89 # Override coverage, Distributed cluster coverage not yet 100% diff --git a/python/cudf_polars/cudf_polars/experimental/parallel.py b/python/cudf_polars/cudf_polars/experimental/parallel.py index 16290fdb663..e81866e68e4 100644 --- a/python/cudf_polars/cudf_polars/experimental/parallel.py +++ b/python/cudf_polars/cudf_polars/experimental/parallel.py @@ -7,7 +7,7 @@ import itertools import operator from functools import reduce -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, ClassVar import cudf_polars.experimental.io import cudf_polars.experimental.join @@ -24,10 +24,38 @@ if TYPE_CHECKING: from collections.abc import MutableMapping + from distributed import Client + from cudf_polars.containers import DataFrame from cudf_polars.experimental.dispatch import LowerIRTransformer +class SerializerManager: + """Manager to ensure ensure serializer is only registered once.""" + + _serializer_registered: bool = False + _client_run_executed: ClassVar[set[str]] = set() + + @classmethod + def register_serialize(cls) -> None: + """Register Dask/cudf-polars serializers in calling process.""" + if not cls._serializer_registered: + from cudf_polars.experimental.dask_serialize import register + + register() + cls._serializer_registered = True + + @classmethod + def run_on_cluster(cls, client: Client) -> None: + """Run serializer registration on the workers and scheduler.""" + if ( + client.id not in cls._client_run_executed + ): # pragma: no cover; Only executes with Distributed scheduler + client.run(cls.register_serialize) + client.run_on_scheduler(cls.register_serialize) + cls._client_run_executed.add(client.id) + + @lower_ir_node.register(IR) def _(ir: IR, rec: LowerIRTransformer) -> tuple[IR, MutableMapping[IR, PartitionInfo]]: # Default logic - Requires single partition @@ -127,12 +155,32 @@ def task_graph( return graph, (key_name, 0) +def get_client(): + """Get appropriate Dask client or scheduler.""" + SerializerManager.register_serialize() + + try: # pragma: no cover; block depends on executor type and Distributed cluster + from distributed import get_client + + client = get_client() + SerializerManager.run_on_cluster(client) + except ( + ImportError, + ValueError, + ): # pragma: no cover; block depends on Dask local scheduler + from dask import get + + return get + else: # pragma: no cover; block depends on executor type and Distributed cluster + return client.get + + def evaluate_dask(ir: IR) -> DataFrame: """Evaluate an IR graph with Dask.""" - from dask import get - ir, partition_info = lower_ir_graph(ir) + get = get_client() + graph, key = task_graph(ir, partition_info) return get(graph, key) diff --git a/python/cudf_polars/tests/conftest.py b/python/cudf_polars/tests/conftest.py index 6338bf0cae1..dbd0989a8b2 100644 --- a/python/cudf_polars/tests/conftest.py +++ b/python/cudf_polars/tests/conftest.py @@ -1,9 +1,11 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 from __future__ import annotations import pytest +DISTRIBUTED_CLUSTER_KEY = pytest.StashKey[dict]() + @pytest.fixture(params=[False, True], ids=["no_nulls", "nulls"], scope="session") def with_nulls(request): @@ -19,8 +21,50 @@ def pytest_addoption(parser): help="Executor to use for GPUEngine.", ) + parser.addoption( + "--dask-cluster", + action="store_true", + help="Executor to use for GPUEngine.", + ) + def pytest_configure(config): import cudf_polars.testing.asserts + if ( + config.getoption("--dask-cluster") + and config.getoption("--executor") != "dask-experimental" + ): + raise pytest.UsageError( + "--dask-cluster requires --executor='dask-experimental'" + ) + cudf_polars.testing.asserts.Executor = config.getoption("--executor") + + +def pytest_sessionstart(session): + if ( + session.config.getoption("--dask-cluster") + and session.config.getoption("--executor") == "dask-experimental" + ): + from dask import config + from dask.distributed import Client, LocalCluster + + # Avoid "Sending large graph of size ..." warnings + # (We expect these for tests using literal/random arrays) + config.set({"distributed.admin.large-graph-warning-threshold": "20MB"}) + + cluster = LocalCluster() + client = Client(cluster) + session.stash[DISTRIBUTED_CLUSTER_KEY] = {"cluster": cluster, "client": client} + + +def pytest_sessionfinish(session): + if DISTRIBUTED_CLUSTER_KEY in session.stash: + cluster_info = session.stash[DISTRIBUTED_CLUSTER_KEY] + client = cluster_info.get("client") + cluster = cluster_info.get("cluster") + if client is not None: + client.shutdown() + if cluster is not None: + cluster.close() From 27d40b90fb8d1b7abe1e7b2fbdd50c3f507f45c2 Mon Sep 17 00:00:00 2001 From: Kyle Edwards Date: Tue, 25 Feb 2025 11:49:52 -0500 Subject: [PATCH 084/129] Remove `FindCUDAToolkit.cmake` from `.pre-commit-config.yaml` (#18087) Follow-up to #18081. Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18087 --- .pre-commit-config.yaml | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 5daf124d83b..889e07bc681 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -107,10 +107,6 @@ repos: - cmakelang==0.6.13 verbose: true require_serial: true - exclude: | - (?x)^( - cpp/cmake/Modules/FindCUDAToolkit[.]cmake$ - ) - id: cmake-lint name: cmake-lint entry: ./cpp/scripts/run-cmake-format.sh cmake-lint @@ -122,10 +118,6 @@ repos: - cmakelang==0.6.13 verbose: true require_serial: true - exclude: | - (?x)^( - cpp/cmake/Modules/FindCUDAToolkit[.]cmake$ - ) - id: doxygen-check name: doxygen-check entry: ./ci/checks/doxygen.sh @@ -159,8 +151,7 @@ repos: (?x)^( cpp/include/cudf_test/cxxopts[.]hpp$| cpp/src/io/parquet/ipc/Message_generated[.]h$| - cpp/src/io/parquet/ipc/Schema_generated[.]h$| - cpp/cmake/Modules/FindCUDAToolkit[.]cmake$ + cpp/src/io/parquet/ipc/Schema_generated[.]h$ ) - id: verify-alpha-spec - id: verify-codeowners From 54c15b2a1a61f4d88437ab0433eecf27241bda77 Mon Sep 17 00:00:00 2001 From: Bradley Dice Date: Tue, 25 Feb 2025 16:21:01 -0600 Subject: [PATCH 085/129] Use conda-build instead of conda-mambabuild (#18092) This changes from `conda mambabuild` to `conda build`. Conda now uses the mamba solver so no performance regressions are expected. This is a temporary change as we plan to migrate to `rattler-build` in the near future. However, this is needed sooner to drop `boa` and unblock Python 3.13 migrations. xref: https://github.com/rapidsai/build-planning/issues/149 Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) - Jake Awe (https://github.com/AyodeAwe) URL: https://github.com/rapidsai/cudf/pull/18092 --- ci/build_cpp.sh | 4 ++-- ci/build_python.sh | 14 +++++++------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/ci/build_cpp.sh b/ci/build_cpp.sh index 3d06eacf9ff..0c324d01cdf 100755 --- a/ci/build_cpp.sh +++ b/ci/build_cpp.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. set -euo pipefail @@ -18,7 +18,7 @@ rapids-logger "Begin cpp build" sccache --zero-stats # With boa installed conda build forward to boa -RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) rapids-conda-retry mambabuild \ +RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) rapids-conda-retry build \ conda/recipes/libcudf sccache --show-adv-stats diff --git a/ci/build_python.sh b/ci/build_python.sh index ed90041cc77..abbdc3f3a3b 100755 --- a/ci/build_python.sh +++ b/ci/build_python.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. set -euo pipefail @@ -25,7 +25,7 @@ sccache --zero-stats # node works correctly # With boa installed conda build forwards to the boa builder -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ +RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ --no-test \ --channel "${CPP_CHANNEL}" \ conda/recipes/pylibcudf @@ -33,7 +33,7 @@ RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ sccache --show-adv-stats sccache --zero-stats -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ +RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ --no-test \ --channel "${CPP_CHANNEL}" \ --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ @@ -42,13 +42,13 @@ RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ sccache --show-adv-stats sccache --zero-stats -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ +RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ --no-test \ --channel "${CPP_CHANNEL}" \ --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ conda/recipes/dask-cudf -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ +RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ --no-test \ --channel "${CPP_CHANNEL}" \ --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ @@ -56,13 +56,13 @@ RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ sccache --show-adv-stats -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ +RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ --no-test \ --channel "${CPP_CHANNEL}" \ --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ conda/recipes/custreamz -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ +RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ --no-test \ --channel "${CPP_CHANNEL}" \ --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ From 0f7a17f8767dfe5c00ea31feb894cf38a9fc1b6d Mon Sep 17 00:00:00 2001 From: Vyas Ramasubramani Date: Tue, 25 Feb 2025 15:17:40 -0800 Subject: [PATCH 086/129] Update numba dep and upper-bound numpy (#18078) This PR updates to numba-cuda 0.4 and numba 0.61. A numpy upper-bound is added since it looks like numpy 2.1 made some changes with which we are currently incompatible. Previously numba provided that upper bound for us. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/18078 --- conda/environments/all_cuda-118_arch-x86_64.yaml | 6 +++--- conda/environments/all_cuda-128_arch-x86_64.yaml | 6 +++--- conda/recipes/cudf/meta.yaml | 6 +++--- conda/recipes/pylibcudf/meta.yaml | 2 +- dependencies.yaml | 9 +++++---- python/cudf/pyproject.toml | 6 +++--- python/cudf_polars/pyproject.toml | 2 +- python/dask_cudf/pyproject.toml | 6 +++--- python/pylibcudf/pyproject.toml | 2 +- 9 files changed, 23 insertions(+), 22 deletions(-) diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index cc674732ba4..e7dbb765099 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -54,9 +54,9 @@ dependencies: - nbsphinx - ninja - notebook -- numba-cuda>=0.2.0,<0.3.0a0 -- numba>=0.59.1,<0.61.0a0 -- numpy>=1.23,<3.0a0 +- numba-cuda>=0.4.0,<0.5.0a0 +- numba>=0.59.1,<0.62.0a0 +- numpy>=1.23,<2.1 - numpydoc - nvcc_linux-64=11.8 - nvcomp==4.2.0.11 diff --git a/conda/environments/all_cuda-128_arch-x86_64.yaml b/conda/environments/all_cuda-128_arch-x86_64.yaml index 7593a72cc68..342ec8d4b59 100644 --- a/conda/environments/all_cuda-128_arch-x86_64.yaml +++ b/conda/environments/all_cuda-128_arch-x86_64.yaml @@ -53,9 +53,9 @@ dependencies: - nbsphinx - ninja - notebook -- numba-cuda>=0.2.0,<0.3.0a0 -- numba>=0.59.1,<0.61.0a0 -- numpy>=1.23,<3.0a0 +- numba-cuda>=0.4.0,<0.5.0a0 +- numba>=0.59.1,<0.62.0a0 +- numpy>=1.23,<2.1 - numpydoc - nvcomp==4.2.0.11 - nvtx>=0.2.1 diff --git a/conda/recipes/cudf/meta.yaml b/conda/recipes/cudf/meta.yaml index f817bc12c5b..43060ef1c87 100644 --- a/conda/recipes/cudf/meta.yaml +++ b/conda/recipes/cudf/meta.yaml @@ -75,9 +75,9 @@ requirements: - typing_extensions >=4.0.0 - pandas >=2.0,<2.2.4dev0 - cupy >=12.0.0 - - numba-cuda >=0.2.0,<0.3.0a0 - - numba >=0.59.1,<0.61.0a0 - - numpy >=1.23,<3.0a0 + - numba-cuda >=0.4.0,<0.5.0a0 + - numba >=0.59.1,<0.62.0a0 + - numpy >=1.23,<2.1 - pyarrow>=14.0.0,<20.0.0a0 - libcudf ={{ version }} - pylibcudf ={{ version }} diff --git a/conda/recipes/pylibcudf/meta.yaml b/conda/recipes/pylibcudf/meta.yaml index 14e2f31a5a5..ae02cf8d4e5 100644 --- a/conda/recipes/pylibcudf/meta.yaml +++ b/conda/recipes/pylibcudf/meta.yaml @@ -73,7 +73,7 @@ requirements: - python - typing_extensions >=4.0.0 - pandas >=2.0,<2.2.4dev0 - - numpy >=1.23,<3.0a0 + - numpy >=1.23,<2.1 - pyarrow>=14.0.0,<20.0.0a0 - libcudf ={{ version }} - {{ pin_compatible('rmm', max_pin='x.x') }} diff --git a/dependencies.yaml b/dependencies.yaml index e7840d56880..c7869eee922 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -723,7 +723,7 @@ dependencies: - output_types: [conda, requirements, pyproject] packages: - fsspec>=0.6.0 - - &numpy numpy>=1.23,<3.0a0 + - &numpy numpy>=1.23,<2.1 - pandas>=2.0,<2.2.4dev0 run_pylibcudf: common: @@ -753,8 +753,8 @@ dependencies: - output_types: [conda, requirements, pyproject] packages: - cachetools - - &numba-cuda-dep numba-cuda>=0.2.0,<0.3.0a0 - - &numba-dep numba>=0.59.1,<0.61.0a0 + - &numba-cuda-dep numba-cuda>=0.4.0,<0.5.0a0 + - &numba-dep numba>=0.59.1,<0.62.0a0 - nvtx>=0.2.1 - packaging - rich @@ -885,7 +885,8 @@ dependencies: matrices: - matrix: {dependencies: "oldest"} packages: - - numba-cuda==0.2.0 + - numba-cuda==0.4.0 + - numba==0.59.1 - pandas==2.0.* - matrix: {dependencies: "latest"} packages: diff --git a/python/cudf/pyproject.toml b/python/cudf/pyproject.toml index 16cd97677ef..8b8abe90ac9 100644 --- a/python/cudf/pyproject.toml +++ b/python/cudf/pyproject.toml @@ -24,9 +24,9 @@ dependencies = [ "cupy-cuda11x>=12.0.0", "fsspec>=0.6.0", "libcudf==25.4.*,>=0.0.0a0", - "numba-cuda>=0.2.0,<0.3.0a0", - "numba>=0.59.1,<0.61.0a0", - "numpy>=1.23,<3.0a0", + "numba-cuda>=0.4.0,<0.5.0a0", + "numba>=0.59.1,<0.62.0a0", + "numpy>=1.23,<2.1", "nvtx>=0.2.1", "packaging", "pandas>=2.0,<2.2.4dev0", diff --git a/python/cudf_polars/pyproject.toml b/python/cudf_polars/pyproject.toml index 872c08a66f9..9026a0c29ca 100644 --- a/python/cudf_polars/pyproject.toml +++ b/python/cudf_polars/pyproject.toml @@ -35,7 +35,7 @@ classifiers = [ [project.optional-dependencies] test = [ - "numpy>=1.23,<3.0a0", + "numpy>=1.23,<2.1", "pytest-cov", "pytest-xdist", "pytest<8", diff --git a/python/dask_cudf/pyproject.toml b/python/dask_cudf/pyproject.toml index 87bf282f376..83493d7f2a4 100644 --- a/python/dask_cudf/pyproject.toml +++ b/python/dask_cudf/pyproject.toml @@ -22,7 +22,7 @@ dependencies = [ "cudf==25.4.*,>=0.0.0a0", "cupy-cuda11x>=12.0.0", "fsspec>=0.6.0", - "numpy>=1.23,<3.0a0", + "numpy>=1.23,<2.1", "pandas>=2.0,<2.2.4dev0", "pynvml>=12.0.0,<13.0.0a0", "rapids-dask-dependency==25.4.*,>=0.0.0a0", @@ -47,8 +47,8 @@ cudf = "dask_cudf.backends:CudfBackendEntrypoint" [project.optional-dependencies] test = [ "dask-cuda==25.4.*,>=0.0.0a0", - "numba-cuda>=0.2.0,<0.3.0a0", - "numba>=0.59.1,<0.61.0a0", + "numba-cuda>=0.4.0,<0.5.0a0", + "numba>=0.59.1,<0.62.0a0", "pytest-cov", "pytest-xdist", "pytest<8", diff --git a/python/pylibcudf/pyproject.toml b/python/pylibcudf/pyproject.toml index 939da65c1ec..e12d1ffdb39 100644 --- a/python/pylibcudf/pyproject.toml +++ b/python/pylibcudf/pyproject.toml @@ -42,7 +42,7 @@ classifiers = [ test = [ "fastavro>=0.22.9", "hypothesis", - "numpy>=1.23,<3.0a0", + "numpy>=1.23,<2.1", "pandas", "pytest-cov", "pytest-xdist", From 8d6bdc34c4b2d0d6be614c04af16b8064d2c723d Mon Sep 17 00:00:00 2001 From: Vyas Ramasubramani Date: Tue, 25 Feb 2025 15:21:19 -0800 Subject: [PATCH 087/129] Remove static configure step (#18091) This check has been superseded by #17781. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - James Lamb (https://github.com/jameslamb) URL: https://github.com/rapidsai/cudf/pull/18091 --- .github/workflows/pr.yaml | 11 ----------- ci/configure_cpp_static.sh | 21 --------------------- 2 files changed, 32 deletions(-) delete mode 100755 ci/configure_cpp_static.sh diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 38b890893d0..2c583598f54 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -24,7 +24,6 @@ jobs: - conda-python-cudf-tests - conda-python-other-tests - conda-java-tests - - static-configure - conda-notebook-tests - docs-build - wheel-build-libcudf @@ -192,16 +191,6 @@ jobs: arch: "amd64" container_image: "rapidsai/ci-conda:latest" run_script: "ci/test_java.sh" - static-configure: - needs: checks - secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 - with: - build_type: pull-request - # Use the wheel container so we can skip conda solves and since our - # primary static consumers (Spark) are not in conda anyway. - container_image: "rapidsai/ci-wheel:latest" - run_script: "ci/configure_cpp_static.sh" conda-notebook-tests: needs: [conda-python-build, changed-files] secrets: inherit diff --git a/ci/configure_cpp_static.sh b/ci/configure_cpp_static.sh deleted file mode 100755 index 3d0647a96f6..00000000000 --- a/ci/configure_cpp_static.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/bin/bash -# Copyright (c) 2024-2025, NVIDIA CORPORATION. - -set -euo pipefail - -source rapids-date-string - -rapids-logger "Configure static cpp build" - -ENV_YAML_DIR="$(mktemp -d)" -REQUIREMENTS_FILE="${ENV_YAML_DIR}/requirements.txt" - -rapids-dependency-file-generator \ - --output requirements \ - --file-key test_static_build \ - --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee "${REQUIREMENTS_FILE}" - -rapids-pip-retry install -r "${REQUIREMENTS_FILE}" -pyenv rehash - -cmake -S cpp -B build_static -GNinja -DBUILD_SHARED_LIBS=OFF -DCUDF_USE_ARROW_STATIC=ON -DBUILD_TESTS=OFF From e365986cf886fe3a9531952fe5b91a34ca466c45 Mon Sep 17 00:00:00 2001 From: Bradley Dice Date: Tue, 25 Feb 2025 17:32:23 -0600 Subject: [PATCH 088/129] Run narwhals tests nightly. (#18093) This enables narwhals tests in nightly CI. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Matthew Murray (https://github.com/Matt711) - Gil Forsyth (https://github.com/gforsyth) URL: https://github.com/rapidsai/cudf/pull/18093 --- .github/workflows/test.yaml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 12f6d751493..7046fd0e5dc 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -168,3 +168,14 @@ jobs: date: ${{ inputs.date }} sha: ${{ inputs.sha }} script: "ci/test_cudf_polars_polars_tests.sh" + narwhals-tests: + secrets: inherit + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 + with: + build_type: ${{ inputs.build_type }} + branch: ${{ inputs.branch }} + date: ${{ inputs.date }} + sha: ${{ inputs.sha }} + node_type: "gpu-l4-latest-1" + container_image: "rapidsai/ci-conda:latest" + run_script: ci/test_narwhals.sh From 18a5412ced238630bb1a6f5b15e6f319dd388090 Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Tue, 25 Feb 2025 18:57:00 -0500 Subject: [PATCH 089/129] Add new nvtext::normalize_characters API (#17818) Adds new normalizer APIs as part of the rework for the subword-tokenizer. The new API is split into 2 parts. First a normalizer object is created with appropriate state: lower-case and special-tokens. The normalizing tables are currently hardcoded inside libcudf. Future versions of the this may load these tables from some other source. The 2nd API is given the input strings column and the normalizer object and returns a normalized strings column. The normalizer object can be reused on all subsequent `normalize_characters` calls. The current `nvtext::normalize_characters` loads the normalizing tables on each call which can be significant overhead. This API will be deprecated and replaced by these 2 new ones. Some utility functions from that implementation have been refactored to be used by both until the old one is removed. The first API creates the normalizer object. ```cpp std::unique_ptr create_character_normalizer( bool do_lower_case, cudf::strings_column_view const& special_tokens, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr); ``` The 2nd API uses the normalizer on a strings column: ```cpp std::unique_ptr normalize_characters( cudf::strings_column_view const& input, character_normalizer const& normalizer, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr); ``` Using the python interface: ```python import cudf from cudf.core.character_normalizer import CharacterNormalizer cn = CharacterNormalizer(do_lower=False) sn = cn.normalize(input_strings) ``` Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Tianyu Liu (https://github.com/kingcrimsontianyu) - Karthikeyan (https://github.com/karthikeyann) - Matthew Murray (https://github.com/Matt711) URL: https://github.com/rapidsai/cudf/pull/17818 --- cpp/benchmarks/text/normalize.cpp | 9 +- cpp/include/cudf/strings/detail/utilities.hpp | 14 +- cpp/include/nvtext/normalize.hpp | 111 ++++- cpp/src/strings/utilities.cu | 14 +- cpp/src/text/normalize.cu | 395 +++++++++++++++++- cpp/src/text/normalize.cuh | 100 +++++ cpp/src/text/subword/data_normalizer.cu | 76 +--- cpp/tests/text/normalize_tests.cpp | 165 +++++++- python/cudf/cudf/core/character_normalizer.py | 46 ++ python/cudf/cudf/core/column/string.py | 28 +- .../cudf/cudf/tests/text/test_text_methods.py | 8 +- .../pylibcudf/libcudf/nvtext/normalize.pxd | 15 +- .../pylibcudf/pylibcudf/nvtext/normalize.pxd | 13 +- .../pylibcudf/pylibcudf/nvtext/normalize.pyi | 10 +- .../pylibcudf/pylibcudf/nvtext/normalize.pyx | 71 +++- .../pylibcudf/tests/test_nvtext_normalize.py | 97 ++++- 16 files changed, 1018 insertions(+), 154 deletions(-) create mode 100644 cpp/src/text/normalize.cuh create mode 100644 python/cudf/cudf/core/character_normalizer.py diff --git a/cpp/benchmarks/text/normalize.cpp b/cpp/benchmarks/text/normalize.cpp index 594dc0de28a..494d5722ae4 100644 --- a/cpp/benchmarks/text/normalize.cpp +++ b/cpp/benchmarks/text/normalize.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -48,8 +48,11 @@ static void bench_normalize(nvbench::state& state) [&](nvbench::launch& launch) { auto result = nvtext::normalize_spaces(input); }); } else { bool const to_lower = (normalize_type == "to_lower"); + // we expect the normalizer to be created once and re-used + // so creating it is not measured + auto normalizer = nvtext::create_character_normalizer(to_lower); state.exec(nvbench::exec_tag::sync, [&](nvbench::launch& launch) { - auto result = nvtext::normalize_characters(input, to_lower); + auto result = nvtext::normalize_characters(input, *normalizer); }); } } @@ -57,6 +60,6 @@ static void bench_normalize(nvbench::state& state) NVBENCH_BENCH(bench_normalize) .set_name("normalize") .add_int64_axis("min_width", {0}) - .add_int64_axis("max_width", {32, 64, 128, 256}) + .add_int64_axis("max_width", {128, 256}) .add_int64_axis("num_rows", {32768, 262144, 2097152}) .add_string_axis("type", {"spaces", "characters", "to_lower"}); diff --git a/cpp/include/cudf/strings/detail/utilities.hpp b/cpp/include/cudf/strings/detail/utilities.hpp index d276c5df7dc..8fb1f30f961 100644 --- a/cpp/include/cudf/strings/detail/utilities.hpp +++ b/cpp/include/cudf/strings/detail/utilities.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -96,5 +96,17 @@ int64_t get_offset_value(cudf::column_view const& offsets, size_type index, rmm::cuda_stream_view stream); +/** + * @brief Return the first and last offset in the given strings column + * + * This accounts for sliced input columns as well. + * + * @param input Strings column + * @param stream CUDA stream used for device memory operations and kernel launches + * @return First and last offset values + */ +std::pair get_first_and_last_offset(cudf::strings_column_view const& input, + rmm::cuda_stream_view stream); + } // namespace strings::detail } // namespace CUDF_EXPORT cudf diff --git a/cpp/include/nvtext/normalize.hpp b/cpp/include/nvtext/normalize.hpp index 74325f4a406..70ee7891ad7 100644 --- a/cpp/include/nvtext/normalize.hpp +++ b/cpp/include/nvtext/normalize.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -16,6 +16,7 @@ #pragma once #include +#include #include #include #include @@ -107,5 +108,113 @@ std::unique_ptr normalize_characters( rmm::cuda_stream_view stream = cudf::get_default_stream(), rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); +/** + * @brief Normalizer object to be used with nvtext::normalize_characters + * + * Use nvtext::create_normalizer to create this object. + * + * This normalizer includes: + * + * - adding padding around punctuation (unicode category starts with "P") + * as well as certain ASCII symbols like "^" and "$" + * - adding padding around the [CJK Unicode block + * characters](https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)) + * - changing whitespace (e.g. `"\t", "\n", "\r"`) to just space `" "` + * - removing control characters (unicode categories "Cc" and "Cf") + * + * The padding process adds a single space before and after the character. + * Details on _unicode category_ can be found here: + * https://unicodebook.readthedocs.io/unicode.html#categories + * + * If `do_lower_case = true`, lower-casing also removes any accents. The + * accents cannot be removed from upper-case characters without lower-casing + * and lower-casing cannot be performed without also removing accents. + * However, if the accented character is already lower-case, then only the + * accent is removed. + * + * If `special_tokens` are included the padding after `[` and before `]` is not + * inserted if the characters between them match one of the given tokens. + * Also, the `special_tokens` are expected to include the `[]` characters + * at the beginning of and end of each string appropriately. + */ +struct character_normalizer { + /** + * @brief Normalizer object constructor + * + * This initializes and holds the character normalizing tables and settings. + * + * @param do_lower_case If true, upper-case characters are converted to + * lower-case and accents are stripped from those characters. + * If false, accented and upper-case characters are not transformed. + * @param special_tokens Each row is a token including the `[]` brackets. + * For example: `[BOS]`, `[EOS]`, `[UNK]`, `[SEP]`, `[PAD]`, `[CLS]`, `[MASK]` + * @param stream CUDA stream used for device memory operations and kernel launches + * @param mr Device memory resource used to allocate the returned column's device memory + */ + character_normalizer(bool do_lower_case, + cudf::strings_column_view const& special_tokens, + rmm::cuda_stream_view stream = cudf::get_default_stream(), + rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); + ~character_normalizer(); + + struct character_normalizer_impl; + std::unique_ptr _impl; +}; + +/** + * @brief Create a normalizer object + * + * Creates a normalizer object which can be reused on multiple calls to + * nvtext::normalize_characters + * + * @see nvtext::character_normalizer + * + * @param do_lower_case If true, upper-case characters are converted to + * lower-case and accents are stripped from those characters. + * If false, accented and upper-case characters are not transformed. + * @param special_tokens Individual tokens including `[]` brackets. + * Default is no special tokens. + * @param stream CUDA stream used for device memory operations and kernel launches + * @param mr Device memory resource used to allocate the returned column's device memory + * @return Object to be used with nvtext::normalize_characters + */ +std::unique_ptr create_character_normalizer( + bool do_lower_case, + cudf::strings_column_view const& special_tokens = cudf::strings_column_view(cudf::column_view{ + cudf::data_type{cudf::type_id::STRING}, 0, nullptr, nullptr, 0}), + rmm::cuda_stream_view stream = cudf::get_default_stream(), + rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); + +/** + * @brief Normalizes the text in input strings column + * + * @see nvtext::character_normalizer for details on the normalizer behavior + * + * @code{.pseudo} + * cn = create_character_normalizer(true) + * s = ["éâîô\teaio", "ĂĆĖÑÜ", "ACENU", "$24.08", "[a,bb]"] + * s1 = normalize_characters(s,cn) + * s1 is now ["eaio eaio", "acenu", "acenu", " $ 24 . 08", " [ a , bb ] "] + * + * cn = create_character_normalizer(false) + * s2 = normalize_characters(s,cn) + * s2 is now ["éâîô eaio", "ĂĆĖÑÜ", "ACENU", " $ 24 . 08", " [ a , bb ] "] + * @endcode + * + * A null input element at row `i` produces a corresponding null entry + * for row `i` in the output column. + * + * @param input The input strings to normalize + * @param normalizer Normalizer to use for this function + * @param stream CUDA stream used for device memory operations and kernel launches + * @param mr Memory resource to allocate any returned objects + * @return Normalized strings column + */ +std::unique_ptr normalize_characters( + cudf::strings_column_view const& input, + character_normalizer const& normalizer, + rmm::cuda_stream_view stream = cudf::get_default_stream(), + rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); + /** @} */ // end of group } // namespace CUDF_EXPORT nvtext diff --git a/cpp/src/strings/utilities.cu b/cpp/src/strings/utilities.cu index 45bd4615435..c5d46598d4a 100644 --- a/cpp/src/strings/utilities.cu +++ b/cpp/src/strings/utilities.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -180,6 +180,18 @@ int64_t get_offset_value(cudf::column_view const& offsets, : cudf::detail::get_value(offsets, index, stream); } +std::pair get_first_and_last_offset(cudf::strings_column_view const& input, + rmm::cuda_stream_view stream) +{ + if (input.is_empty()) { return {0L, 0L}; } + auto const first_offset = (input.offset() == 0) ? 0 + : cudf::strings::detail::get_offset_value( + input.offsets(), input.offset(), stream); + auto const last_offset = + cudf::strings::detail::get_offset_value(input.offsets(), input.size() + input.offset(), stream); + return {first_offset, last_offset}; +} + } // namespace detail rmm::device_uvector create_string_vector_from_column( diff --git a/cpp/src/text/normalize.cu b/cpp/src/text/normalize.cu index 7e2b766862d..0e680e98ec5 100644 --- a/cpp/src/text/normalize.cu +++ b/cpp/src/text/normalize.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -14,6 +14,7 @@ * limitations under the License. */ +#include "text/normalize.cuh" #include "text/subword/detail/data_normalizer.hpp" #include "text/subword/detail/tokenizer_utils.cuh" #include "text/utilities/tokenize_ops.cuh" @@ -22,10 +23,11 @@ #include #include #include -#include #include #include #include +#include +#include #include #include #include @@ -38,9 +40,13 @@ #include +#include +#include +#include #include #include #include +#include #include #include @@ -103,6 +109,12 @@ constexpr uint32_t UTF8_1BYTE = 0x0080; constexpr uint32_t UTF8_2BYTE = 0x0800; constexpr uint32_t UTF8_3BYTE = 0x01'0000; +__device__ int8_t cp_to_utf8(uint32_t codepoint, char* out) +{ + auto utf8 = cudf::strings::detail::codepoint_to_utf8(codepoint); + return cudf::strings::detail::from_char_utf8(utf8, out); +} + /** * @brief Convert code-point arrays into UTF-8 bytes for each string. */ @@ -148,26 +160,8 @@ struct codepoint_to_utf8_fn { // convert each code-point to 1-4 UTF-8 encoded bytes char* out_ptr = d_chars + d_offsets[idx]; for (uint32_t jdx = 0; jdx < count; ++jdx) { - uint32_t code_point = *str_cps++; - if (code_point < UTF8_1BYTE) // ASCII range - *out_ptr++ = static_cast(code_point); - else if (code_point < UTF8_2BYTE) { // create two-byte UTF-8 - // b00001xxx:byyyyyyyy => b110xxxyy:b10yyyyyy - *out_ptr++ = static_cast((((code_point << 2) & 0x00'1F00) | 0x00'C000) >> 8); - *out_ptr++ = static_cast((code_point & 0x3F) | 0x0080); - } else if (code_point < UTF8_3BYTE) { // create three-byte UTF-8 - // bxxxxxxxx:byyyyyyyy => b1110xxxx:b10xxxxyy:b10yyyyyy - *out_ptr++ = static_cast((((code_point << 4) & 0x0F'0000) | 0x00E0'0000) >> 16); - *out_ptr++ = static_cast((((code_point << 2) & 0x00'3F00) | 0x00'8000) >> 8); - *out_ptr++ = static_cast((code_point & 0x3F) | 0x0080); - } else { // create four-byte UTF-8 - // maximum code-point value is 0x0011'0000 - // b000xxxxx:byyyyyyyy:bzzzzzzzz => b11110xxx:b10xxyyyy:b10yyyyzz:b10zzzzzz - *out_ptr++ = static_cast((((code_point << 6) & 0x0700'0000u) | 0xF000'0000u) >> 24); - *out_ptr++ = static_cast((((code_point << 4) & 0x003F'0000u) | 0x0080'0000u) >> 16); - *out_ptr++ = static_cast((((code_point << 2) & 0x00'3F00u) | 0x00'8000u) >> 8); - *out_ptr++ = static_cast((code_point & 0x3F) | 0x0080); - } + uint32_t codepoint = *str_cps++; + out_ptr += cp_to_utf8(codepoint, out_ptr); } } }; @@ -261,4 +255,361 @@ std::unique_ptr normalize_characters(cudf::strings_column_view con return detail::normalize_characters(input, do_lower_case, stream, mr); } +struct character_normalizer::character_normalizer_impl { + rmm::device_uvector cp_metadata; + rmm::device_uvector aux_table; + bool do_lower_case; + std::unique_ptr special_tokens; + rmm::device_uvector special_tokens_view; + + cudf::device_span get_special_tokens() const + { + return special_tokens_view; + } + + character_normalizer_impl(rmm::device_uvector&& cp_metadata, + rmm::device_uvector&& aux_table, + bool do_lower_case, + std::unique_ptr&& special_tokens, + rmm::device_uvector&& special_tokens_view) + : cp_metadata(std::move(cp_metadata)), + aux_table(std::move(aux_table)), + do_lower_case{do_lower_case}, + special_tokens{std::move(special_tokens)}, + special_tokens_view{std::move(special_tokens_view)} + { + } +}; + +character_normalizer::character_normalizer(bool do_lower_case, + cudf::strings_column_view const& special_tokens, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref) +{ + auto cp_metadata = nvtext::detail::get_codepoint_metadata(stream); + auto aux_table = nvtext::detail::get_aux_codepoint_data(stream); + CUDF_EXPECTS( + !special_tokens.has_nulls(), "special tokens should not have nulls", std::invalid_argument); + + auto sorted = std::move( + cudf::sort(cudf::table_view({special_tokens.parent()}), {}, {}, stream)->release().front()); + if (do_lower_case) { + // lower-case the tokens so they will match the normalized input + sorted = cudf::strings::to_lower(cudf::strings_column_view(sorted->view()), stream); + } + + auto tokens_view = cudf::strings::detail::create_string_vector_from_column( + cudf::strings_column_view(sorted->view()), stream, cudf::get_current_device_resource_ref()); + + _impl = std::make_unique(std::move(cp_metadata), + std::move(aux_table), + do_lower_case, + std::move(sorted), + std::move(tokens_view)); +} + +character_normalizer::~character_normalizer() {} + +std::unique_ptr create_character_normalizer( + bool do_lower_case, + cudf::strings_column_view const& special_tokens, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) +{ + CUDF_FUNC_RANGE(); + return std::make_unique(do_lower_case, special_tokens, stream, mr); +} + +namespace detail { +namespace { + +/** + * @brief Kernel handles fixing up the normalized data to account for any special tokens + * + * This undoes the padding added around the `[]` for patterns matching the strings in the + * special_tokens array. + * + * Launched as a thread per input byte (total_count). + * + * @param d_normalized The normalized set of UTF-8 characters; 3 uints per input byte + * @param total_count Number of bytes represented by d_normalized; len(d_normalized)/3 + * @param special_tokens Tokens to check against + */ +CUDF_KERNEL void special_tokens_kernel(uint32_t* d_normalized, + int64_t total_count, + cudf::device_span special_tokens) +{ + auto const idx = cudf::detail::grid_1d::global_thread_id(); + if (idx >= total_count) { return; } + auto const begin = d_normalized + (idx * MAX_NEW_CHARS) + 1; + if (*begin != '[') { return; } + auto const end = begin + cuda::std::min(6L, total_count - idx) * MAX_NEW_CHARS; + auto const match = thrust::find(thrust::seq, begin, end, static_cast(']')); + if (match == end) { return; } + char candidate[8]; + auto const ch_begin = + thrust::transform_iterator(begin, [](auto v) { return static_cast(v); }); + auto const ch_end = ch_begin + thrust::distance(begin, match + 1); + auto last = thrust::copy_if( + thrust::seq, ch_begin, ch_end, candidate, [](auto c) { return c != 0 && c != ' '; }); + *last = 0; // only needed for debug + + auto const size = static_cast(thrust::distance(candidate, last)); + auto const token = cudf::string_view(candidate, size); + // the binary_search expects the special_tokens to be sorted + if (!thrust::binary_search(thrust::seq, special_tokens.begin(), special_tokens.end(), token)) { + return; + } + + // fix up chars to remove the extra spaces + *(begin + 1) = 0; // removes space after '[' + *(match - 1) = 0; // removes space before ']' +} + +/** + * @brief The normalizer kernel + * + * Launched as a thread per input byte (total_bytes). + * + * Converts the input d_chars into codepoints to lookup in the provided tables. + * Once processed, the d_output contains 3 uints per input byte each encoded + * as output UTF-8. Any zero values are to removed by a subsequent kernel call. + * + * @param d_chars The characters for the input strings column to normalize + * @param total_bytes The number of bytes in the d_chars + * @param cp_metadata First lookup table for codepoint metadata + * @param aux_table Second lookup table containing possible replacement characters + * @param do_lower_case True if the normalization includes lower-casing characters + * @param d_output The output of the normalization (UTF-8 encoded) + */ +CUDF_KERNEL void data_normalizer_kernel(char const* d_chars, + int64_t total_bytes, + codepoint_metadata_type const* cp_metadata, + aux_codepoint_data_type const* aux_table, + bool do_lower_case, + uint32_t* d_output) +{ + uint32_t replacement[MAX_NEW_CHARS] = {0}; + + auto const idx = cudf::detail::grid_1d::global_thread_id(); + + if ((idx < total_bytes) && cudf::strings::detail::is_begin_utf8_char(d_chars[idx])) { + auto const cp = [utf8 = d_chars + idx] { + cudf::char_utf8 ch_utf8 = *utf8; + if (ch_utf8 > 0x7F) { cudf::strings::detail::to_char_utf8(utf8, ch_utf8); } + return cudf::strings::detail::utf8_to_codepoint(ch_utf8); + }(); + auto const metadata = cp_metadata[cp]; + + if (!should_remove_cp(metadata, do_lower_case)) { + int8_t num_new_chars = 1; + // retrieve the normalized value for cp + auto const new_cp = do_lower_case || always_replace(metadata) ? get_first_cp(metadata) : cp; + replacement[0] = new_cp == 0 ? cp : new_cp; + + if (do_lower_case && is_multi_char_transform(metadata)) { + auto const next_cps = aux_table[cp]; + replacement[1] = static_cast(next_cps >> 32); + replacement[2] = static_cast(next_cps & 0xFFFFFFFF); + num_new_chars = 2 + (replacement[2] != 0); + } + + if (should_add_spaces(metadata, do_lower_case) && (num_new_chars == 1)) { + replacement[1] = replacement[0]; + replacement[0] = SPACE_CODE_POINT; // add spaces around the new codepoint + replacement[2] = SPACE_CODE_POINT; + num_new_chars = 3; + } + + // convert codepoints back to UTF-8 in-place + for (int k = 0; k < num_new_chars; ++k) { + auto const new_cp = replacement[k]; + if (new_cp) { cp_to_utf8(new_cp, reinterpret_cast(replacement + k)); } + } + } + } + + // employ an optimized coalesced writer to output replacement as a block of transposed data + using block_store = + cub::BlockStore; + __shared__ typename block_store::TempStorage bs_stg; + auto block_base = d_output + blockIdx.x * blockDim.x * MAX_NEW_CHARS; + block_store(bs_stg).Store(block_base, replacement); +} + +/** + * @brief Computes the output sizes for each row + * + * The input offsets are used with segmented-reduce to count the number of + * non-zero values for each output row. + * + * @param d_normalized The UTF-8 encoded normalized values + * @param offsets These identify the row boundaries + * @param offset Only non-zero if the input column has been sliced + * @param size The number of output rows (sames as the number of input rows) + * @param stream Stream used for allocating device memory and launching kernels + * @return The sizes of each output row + */ +template +rmm::device_uvector compute_sizes(cudf::device_span d_normalized, + OffsetType offsets, + int64_t offset, + cudf::size_type size, + rmm::cuda_stream_view stream) +{ + auto output_sizes = rmm::device_uvector(size, stream); + + auto d_data = d_normalized.data(); + + // counts the non-zero bytes in the d_data array + auto d_in = cudf::detail::make_counting_transform_iterator( + 0, cuda::proclaim_return_type([d_data] __device__(auto idx) { + idx = idx * MAX_NEW_CHARS; + // transform function counts number of non-zero bytes in uint32_t value + auto tfn = [](uint32_t v) -> cudf::size_type { + return ((v & 0xFF) > 0) + ((v & 0xFF00) > 0) + ((v & 0xFF0000) > 0) + + ((v & 0xFF000000) > 0); + }; + auto const begin = d_data + idx; + auto const end = begin + MAX_NEW_CHARS; + return thrust::transform_reduce(thrust::seq, begin, end, tfn, 0, thrust::plus{}); + })); + + // DeviceSegmentedReduce is used to compute the size of each output row + auto d_out = output_sizes.begin(); + auto temp = std::size_t{0}; + if (offset == 0) { + cub::DeviceSegmentedReduce::Sum( + nullptr, temp, d_in, d_out, size, offsets, offsets + 1, stream.value()); + auto d_temp = rmm::device_buffer{temp, stream}; + cub::DeviceSegmentedReduce::Sum( + d_temp.data(), temp, d_in, d_out, size, offsets, offsets + 1, stream.value()); + } else { + // offsets need to be normalized for segmented-reduce to work efficiently + auto offsets_itr = thrust::transform_iterator( + offsets, + cuda::proclaim_return_type([offset] __device__(auto o) { return o - offset; })); + cub::DeviceSegmentedReduce::Sum( + nullptr, temp, d_in, d_out, size, offsets_itr, offsets_itr + 1, stream.value()); + auto d_temp = rmm::device_buffer{temp, stream}; + cub::DeviceSegmentedReduce::Sum( + d_temp.data(), temp, d_in, d_out, size, offsets_itr, offsets_itr + 1, stream.value()); + } + + return output_sizes; +} + +// handles ranges above int32 max +template +OutputIterator remove_copy_safe(InputIterator first, + InputIterator last, + OutputIterator result, + T const& value, + rmm::cuda_stream_view stream) +{ + auto const copy_size = std::min(static_cast(std::distance(first, last)), + static_cast(std::numeric_limits::max())); + + auto itr = first; + while (itr != last) { + auto const copy_end = + static_cast(std::distance(itr, last)) <= copy_size ? last : itr + copy_size; + result = thrust::remove_copy(rmm::exec_policy(stream), itr, copy_end, result, value); + itr = copy_end; + } + return result; +} + +// handles ranges above int32 max +template +Iterator remove_safe(Iterator first, Iterator last, T const& value, rmm::cuda_stream_view stream) +{ + auto const size = std::min(static_cast(std::distance(first, last)), + static_cast(std::numeric_limits::max())); + + auto result = first; + auto itr = first; + while (itr != last) { + auto end = static_cast(std::distance(itr, last)) <= size ? last : itr + size; + result = thrust::remove(rmm::exec_policy(stream), itr, end, value); + itr = end; + } + return result; +} +} // namespace + +std::unique_ptr normalize_characters(cudf::strings_column_view const& input, + character_normalizer const& normalizer, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) +{ + if (input.is_empty()) { return cudf::make_empty_column(cudf::data_type{cudf::type_id::STRING}); } + + auto [first_offset, last_offset] = + cudf::strings::detail::get_first_and_last_offset(input, stream); + auto const chars_size = last_offset - first_offset; + auto const d_input_chars = input.chars_begin(stream) + first_offset; + + if (chars_size == 0) { return std::make_unique(input.parent(), stream, mr); } + + constexpr int64_t block_size = 256; + cudf::detail::grid_1d grid{chars_size, block_size}; + auto const max_new_char_total = cudf::util::round_up_safe(chars_size, block_size) * MAX_NEW_CHARS; + + auto const& parameters = normalizer._impl; + + auto d_normalized = rmm::device_uvector(max_new_char_total, stream); + data_normalizer_kernel<<>>( + d_input_chars, + chars_size, + parameters->cp_metadata.data(), + parameters->aux_table.data(), + parameters->do_lower_case, + d_normalized.data()); + + // This removes space added around any special tokens in the form of [ttt]. + // An alternate approach is to do a multi-replace of '[ ttt ]' with '[ttt]' right + // before returning the output strings column. + auto const special_tokens = parameters->get_special_tokens(); + if (!special_tokens.empty()) { + special_tokens_kernel<<>>( + d_normalized.data(), chars_size, special_tokens); + } + + // Use segmented-reduce over the non-zero codepoints to get the size of the output rows + auto const input_offsets = + cudf::detail::offsetalator_factory::make_input_iterator(input.offsets(), input.offset()); + auto output_sizes = + compute_sizes(d_normalized, input_offsets, first_offset, input.size(), stream); + + // convert the sizes to offsets + auto [offsets, total_size] = cudf::strings::detail::make_offsets_child_column( + output_sizes.begin(), output_sizes.end(), stream, mr); + + // create output chars by calling remove_copy(0) on the bytes in d_normalized + auto chars = rmm::device_uvector(total_size, stream, mr); + auto const begin = reinterpret_cast(d_normalized.begin()); + // the remove() above speeds up the remove_copy() by roughly 10% + auto const end = + reinterpret_cast(remove_safe(d_normalized.begin(), d_normalized.end(), 0, stream)); + remove_copy_safe(begin, end, chars.data(), 0, stream); + + return cudf::make_strings_column(input.size(), + std::move(offsets), + chars.release(), + input.null_count(), + cudf::detail::copy_bitmask(input.parent(), stream, mr)); +} + +} // namespace detail + +std::unique_ptr normalize_characters(cudf::strings_column_view const& input, + character_normalizer const& normalizer, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) +{ + CUDF_FUNC_RANGE(); + return detail::normalize_characters(input, normalizer, stream, mr); +} + } // namespace nvtext diff --git a/cpp/src/text/normalize.cuh b/cpp/src/text/normalize.cuh new file mode 100644 index 00000000000..3972726d536 --- /dev/null +++ b/cpp/src/text/normalize.cuh @@ -0,0 +1,100 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "text/subword/detail/cp_data.h" + +namespace nvtext { +namespace detail { + +/** + * @brief Bit used to filter out invalid code points. + * + * When normalizing characters to code point values, if this bit is set, + * the code point should be filtered out before returning from the normalizer. + */ +constexpr uint32_t FILTER_BIT = 22; + +/** + * @brief Retrieve new code point from metadata value. + * + * @param metadata Value from the codepoint_metadata table. + * @return The replacement character if appropriate. + */ +__device__ constexpr uint32_t get_first_cp(uint32_t metadata) { return metadata & NEW_CP_MASK; } + +/** + * @brief Retrieve token category from the metadata value. + * + * Category values are 0-5: + * 0 - character should be padded + * 1 - pad character if lower-case + * 2 - character should be removed + * 3 - remove character if lower-case + * 4 - whitespace character -- always replace + * 5 - uncategorized + * + * @param metadata Value from the codepoint_metadata table. + * @return Category value. + */ +__device__ constexpr uint32_t extract_token_cat(uint32_t metadata) +{ + return (metadata >> TOKEN_CAT_SHIFT) & TOKEN_CAT_MASK; +} + +/** + * @brief Return true if category of metadata value specifies the character should be replaced. + */ +__device__ constexpr bool should_remove_cp(uint32_t metadata, bool lower_case) +{ + auto const cat = extract_token_cat(metadata); + return (cat == TOKEN_CAT_REMOVE_CHAR) || (lower_case && (cat == TOKEN_CAT_REMOVE_CHAR_IF_LOWER)); +} + +/** + * @brief Return true if category of metadata value specifies the character should be padded. + */ +__device__ constexpr bool should_add_spaces(uint32_t metadata, bool lower_case) +{ + auto const cat = extract_token_cat(metadata); + return (cat == TOKEN_CAT_ADD_SPACE) || (lower_case && (cat == TOKEN_CAT_ADD_SPACE_IF_LOWER)); +} + +/** + * @brief Return true if category of metadata value specifies the character should be replaced. + */ +__device__ constexpr bool always_replace(uint32_t metadata) +{ + return extract_token_cat(metadata) == TOKEN_CAT_ALWAYS_REPLACE; +} + +/** + * @brief Returns true if metadata value includes a multi-character transform bit equal to 1. + */ +__device__ constexpr bool is_multi_char_transform(uint32_t metadata) +{ + return (metadata >> MULTICHAR_SHIFT) & MULTICHAR_MASK; +} + +/** + * @brief Returns true if the byte passed in could be a valid head byte for + * a utf8 character. That is, not binary `10xxxxxx` + */ +__device__ constexpr bool is_head_byte(unsigned char utf8_byte) { return (utf8_byte >> 6) != 2; } + +} // namespace detail +} // namespace nvtext diff --git a/cpp/src/text/subword/data_normalizer.cu b/cpp/src/text/subword/data_normalizer.cu index 7a39199011e..4c54409c41a 100644 --- a/cpp/src/text/subword/data_normalizer.cu +++ b/cpp/src/text/subword/data_normalizer.cu @@ -14,6 +14,7 @@ * limitations under the License. */ +#include "text/normalize.cuh" #include "text/subword/detail/data_normalizer.hpp" #include "text/subword/detail/tokenizer_utils.cuh" @@ -38,81 +39,6 @@ namespace nvtext { namespace detail { namespace { -/** - * @brief Bit used to filter out invalid code points. - * - * When normalizing characters to code point values, if this bit is set, - * the code point should be filtered out before returning from the normalizer. - */ -constexpr uint32_t FILTER_BIT = 22; - -/** - * @brief Retrieve new code point from metadata value. - * - * @param metadata Value from the codepoint_metadata table. - * @return The replacement character if appropriate. - */ -__device__ uint32_t get_first_cp(uint32_t metadata) { return metadata & NEW_CP_MASK; } - -/** - * @brief Retrieve token category from the metadata value. - * - * Category values are 0-5: - * 0 - character should be padded - * 1 - pad character if lower-case - * 2 - character should be removed - * 3 - remove character if lower-case - * 4 - whitespace character -- always replace - * 5 - uncategorized - * - * @param metadata Value from the codepoint_metadata table. - * @return Category value. - */ -__device__ uint32_t extract_token_cat(uint32_t metadata) -{ - return (metadata >> TOKEN_CAT_SHIFT) & TOKEN_CAT_MASK; -} - -/** - * @brief Return true if category of metadata value specifies the character should be replaced. - */ -__device__ bool should_remove_cp(uint32_t metadata, bool lower_case) -{ - auto const cat = extract_token_cat(metadata); - return (cat == TOKEN_CAT_REMOVE_CHAR) || (lower_case && (cat == TOKEN_CAT_REMOVE_CHAR_IF_LOWER)); -} - -/** - * @brief Return true if category of metadata value specifies the character should be padded. - */ -__device__ bool should_add_spaces(uint32_t metadata, bool lower_case) -{ - auto const cat = extract_token_cat(metadata); - return (cat == TOKEN_CAT_ADD_SPACE) || (lower_case && (cat == TOKEN_CAT_ADD_SPACE_IF_LOWER)); -} - -/** - * @brief Return true if category of metadata value specifies the character should be replaced. - */ -__device__ bool always_replace(uint32_t metadata) -{ - return extract_token_cat(metadata) == TOKEN_CAT_ALWAYS_REPLACE; -} - -/** - * @brief Returns true if metadata value includes a multi-character transform bit equal to 1. - */ -__device__ bool is_multi_char_transform(uint32_t metadata) -{ - return (metadata >> MULTICHAR_SHIFT) & MULTICHAR_MASK; -} - -/** - * @brief Returns true if the byte passed in could be a valid head byte for - * a utf8 character. That is, not binary `10xxxxxx` - */ -__device__ bool is_head_byte(unsigned char utf8_byte) { return (utf8_byte >> 6) != 2; } - /** * @brief Converts a UTF-8 character into a unicode code point value. * diff --git a/cpp/tests/text/normalize_tests.cpp b/cpp/tests/text/normalize_tests.cpp index 2515cc917fa..530148eb654 100644 --- a/cpp/tests/text/normalize_tests.cpp +++ b/cpp/tests/text/normalize_tests.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -74,6 +74,10 @@ TEST_F(TextNormalizeTest, NormalizeEmptyTest) EXPECT_EQ(results->size(), 0); results = nvtext::normalize_characters(strings_view, false); EXPECT_EQ(results->size(), 0); + + auto normalizer = nvtext::create_character_normalizer(true); + results = nvtext::normalize_characters(strings_view, *normalizer); + EXPECT_EQ(results->size(), 0); } TEST_F(TextNormalizeTest, AllNullStrings) @@ -84,6 +88,10 @@ TEST_F(TextNormalizeTest, AllNullStrings) CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, strings); results = nvtext::normalize_characters(strings_view, false); CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, strings); + + auto normalizer = nvtext::create_character_normalizer(true); + results = nvtext::normalize_characters(strings_view, *normalizer); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, strings); } TEST_F(TextNormalizeTest, SomeNullStrings) @@ -93,27 +101,21 @@ TEST_F(TextNormalizeTest, SomeNullStrings) auto results = nvtext::normalize_characters(strings_view, false); cudf::test::strings_column_wrapper expected({"", " . ", "a"}, {false, true, true}); CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + + auto normalizer = nvtext::create_character_normalizer(true); + results = nvtext::normalize_characters(strings_view, *normalizer); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); } TEST_F(TextNormalizeTest, NormalizeCharacters) { // These include punctuation, accents, whitespace, and CJK characters - std::vector h_strings{"abc£def", - nullptr, - "éè â îô\taeio", - "\tĂĆĖÑ Ü", - "ACEN U", - "P^NP", - "$41.07", - "[a,b]", - "丏丟", - ""}; - auto validity = - thrust::make_transform_iterator(h_strings.begin(), [](auto str) { return str != nullptr; }); - cudf::test::strings_column_wrapper strings(h_strings.begin(), h_strings.end(), validity); - cudf::strings_column_view strings_view(strings); + auto input = cudf::test::strings_column_wrapper( + {"abc£def", "", "éè â îô\taeio", "\tĂĆĖÑ Ü", "ACEN U", "P^NP", "$41.07", "[a,b]", "丏丟", ""}, + {1, 0, 1, 1, 1, 1, 1, 1, 1, 1}); + auto sv = cudf::strings_column_view(input); { - auto results = nvtext::normalize_characters(strings_view, true); + auto results = nvtext::normalize_characters(sv, true); cudf::test::strings_column_wrapper expected({"abc£def", "", "ee a io aeio", @@ -124,11 +126,11 @@ TEST_F(TextNormalizeTest, NormalizeCharacters) " [ a , b ] ", " 丏 丟 ", ""}, - validity); + {1, 0, 1, 1, 1, 1, 1, 1, 1, 1}); CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); } { - auto results = nvtext::normalize_characters(strings_view, false); + auto results = nvtext::normalize_characters(sv, false); cudf::test::strings_column_wrapper expected({"abc£def", "", "éè â îô aeio", @@ -139,11 +141,117 @@ TEST_F(TextNormalizeTest, NormalizeCharacters) " [ a , b ] ", " 丏 丟 ", ""}, - validity); + {1, 0, 1, 1, 1, 1, 1, 1, 1, 1}); CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); } } +TEST_F(TextNormalizeTest, WithNormalizer) +{ + auto long_row = + "this entry is intended to pad out past 256 bytes which is currently the block size"; + // the following include punctuation, accents, whitespace, and CJK characters + auto input = cudf::test::strings_column_wrapper({"abc£def", + "", + "éè â îô\taeio", + "\tĂĆĖÑ Ü", + "ACEN U", + "P^NP", + "$41.07", + "[a,b]", + "丏丟", + "", + long_row, + long_row, + long_row}, + {1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}); + + auto const sv = cudf::strings_column_view(input); + + auto normalizer = nvtext::create_character_normalizer(true); + auto results = nvtext::normalize_characters(sv, *normalizer); + auto expected = cudf::test::strings_column_wrapper({"abc£def", + "", + "ee a io aeio", + " acen u", + "acen u", + "p ^ np", + " $ 41 . 07", + " [ a , b ] ", + " 丏 丟 ", + "", + long_row, + long_row, + long_row}, + {1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + results = nvtext::normalize_characters(sv, *normalizer); // test normalizer re-use + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + + normalizer = nvtext::create_character_normalizer(false); + results = nvtext::normalize_characters(sv, *normalizer); + expected = cudf::test::strings_column_wrapper({"abc£def", + "", + "éè â îô aeio", + " ĂĆĖÑ Ü", + "ACEN U", + "P ^ NP", + " $ 41 . 07", + " [ a , b ] ", + " 丏 丟 ", + "", + long_row, + long_row, + long_row}, + {1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + results = nvtext::normalize_characters(sv, *normalizer); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); +} + +TEST_F(TextNormalizeTest, SpecialTokens) +{ + auto long_row = + "this entry is intended to pad out past 256 bytes which is currently the block size"; + auto input = + cudf::test::strings_column_wrapper({"[BOS]Some strings with [PAD] special[SEP]tokens[EOS]", + "[bos]these should[sep]work too[eos]", + "some[non]tokens[eol]too", + long_row, + long_row, + long_row}); + + auto sv = cudf::strings_column_view(input); + auto special_tokens = cudf::test::strings_column_wrapper({"[BOS]", "[EOS]", "[SEP]", "[PAD]"}); + auto stv = cudf::strings_column_view(special_tokens); + + auto normalizer = nvtext::create_character_normalizer(true, stv); + auto results = nvtext::normalize_characters(sv, *normalizer); + auto expected = cudf::test::strings_column_wrapper( + {" [bos] some strings with [pad] special [sep] tokens [eos] ", + " [bos] these should [sep] work too [eos] ", + "some [ non ] tokens [ eol ] too", + long_row, + long_row, + long_row}); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + results = nvtext::normalize_characters(sv, *normalizer); // and again + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + + normalizer = nvtext::create_character_normalizer(false, stv); + results = nvtext::normalize_characters(sv, *normalizer); + expected = cudf::test::strings_column_wrapper( + {" [BOS] Some strings with [PAD] special [SEP] tokens [EOS] ", + " [ bos ] these should [ sep ] work too [ eos ] ", + "some [ non ] tokens [ eol ] too", + long_row, + long_row, + long_row}); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + results = nvtext::normalize_characters(sv, *normalizer); // and again + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); +} + TEST_F(TextNormalizeTest, NormalizeSlicedColumn) { cudf::test::strings_column_wrapper strings( @@ -151,10 +259,21 @@ TEST_F(TextNormalizeTest, NormalizeSlicedColumn) std::vector sliced = cudf::split(strings, {4}); auto results = nvtext::normalize_characters(cudf::strings_column_view(sliced.front()), true); - cudf::test::strings_column_wrapper expected({"abc£def", "ee a io aeio", "acen u", "p ^ np"}); + auto expected = + cudf::test::strings_column_wrapper({"abc£def", "ee a io aeio", "acen u", "p ^ np"}); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + + results = nvtext::normalize_characters(cudf::strings_column_view(sliced[1]), false); + expected = cudf::test::strings_column_wrapper({" $ 41 . 07", " [ a , b ] ", " 丏 丟 "}); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + + auto normalizer = nvtext::create_character_normalizer(true); + results = nvtext::normalize_characters(cudf::strings_column_view(sliced.front()), *normalizer); + expected = cudf::test::strings_column_wrapper({"abc£def", "ee a io aeio", "acen u", "p ^ np"}); CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); - results = nvtext::normalize_characters(cudf::strings_column_view(sliced[1]), false); - cudf::test::strings_column_wrapper expected2({" $ 41 . 07", " [ a , b ] ", " 丏 丟 "}); - CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected2); + normalizer = nvtext::create_character_normalizer(false); + results = nvtext::normalize_characters(cudf::strings_column_view(sliced[1]), *normalizer); + expected = cudf::test::strings_column_wrapper({" $ 41 . 07", " [ a , b ] ", " 丏 丟 "}); + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); } diff --git a/python/cudf/cudf/core/character_normalizer.py b/python/cudf/cudf/core/character_normalizer.py new file mode 100644 index 00000000000..1240c0e1eb7 --- /dev/null +++ b/python/cudf/cudf/core/character_normalizer.py @@ -0,0 +1,46 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. + +from __future__ import annotations + +import pylibcudf as plc + +import cudf + + +class CharacterNormalizer: + """ + A normalizer object used to normalize input text. + + Parameters + ---------- + do_lower : bool + If True, the normalizer should also lower-case + while normalizing. + special_tokens : cudf.Series + Series of special tokens. + """ + + def __init__( + self, + do_lower: bool, + special_tokens: cudf.Series = cudf.Series([], dtype="object"), + ) -> None: + self.normalizer = plc.nvtext.normalize.CharacterNormalizer( + do_lower, special_tokens._column.to_pylibcudf(mode="read") + ) + + def normalize(self, text: cudf.Series) -> cudf.Series: + """ + Parameters + ---------- + text : cudf.Series + The strings to be normalized. + + Returns + ------- + cudf.Series + Normalized strings + """ + result = text._column.normalize_characters(self.normalizer) + + return cudf.Series._from_column(result) diff --git a/python/cudf/cudf/core/column/string.py b/python/cudf/cudf/core/column/string.py index 04a72017c33..c0ad33ec7d6 100644 --- a/python/cudf/cudf/core/column/string.py +++ b/python/cudf/cudf/core/column/string.py @@ -4679,8 +4679,10 @@ def normalize_characters(self, do_lower: bool = True) -> SeriesOrIndex: r""" Normalizes strings characters for tokenizing. - This uses the normalizer that is built into the - subword_tokenize function which includes: + .. deprecated:: 25.04 + Use `CharacterNormalizer` instead. + + The normalizer function includes: - adding padding around punctuation (unicode category starts with "P") as well as certain ASCII symbols like "^" and "$" @@ -4720,8 +4722,13 @@ def normalize_characters(self, do_lower: bool = True) -> SeriesOrIndex: 2 $ 99 dtype: object """ + warnings.warn( + "normalize_characters is deprecated and will be removed in a future " + "version. Use CharacterNormalizer instead.", + FutureWarning, + ) return self._return_or_inplace( - self._column.normalize_characters(do_lower) + self._column.characters_normalize(do_lower) ) def tokenize(self, delimiter: str = " ") -> SeriesOrIndex: @@ -6256,14 +6263,25 @@ def normalize_spaces(self) -> Self: ) @acquire_spill_lock() - def normalize_characters(self, do_lower: bool = True) -> Self: + def characters_normalize(self, do_lower: bool = True) -> Self: return ColumnBase.from_pylibcudf( # type: ignore[return-value] - plc.nvtext.normalize.normalize_characters( + plc.nvtext.normalize.characters_normalize( self.to_pylibcudf(mode="read"), do_lower, ) ) + @acquire_spill_lock() + def normalize_characters( + self, normalizer: plc.nvtext.normalize.CharacterNormalizer + ) -> Self: + return ColumnBase.from_pylibcudf( # type: ignore[return-value] + plc.nvtext.normalize.normalize_characters( + self.to_pylibcudf(mode="read"), + normalizer, + ) + ) + @acquire_spill_lock() def replace_tokens( self, targets: Self, replacements: Self, delimiter: plc.Scalar diff --git a/python/cudf/cudf/tests/text/test_text_methods.py b/python/cudf/cudf/tests/text/test_text_methods.py index 86e1e46c1a2..dc45827d2e8 100644 --- a/python/cudf/cudf/tests/text/test_text_methods.py +++ b/python/cudf/cudf/tests/text/test_text_methods.py @@ -8,6 +8,7 @@ import cudf from cudf.core.byte_pair_encoding import BytePairEncoder +from cudf.core.character_normalizer import CharacterNormalizer from cudf.core.tokenize_vocabulary import TokenizeVocabulary from cudf.testing import assert_eq @@ -251,7 +252,8 @@ def test_normalize_characters(): ] ) - actual = strings.str.normalize_characters() + normalizer_lower = CharacterNormalizer(True) + actual = normalizer_lower.normalize(strings.str) assert type(expected) is type(actual) assert_eq(expected, actual) @@ -265,7 +267,9 @@ def test_normalize_characters(): "Stock ^ $ 1", ] ) - actual = strings.str.normalize_characters(do_lower=False) + + normalizer = CharacterNormalizer(False) + actual = normalizer.normalize(strings.str) assert type(expected) is type(actual) assert_eq(expected, actual) diff --git a/python/pylibcudf/pylibcudf/libcudf/nvtext/normalize.pxd b/python/pylibcudf/pylibcudf/libcudf/nvtext/normalize.pxd index f8b082c8429..2cf2bfb8ac9 100644 --- a/python/pylibcudf/pylibcudf/libcudf/nvtext/normalize.pxd +++ b/python/pylibcudf/pylibcudf/libcudf/nvtext/normalize.pxd @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from libcpp cimport bool from libcpp.memory cimport unique_ptr from pylibcudf.exception_handler cimport libcudf_exception_handler @@ -16,3 +16,16 @@ cdef extern from "nvtext/normalize.hpp" namespace "nvtext" nogil: const column_view & strings, bool do_lower_case ) except +libcudf_exception_handler + + cdef struct character_normalizer "nvtext::character_normalizer": + pass + + cdef unique_ptr[character_normalizer] create_character_normalizer( + bool do_lower_case, + const column_view & strings + ) except +libcudf_exception_handler + + cdef unique_ptr[column] normalize_characters( + const column_view & strings, + const character_normalizer & normalizer + ) except +libcudf_exception_handler diff --git a/python/pylibcudf/pylibcudf/nvtext/normalize.pxd b/python/pylibcudf/pylibcudf/nvtext/normalize.pxd index 90676145afa..e6688e19762 100644 --- a/python/pylibcudf/pylibcudf/nvtext/normalize.pxd +++ b/python/pylibcudf/pylibcudf/nvtext/normalize.pxd @@ -1,9 +1,18 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from libcpp cimport bool +from libcpp.memory cimport unique_ptr from pylibcudf.column cimport Column +from pylibcudf.libcudf.nvtext.normalize cimport character_normalizer +cdef class CharacterNormalizer: + cdef unique_ptr[character_normalizer] c_obj cpdef Column normalize_spaces(Column input) -cpdef Column normalize_characters(Column input, bool do_lower_case) +cpdef Column characters_normalize(Column input, bool do_lower_case) + +cpdef Column normalize_characters( + Column input, + CharacterNormalizer normalizer +) diff --git a/python/pylibcudf/pylibcudf/nvtext/normalize.pyi b/python/pylibcudf/pylibcudf/nvtext/normalize.pyi index 1d90a5a8960..d722ef6c79e 100644 --- a/python/pylibcudf/pylibcudf/nvtext/normalize.pyi +++ b/python/pylibcudf/pylibcudf/nvtext/normalize.pyi @@ -1,6 +1,12 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from pylibcudf.column import Column +class CharacterNormalizer: + def __init__(self, do_lower_case: bool, special_tokens: Column): ... + def normalize_spaces(input: Column) -> Column: ... -def normalize_characters(input: Column, do_lower_case: bool) -> Column: ... +def characters_normalize(input: Column, do_lower_case: bool) -> Column: ... +def normalize_characters( + input: Column, normalizer: CharacterNormalizer +) -> Column: ... diff --git a/python/pylibcudf/pylibcudf/nvtext/normalize.pyx b/python/pylibcudf/pylibcudf/nvtext/normalize.pyx index b259ccaefa6..6a18c205841 100644 --- a/python/pylibcudf/pylibcudf/nvtext/normalize.pyx +++ b/python/pylibcudf/pylibcudf/nvtext/normalize.pyx @@ -1,16 +1,37 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. +from cython.operator cimport dereference from libcpp cimport bool from libcpp.memory cimport unique_ptr from libcpp.utility cimport move from pylibcudf.column cimport Column from pylibcudf.libcudf.column.column cimport column -from pylibcudf.libcudf.nvtext.normalize cimport ( - normalize_characters as cpp_normalize_characters, - normalize_spaces as cpp_normalize_spaces, -) +from pylibcudf.libcudf.column.column_view cimport column_view +from pylibcudf.libcudf.nvtext cimport normalize as cpp_normalize -__all__ = ["normalize_characters", "normalize_spaces"] +__all__ = [ + "CharacterNormalizer" + "normalize_characters", + "normalize_spaces", + "characters_normalize" +] + +cdef class CharacterNormalizer: + """The normalizer object to be used with ``normalize_characters``. + + For details, see :cpp:class:`cudf::nvtext::character_normalizer`. + """ + def __cinit__(self, bool do_lower_case, Column tokens): + cdef column_view c_tokens = tokens.view() + with nogil: + self.c_obj = move( + cpp_normalize.create_character_normalizer( + do_lower_case, + c_tokens + ) + ) + + __hash__ = None cpdef Column normalize_spaces(Column input): """ @@ -32,12 +53,12 @@ cpdef Column normalize_spaces(Column input): cdef unique_ptr[column] c_result with nogil: - c_result = cpp_normalize_spaces(input.view()) + c_result = cpp_normalize.normalize_spaces(input.view()) return Column.from_libcudf(move(c_result)) -cpdef Column normalize_characters(Column input, bool do_lower_case): +cpdef Column characters_normalize(Column input, bool do_lower_case): """ Normalizes strings characters for tokenizing. @@ -60,6 +81,38 @@ cpdef Column normalize_characters(Column input, bool do_lower_case): cdef unique_ptr[column] c_result with nogil: - c_result = cpp_normalize_characters(input.view(), do_lower_case) + c_result = cpp_normalize.normalize_characters( + input.view(), + do_lower_case + ) + + return Column.from_libcudf(move(c_result)) + + +cpdef Column normalize_characters(Column input, CharacterNormalizer normalizer): + """ + Normalizes strings characters for tokenizing. + + For details, see :cpp:func:`normalize_characters` + + Parameters + ---------- + input : Column + Input strings + normalizer : CharacterNormalizer + Normalizer object used for modifying the input column text + + Returns + ------- + Column + Normalized strings column + """ + cdef unique_ptr[column] c_result + + with nogil: + c_result = cpp_normalize.normalize_characters( + input.view(), + dereference(normalizer.c_obj.get()) + ) return Column.from_libcudf(move(c_result)) diff --git a/python/pylibcudf/pylibcudf/tests/test_nvtext_normalize.py b/python/pylibcudf/pylibcudf/tests/test_nvtext_normalize.py index 25b6d1389ec..47bbb191be6 100644 --- a/python/pylibcudf/pylibcudf/tests/test_nvtext_normalize.py +++ b/python/pylibcudf/pylibcudf/tests/test_nvtext_normalize.py @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. import pyarrow as pa import pytest @@ -15,7 +15,7 @@ def norm_spaces_input_data(): @pytest.fixture(scope="module") def norm_chars_input_data(): - arr = ["éâîô\teaio", "ĂĆĖÑÜ", "ACENU", "$24.08", "[a,bb]"] + arr = ["éâîô\teaio", "ĂĆĖÑÜ", "ACENU", "$24.08", "[a,bb]", "[pad]"] return pa.array(arr) @@ -29,15 +29,98 @@ def test_normalize_spaces(norm_spaces_input_data): @pytest.mark.parametrize("do_lower", [True, False]) def test_normalize_characters(norm_chars_input_data, do_lower): - result = plc.nvtext.normalize.normalize_characters( + result = plc.nvtext.normalize.characters_normalize( plc.interop.from_arrow(norm_chars_input_data), do_lower, ) - expected = pa.array( - ["eaio eaio", "acenu", "acenu", " $ 24 . 08", " [ a , bb ] "] + if do_lower: + expected = pa.array( + [ + "eaio eaio", + "acenu", + "acenu", + " $ 24 . 08", + " [ a , bb ] ", + " [ pad ] ", + ] + ) + else: + expected = pa.array( + [ + "éâîô eaio", + "ĂĆĖÑÜ", + "ACENU", + " $ 24 . 08", + " [ a , bb ] ", + " [ pad ] ", + ] + ) + assert_column_eq(result, expected) + + +@pytest.mark.parametrize("do_lower", [True, False]) +def test_normalizer(norm_chars_input_data, do_lower): + result = plc.nvtext.normalize.normalize_characters( + plc.interop.from_arrow(norm_chars_input_data), + plc.nvtext.normalize.CharacterNormalizer( + do_lower, + plc.column_factories.make_empty_column(plc.types.TypeId.STRING), + ), + ) + if do_lower: + expected = pa.array( + [ + "eaio eaio", + "acenu", + "acenu", + " $ 24 . 08", + " [ a , bb ] ", + " [ pad ] ", + ] + ) + else: + expected = pa.array( + [ + "éâîô eaio", + "ĂĆĖÑÜ", + "ACENU", + " $ 24 . 08", + " [ a , bb ] ", + " [ pad ] ", + ] + ) + assert_column_eq(result, expected) + + +@pytest.mark.parametrize("do_lower", [True, False]) +def test_normalizer_with_special_tokens(norm_chars_input_data, do_lower): + special_tokens = pa.array(["[pad]"]) + result = plc.nvtext.normalize.normalize_characters( + plc.interop.from_arrow(norm_chars_input_data), + plc.nvtext.normalize.CharacterNormalizer( + do_lower, plc.interop.from_arrow(special_tokens) + ), ) - if not do_lower: + if do_lower: + expected = pa.array( + [ + "eaio eaio", + "acenu", + "acenu", + " $ 24 . 08", + " [ a , bb ] ", + " [pad] ", + ] + ) + else: expected = pa.array( - ["éâîô eaio", "ĂĆĖÑÜ", "ACENU", " $ 24 . 08", " [ a , bb ] "] + [ + "éâîô eaio", + "ĂĆĖÑÜ", + "ACENU", + " $ 24 . 08", + " [ a , bb ] ", + " [pad] ", + ] ) assert_column_eq(result, expected) From 5eb552754020bed652f3f278a6b5cc494eeb9bce Mon Sep 17 00:00:00 2001 From: Vyas Ramasubramani Date: Tue, 25 Feb 2025 16:15:40 -0800 Subject: [PATCH 090/129] Remove unused var (#18096) The `cython_lib_dir` was removed as part of the switch to publishing a libcudf wheel. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18096 --- python/cudf/CMakeLists.txt | 4 ---- python/cudf_kafka/CMakeLists.txt | 4 ---- python/pylibcudf/CMakeLists.txt | 4 ---- 3 files changed, 12 deletions(-) diff --git a/python/cudf/CMakeLists.txt b/python/cudf/CMakeLists.txt index 2a17bc5dbb7..090e475471d 100644 --- a/python/cudf/CMakeLists.txt +++ b/python/cudf/CMakeLists.txt @@ -37,7 +37,3 @@ rapids_cython_init() add_subdirectory(cudf/_lib) add_subdirectory(udf_cpp) - -if(DEFINED cython_lib_dir) - rapids_cython_add_rpath_entries(TARGET cudf PATHS "${cython_lib_dir}") -endif() diff --git a/python/cudf_kafka/CMakeLists.txt b/python/cudf_kafka/CMakeLists.txt index 3e12eb6aa41..13b859bc33b 100644 --- a/python/cudf_kafka/CMakeLists.txt +++ b/python/cudf_kafka/CMakeLists.txt @@ -35,7 +35,3 @@ include(rapids-cython-core) rapids_cython_init() add_subdirectory(cudf_kafka/_lib) - -if(DEFINED cython_lib_dir) - rapids_cython_add_rpath_entries(TARGET cudf_kafka PATHS "${cython_lib_dir}") -endif() diff --git a/python/pylibcudf/CMakeLists.txt b/python/pylibcudf/CMakeLists.txt index fe6e73a3f14..153570a4a7e 100644 --- a/python/pylibcudf/CMakeLists.txt +++ b/python/pylibcudf/CMakeLists.txt @@ -37,7 +37,3 @@ include(rapids-cython-core) rapids_cython_init() add_subdirectory(pylibcudf) - -if(DEFINED cython_lib_dir) - rapids_cython_add_rpath_entries(TARGET cudf PATHS "${cython_lib_dir}") -endif() From d8b3d801ec4830102242db1fa60a88e1a0bb7299 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Tue, 25 Feb 2025 16:58:14 -0800 Subject: [PATCH 091/129] Fix scatter_by_map with spilling enabled (#18095) closes https://github.com/rapidsai/cudf/issues/18088 Before the old Cython bindings of `columns_split` spill locked the conversion from libcudf to a cudf Python column. When I replaced these bindings, this spill locking was removed during the refactor. I'm spot checking that other APIs are not affected. If so I can open PRs for those Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Matthew Murray (https://github.com/Matt711) URL: https://github.com/rapidsai/cudf/pull/18095 --- python/cudf/cudf/core/indexed_frame.py | 6 +++++- python/cudf/cudf/tests/test_spilling.py | 11 ++++++++++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index 9c48b31a309..211d161696e 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -3308,9 +3308,13 @@ def _split(self, splits, keep_index: bool = True) -> list[Self]: splits, ) + @acquire_spill_lock() + def split_from_pylibcudf(split: list[plc.Column]) -> list[ColumnBase]: + return [ColumnBase.from_pylibcudf(col) for col in split] + return [ self._from_columns_like_self( - [ColumnBase.from_pylibcudf(col) for col in split], + split_from_pylibcudf(split), self._column_names, self.index.names if keep_index else None, ) diff --git a/python/cudf/cudf/tests/test_spilling.py b/python/cudf/cudf/tests/test_spilling.py index 13d98e43ddc..08226dd7f6d 100644 --- a/python/cudf/cudf/tests/test_spilling.py +++ b/python/cudf/cudf/tests/test_spilling.py @@ -1,4 +1,4 @@ -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. from __future__ import annotations import contextlib @@ -784,3 +784,12 @@ def test_spilling_and_copy_on_write(manager: SpillManager): assert not a.is_spilled assert a.owner.exposed assert not b.owner.exposed + + +def test_scatter_by_map(): + data = range(10) + with cudf.option_context("spill", True): + df = cudf.DataFrame(data) + result = df.scatter_by_map(data) + for i, res in zip(data, result): + assert_eq(res, cudf.DataFrame([i], index=[i])) From 46b9799ea55b899e08f6b758ec90e9742a72d159 Mon Sep 17 00:00:00 2001 From: "Richard (Rick) Zamora" Date: Tue, 25 Feb 2025 19:14:57 -0600 Subject: [PATCH 092/129] Fix `test_scan_csv_multi` cudf-polars test (#18064) The current implementation of `test_scan_csv_multi` does not work if the compute task is run on distinct worker processes (because it changes directory in lieu of using a proper file path). Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/18064 --- python/cudf_polars/tests/test_scan.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/python/cudf_polars/tests/test_scan.py b/python/cudf_polars/tests/test_scan.py index 9c58a24c065..8ff0db084b1 100644 --- a/python/cudf_polars/tests/test_scan.py +++ b/python/cudf_polars/tests/test_scan.py @@ -1,9 +1,7 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 from __future__ import annotations -import os - import pytest import polars as pl @@ -203,8 +201,11 @@ def test_scan_csv_multi(tmp_path, filename, glob, nrows_skiprows): f.write("""foo,bar,baz\n1,2,3\n3,4,5""") with (tmp_path / "test*.csv").open("w") as f: f.write("""foo,bar,baz\n1,2,3\n3,4,5""") - os.chdir(tmp_path) - q = pl.scan_csv(filename, glob=glob, n_rows=n_rows, skip_rows=skiprows) + if isinstance(filename, list): + source = [tmp_path / fn for fn in filename] + else: + source = tmp_path / filename + q = pl.scan_csv(source, glob=glob, n_rows=n_rows, skip_rows=skiprows) assert_gpu_result_equal(q) From 72d5792c79f11c90f43c6991dd54e082b3c0ad98 Mon Sep 17 00:00:00 2001 From: "Richard (Rick) Zamora" Date: Wed, 26 Feb 2025 08:14:26 -0600 Subject: [PATCH 093/129] Relax inconsistent schema handling in `dask_cudf.read_parquet` (#17554) Addresses an issue raised offline by @praateekmahajan Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) - Mads R. B. Kristensen (https://github.com/madsbk) URL: https://github.com/rapidsai/cudf/pull/17554 --- .../dask_cudf/dask_cudf/_legacy/io/parquet.py | 18 +++---- .../dask_cudf/io/tests/test_parquet.py | 48 ++++++++++++++++++- 2 files changed, 53 insertions(+), 13 deletions(-) diff --git a/python/dask_cudf/dask_cudf/_legacy/io/parquet.py b/python/dask_cudf/dask_cudf/_legacy/io/parquet.py index c0792663c7e..c0b9d71653c 100644 --- a/python/dask_cudf/dask_cudf/_legacy/io/parquet.py +++ b/python/dask_cudf/dask_cudf/_legacy/io/parquet.py @@ -434,18 +434,12 @@ def set_object_dtypes_from_pa_schema(df, schema): # pyarrow schema. if schema: for col_name, col in df._data.items(): - if col_name is None: - # Pyarrow cannot handle `None` as a field name. - # However, this should be a simple range index that - # we can ignore anyway - continue - typ = cudf_dtype_from_pa_type(schema.field(col_name).type) - if ( - col_name in schema.names - and not isinstance(typ, (cudf.ListDtype, cudf.StructDtype)) - and isinstance(col, cudf.core.column.StringColumn) - ): - df._data[col_name] = col.astype(typ) + if col_name in schema.names: + typ = cudf_dtype_from_pa_type(schema.field(col_name).type) + if not isinstance( + typ, (cudf.ListDtype, cudf.StructDtype) + ) and isinstance(col, cudf.core.column.StringColumn): + df._data[col_name] = col.astype(typ) to_parquet = dd.to_parquet diff --git a/python/dask_cudf/dask_cudf/io/tests/test_parquet.py b/python/dask_cudf/dask_cudf/io/tests/test_parquet.py index 9f7031f4d2a..3a88668e6d2 100644 --- a/python/dask_cudf/dask_cudf/io/tests/test_parquet.py +++ b/python/dask_cudf/dask_cudf/io/tests/test_parquet.py @@ -6,6 +6,7 @@ import numpy as np import pandas as pd +import pyarrow as pa import pytest import dask @@ -486,6 +487,52 @@ def test_create_metadata_file_inconsistent_schema(tmpdir): dd.assert_eq(ddf1.compute(), ddf2.compute()) +@pytest.mark.parametrize("specify_schema", [True, False]) +def test_read_inconsistent_schema(tmpdir, specify_schema): + if specify_schema: + # If we specify the expected schema, + # we also need to specify the partitioning. + kwargs = { + "dataset": { + "schema": pa.schema( + [ + ("id", pa.int64()), + ("text", pa.string()), + ("meta1", pa.struct([("field1", pa.string())])), + ] + ), + "partitioning": None, + }, + } + else: + kwargs = {} + + records = [ + {"id": 123, "text": "foo"}, + { + "text": "bar", + "meta1": [{"field1": "cat"}], + "id": 456, + }, + ] + columns = ["text", "id"] + pd.DataFrame(records[:1]).to_parquet(tmpdir / "part.0.parquet") + pd.DataFrame(records[1:]).to_parquet(tmpdir / "part.1.parquet") + # Check that cuDF and Dask cuDF match + dd.assert_eq( + cudf.read_parquet( + tmpdir, columns=columns, allow_mismatched_pq_schemas=True + ), + dask_cudf.read_parquet(tmpdir, columns=columns, **kwargs), + check_index=False, + ) + # Check that "pandas" and "cudf" backends match + dd.assert_eq( + dd.read_parquet(tmpdir, columns=columns), + dask_cudf.read_parquet(tmpdir, columns=columns, **kwargs), + ) + + @pytest.mark.parametrize( "data", [ @@ -526,7 +573,6 @@ def test_cudf_list_struct_write(tmpdir): def test_null_partition(tmpdir): - import pyarrow as pa from pyarrow.dataset import HivePartitioning ids = pd.Series([0, 1, None], dtype="Int64") From e5d866bc68c4762ebd6e3e888e4abeaf4ccd9302 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Wed, 26 Feb 2025 07:01:27 -0800 Subject: [PATCH 094/129] Short circuit Index.equal if compared Index isn't same type (#18067) closes https://github.com/rapidsai/cudf/issues/8689 Before, comparing two different Index subclasses would execute a GPU kernel when we know they wouldn't be equal (e.g. DatetimeIndex equals RangeIndex). This PR add a short circuit clause to check that we are comparing the same subclasses. Also ensures we don't return a `np.bool_` object from this result. Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/18067 --- python/cudf/cudf/core/column/column.py | 2 +- python/cudf/cudf/core/index.py | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index 06dc4058115..67a0aa7a781 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -713,7 +713,7 @@ def all(self, skipna: bool = True) -> bool: # is empty. if self.null_count == self.size: return True - return self.reduce("all") + return bool(self.reduce("all")) def any(self, skipna: bool = True) -> bool: # Early exit for fast cases. diff --git a/python/cudf/cudf/core/index.py b/python/cudf/cudf/core/index.py index 1730a692dc1..f4e5f6e96ae 100644 --- a/python/cudf/cudf/core/index.py +++ b/python/cudf/cudf/core/index.py @@ -1286,6 +1286,15 @@ def equals(self, other) -> bool: elif other_is_categorical and not self_is_categorical: self = self.astype(other.dtype) check_dtypes = True + elif ( + not self_is_categorical + and not other_is_categorical + and not isinstance(other, RangeIndex) + and not isinstance(self, type(other)) + ): + # Can compare Index to CategoricalIndex or RangeIndex + # Other comparisons are invalid + return False try: return self._column.equals( From 1a8d6368405fac3c5e55592fef2d9259081b045c Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Wed, 26 Feb 2025 11:00:27 -0800 Subject: [PATCH 095/129] Enforce deprecation of dtype parameter in sum/product (#18070) xref https://github.com/rapidsai/cudf/pull/16313 Deprecated in 24.08 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18070 --- python/cudf/cudf/core/column/column.py | 20 +++++--------------- python/cudf/cudf/core/column/timedelta.py | 3 +-- python/cudf/cudf/core/indexed_frame.py | 8 -------- python/cudf/cudf/tests/test_reductions.py | 8 -------- 4 files changed, 6 insertions(+), 33 deletions(-) diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index 67a0aa7a781..b57d1f03981 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -2,7 +2,6 @@ from __future__ import annotations -import warnings from collections import abc from collections.abc import MutableSequence, Sequence from functools import cached_property @@ -1946,8 +1945,7 @@ def _reduce( skipna=skipna, min_count=min_count ) if isinstance(preprocessed, ColumnBase): - dtype = kwargs.pop("dtype", None) - return preprocessed.reduce(op, dtype, **kwargs) + return preprocessed.reduce(op, **kwargs) return preprocessed def _can_return_nan(self, skipna: bool | None = None) -> bool: @@ -2110,16 +2108,8 @@ def scan(self, scan_op: str, inclusive: bool, **kwargs) -> Self: ) ) - def reduce(self, reduction_op: str, dtype=None, **kwargs) -> ScalarLike: - if dtype is not None: - warnings.warn( - "dtype is deprecated and will be remove in a future release. " - "Cast the result (e.g. .astype) after the operation instead.", - FutureWarning, - ) - col_dtype = dtype - else: - col_dtype = self._reduction_result_dtype(reduction_op) + def reduce(self, reduction_op: str, **kwargs) -> ScalarLike: + col_dtype = self._reduction_result_dtype(reduction_op) # check empty case if len(self) <= self.null_count: @@ -2148,7 +2138,7 @@ def reduce(self, reduction_op: str, dtype=None, **kwargs) -> ScalarLike: }: scale = -plc_scalar.type().scale() # https://docs.microsoft.com/en-us/sql/t-sql/data-types/precision-scale-and-length-transact-sql - p = col_dtype.precision + p = col_dtype.precision # type: ignore[union-attr] nrows = len(self) if reduction_op in {"min", "max"}: new_p = p @@ -2162,7 +2152,7 @@ def reduce(self, reduction_op: str, dtype=None, **kwargs) -> ScalarLike: raise NotImplementedError( f"{reduction_op} not implemented for decimal types." ) - precision = max(min(new_p, col_dtype.MAX_PRECISION), 0) + precision = max(min(new_p, col_dtype.MAX_PRECISION), 0) # type: ignore[union-attr] new_dtype = type(col_dtype)(precision, scale) result_col = result_col.astype(new_dtype) elif isinstance(col_dtype, IntervalDtype): diff --git a/python/cudf/cudf/core/column/timedelta.py b/python/cudf/cudf/core/column/timedelta.py index 1cbbac0f8cc..8b0ef9f0cc8 100644 --- a/python/cudf/cudf/core/column/timedelta.py +++ b/python/cudf/cudf/core/column/timedelta.py @@ -452,14 +452,13 @@ def sum( self, skipna: bool | None = None, min_count: int = 0, - dtype: Dtype | None = None, ) -> pd.Timedelta: return pd.Timedelta( # Since sum isn't overridden in Numerical[Base]Column, mypy only # sees the signature from Reducible (which doesn't have the extra # parameters from ColumnBase._reduce) so we have to ignore this. self.astype(np.dtype(np.int64)).sum( # type: ignore - skipna=skipna, min_count=min_count, dtype=dtype + skipna=skipna, min_count=min_count ), unit=self.time_unit, ).as_unit(self.time_unit) diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index 211d161696e..9d426ad6bf7 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -1328,7 +1328,6 @@ def sum( self, axis=no_default, skipna=True, - dtype=None, numeric_only=False, min_count=0, **kwargs, @@ -1342,8 +1341,6 @@ def sum( Axis for the function to be applied on. skipna: bool, default True Exclude NA/null values when computing the result. - dtype: data type - Data type to cast the result to. numeric_only : bool, default False If True, includes only float, int, boolean columns. If False, will raise error in-case there are @@ -1373,7 +1370,6 @@ def sum( "sum", axis=axis, skipna=skipna, - dtype=dtype, numeric_only=numeric_only, min_count=min_count, **kwargs, @@ -1384,7 +1380,6 @@ def product( self, axis=no_default, skipna=True, - dtype=None, numeric_only=False, min_count=0, **kwargs, @@ -1398,8 +1393,6 @@ def product( Axis for the function to be applied on. skipna: bool, default True Exclude NA/null values when computing the result. - dtype: data type - Data type to cast the result to. numeric_only : bool, default False If True, includes only float, int, boolean columns. If False, will raise error in-case there are @@ -1432,7 +1425,6 @@ def product( "prod" if axis in {1, "columns"} else "product", axis=axis, skipna=skipna, - dtype=dtype, numeric_only=numeric_only, min_count=min_count, **kwargs, diff --git a/python/cudf/cudf/tests/test_reductions.py b/python/cudf/cudf/tests/test_reductions.py index 80ffce9e8be..75e38b9246a 100644 --- a/python/cudf/cudf/tests/test_reductions.py +++ b/python/cudf/cudf/tests/test_reductions.py @@ -512,14 +512,6 @@ def test_reduction_column_multiindex(): assert_eq(result, expected) -@pytest.mark.parametrize("op", ["sum", "product"]) -def test_dtype_deprecated(op): - ser = cudf.Series(range(5)) - with pytest.warns(FutureWarning): - result = getattr(ser, op)(dtype=np.dtype(np.int8)) - assert isinstance(result, np.int8) - - @pytest.mark.parametrize( "columns", [pd.RangeIndex(2), pd.Index([0, 1], dtype="int8")] ) From 54e740af7a08b99cca84f4f668886031a2c36e71 Mon Sep 17 00:00:00 2001 From: MithunR Date: Wed, 26 Feb 2025 11:12:10 -0800 Subject: [PATCH 096/129] Remove static column vectors from window function tests. (#18099) Fixes #18079. This commit fixes the failures reported in #18079, where the use of static column vector objects in the tests causes the use of a CUDA runtime context before it's been initialized, causing the tests to fail with: ``` parallel_for failed: cudaErrorInvalidResourceHandle: invalid resource handle ``` The solution is to switch the static column vectors to runtime, as a member of the test utility class `rolling_runner`. Authors: - MithunR (https://github.com/mythrocks) Approvers: - Bradley Dice (https://github.com/bdice) - David Wendt (https://github.com/davidwendt) URL: https://github.com/rapidsai/cudf/pull/18099 --- cpp/tests/rolling/offset_row_window_test.cpp | 28 +++++++++++--------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/cpp/tests/rolling/offset_row_window_test.cpp b/cpp/tests/rolling/offset_row_window_test.cpp index dcaa47e722b..4477ca388df 100644 --- a/cpp/tests/rolling/offset_row_window_test.cpp +++ b/cpp/tests/rolling/offset_row_window_test.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -43,18 +43,21 @@ auto constexpr null = int32_t{0}; // NULL representation for int32_t; auto no_nulls_list() { return nulls_at({}); } struct OffsetRowWindowTest : public cudf::test::BaseFixture { - static ints_column const _keys; // {0, 0, 0, 0, 0, 0, 1, 1, 1, 1}; - static ints_column const _values; // {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}; - struct rolling_runner { cudf::window_bounds _preceding, _following; cudf::size_type _min_periods; bool _grouped = true; + ints_column const _keys; // {0, 0, 0, 0, 0, 0, 1, 1, 1, 1}; + ints_column const _values; // {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}; rolling_runner(cudf::window_bounds const& preceding, cudf::window_bounds const& following, cudf::size_type min_periods_ = 1) - : _preceding{preceding}, _following{following}, _min_periods{min_periods_} + : _preceding{preceding}, + _following{following}, + _min_periods{min_periods_}, + _keys{0, 0, 0, 0, 0, 0, 1, 1, 1, 1}, + _values{0, 1, 2, 3, 4, 5, 6, 7, 8, 9} { } @@ -80,9 +83,6 @@ struct OffsetRowWindowTest : public cudf::test::BaseFixture { }; }; -ints_column const OffsetRowWindowTest::_keys{0, 0, 0, 0, 0, 0, 1, 1, 1, 1}; -ints_column const OffsetRowWindowTest::_values{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}; - auto const AGG_COUNT_NON_NULL = cudf::make_count_aggregation(cudf::null_policy::EXCLUDE); auto const AGG_COUNT_ALL = @@ -96,7 +96,8 @@ TEST_F(OffsetRowWindowTest, OffsetRowWindow_Grouped_3_to_Minus_1) { auto const preceding = cudf::window_bounds::get(3); auto const following = cudf::window_bounds::get(-1); - auto run_rolling = rolling_runner{preceding, following}.min_periods(1).grouped(true); + auto run_rolling = rolling_runner{preceding, following}; + run_rolling.min_periods(1).grouped(true); CUDF_TEST_EXPECT_COLUMNS_EQUAL(*run_rolling(*AGG_COUNT_NON_NULL), ints_column{{0, 1, 2, 2, 2, 2, 0, 1, 2, 2}, nulls_at({0, 6})}); @@ -136,7 +137,8 @@ TEST_F(OffsetRowWindowTest, OffsetRowWindow_Ungrouped_3_to_Minus_1) { auto const preceding = cudf::window_bounds::get(3); auto const following = cudf::window_bounds::get(-1); - auto run_rolling = rolling_runner{preceding, following}.min_periods(1).grouped(false); + auto run_rolling = rolling_runner{preceding, following}; + run_rolling.min_periods(1).grouped(false); CUDF_TEST_EXPECT_COLUMNS_EQUAL(*run_rolling(*AGG_COUNT_NON_NULL), ints_column{{0, 1, 2, 2, 2, 2, 2, 2, 2, 2}, nulls_at({0})}); @@ -176,7 +178,8 @@ TEST_F(OffsetRowWindowTest, OffsetRowWindow_Grouped_0_to_2) { auto const preceding = cudf::window_bounds::get(0); auto const following = cudf::window_bounds::get(2); - auto run_rolling = rolling_runner{preceding, following}.min_periods(1).grouped(true); + auto run_rolling = rolling_runner{preceding, following}; + run_rolling.min_periods(1).grouped(true); CUDF_TEST_EXPECT_COLUMNS_EQUAL( *run_rolling(*AGG_COUNT_NON_NULL), @@ -219,7 +222,8 @@ TEST_F(OffsetRowWindowTest, OffsetRowWindow_Ungrouped_0_to_2) { auto const preceding = cudf::window_bounds::get(0); auto const following = cudf::window_bounds::get(2); - auto run_rolling = rolling_runner{preceding, following}.min_periods(1).grouped(false); + auto run_rolling = rolling_runner{preceding, following}; + run_rolling.min_periods(1).grouped(false); CUDF_TEST_EXPECT_COLUMNS_EQUAL(*run_rolling(*AGG_COUNT_NON_NULL), ints_column{{2, 2, 2, 2, 2, 2, 2, 2, 1, null}, nulls_at({9})}); From 79d0b75a5327f72cdc14297885257a8979bdf0f2 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Wed, 26 Feb 2025 12:01:22 -0800 Subject: [PATCH 097/129] Align StringColumn constructor with ColumnBase base class (#18086) With this PR, the constructors of all subclasses of `ColumnBase` are aligned. This will allow us to, in the future, more easily align on an interface for a public `ColumnBase` that other libraries can use to extend cudf. Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18086 --- python/cudf/cudf/core/column/column.py | 9 +++++---- python/cudf/cudf/core/column/string.py | 24 ++++++++++++++---------- python/cudf/cudf/tests/test_string.py | 10 +++++++++- 3 files changed, 28 insertions(+), 15 deletions(-) diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index b57d1f03981..89ac39b2be5 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -2312,13 +2312,14 @@ def build_column( offset=offset, null_count=null_count, ) - elif dtype.type in (np.object_, np.str_): + elif dtype == CUDF_STRING_DTYPE: return cudf.core.column.StringColumn( - data=data, - mask=mask, + data=data, # type: ignore[arg-type] size=size, + dtype=dtype, + mask=mask, offset=offset, - children=children, + children=children, # type: ignore[arg-type] null_count=null_count, ) elif isinstance(dtype, ListDtype): diff --git a/python/cudf/cudf/core/column/string.py b/python/cudf/cudf/core/column/string.py index c0ad33ec7d6..28e8b98edfe 100644 --- a/python/cudf/cudf/core/column/string.py +++ b/python/cudf/cudf/core/column/string.py @@ -21,7 +21,7 @@ import cudf.core.column.datetime as datetime from cudf.api.types import is_integer, is_scalar, is_string_dtype from cudf.core._internals import binaryop -from cudf.core.buffer import acquire_spill_lock +from cudf.core.buffer import Buffer, acquire_spill_lock from cudf.core.column.column import ColumnBase from cudf.core.column.methods import ColumnMethods from cudf.core.scalar import pa_scalar_to_plc_scalar @@ -46,7 +46,6 @@ ScalarLike, SeriesOrIndex, ) - from cudf.core.buffer import Buffer from cudf.core.column.lists import ListColumn from cudf.core.column.numerical import NumericalColumn @@ -5595,13 +5594,14 @@ class StringColumn(column.ColumnBase): Parameters ---------- + data : Buffer + Buffer of the string data mask : Buffer The validity mask offset : int Data offset children : Tuple[Column] - Two non-null columns containing the string data and offsets - respectively + Columns containing the offsets """ _start_offset: int | None @@ -5629,14 +5629,20 @@ class StringColumn(column.ColumnBase): def __init__( self, - data: Buffer | None = None, + data: Buffer, + size: int | None, + dtype: np.dtype, mask: Buffer | None = None, - size: int | None = None, # TODO: make non-optional offset: int = 0, null_count: int | None = None, - children: tuple["column.ColumnBase", ...] = (), + children: tuple[column.ColumnBase] = (), # type: ignore[assignment] ): - dtype = cudf.api.types.dtype("object") + if not isinstance(data, Buffer): + raise ValueError("data must be a Buffer") + if dtype != CUDF_STRING_DTYPE: + raise ValueError(f"dtypy must be {CUDF_STRING_DTYPE}") + if len(children) > 1: + raise ValueError("StringColumn must have at most 1 offset column.") if size is None: for child in children: @@ -5731,8 +5737,6 @@ def base_size(self) -> int: # override for string column @property def data(self): - if self.base_data is None: - return None if self._data is None: if ( self.offset == 0 diff --git a/python/cudf/cudf/tests/test_string.py b/python/cudf/cudf/tests/test_string.py index 164fcb06624..18aee0001c4 100644 --- a/python/cudf/cudf/tests/test_string.py +++ b/python/cudf/cudf/tests/test_string.py @@ -13,8 +13,11 @@ import pyarrow as pa import pytest +import rmm + import cudf from cudf import concat +from cudf.core.buffer import as_buffer from cudf.core.column.string import StringColumn from cudf.core.index import Index from cudf.testing import assert_eq @@ -1202,7 +1205,12 @@ def test_string_misc_name(ps_gs, name): def test_string_no_children_properties(): - empty_col = StringColumn(children=()) + empty_col = StringColumn( + as_buffer(rmm.DeviceBuffer(size=0)), + size=0, + dtype=np.dtype("object"), + children=(), + ) assert empty_col.base_children == () assert empty_col.base_size == 0 From aa7f436bdc22fb5b25903252c437e32fbc8b33c0 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Wed, 26 Feb 2025 18:55:25 -0800 Subject: [PATCH 098/129] Allow pivot_table to accept single label index and column arguments (#18115) closes https://github.com/rapidsai/cudf/issues/12410 closes https://github.com/rapidsai/cudf/issues/12409 The fix just mirrors the pandas logic. Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18115 --- python/cudf/cudf/core/reshape.py | 20 +++++++++----------- python/cudf/cudf/tests/test_reshape.py | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/python/cudf/cudf/core/reshape.py b/python/cudf/cudf/core/reshape.py index c5d2fd349e9..7d76907916f 100644 --- a/python/cudf/cudf/core/reshape.py +++ b/python/cudf/cudf/core/reshape.py @@ -1526,9 +1526,9 @@ def pivot_table( ---------- data : DataFrame values : column name or list of column names to aggregate, optional - index : list of column names + index : scalar or list of column names Values to group by in the rows. - columns : list of column names + columns : scalar or list of column names Values to group by in the columns. aggfunc : str or dict, default "mean" If dict is passed, the key is column to aggregate @@ -1562,6 +1562,11 @@ def pivot_table( if sort is not True: raise NotImplementedError("sort is not supported yet") + if is_scalar(index): + index = [index] + if is_scalar(columns): + columns = [columns] + keys = index + columns values_passed = values is not None @@ -1620,15 +1625,8 @@ def pivot_table( table = table.fillna(fill_value) # discard the top level - if values_passed and not values_multi and table._data.multiindex: - column_names = table._data.level_names[1:] - table_columns = tuple( - map(lambda column: column[1:], table._column_names) - ) - table.columns = pd.MultiIndex.from_tuples( - tuples=table_columns, names=column_names - ) - + if values_passed and not values_multi and table._data.nlevels > 1: + table.columns = table._data.to_pandas_index.droplevel(0) if len(index) == 0 and len(columns) > 0: table = table.T diff --git a/python/cudf/cudf/tests/test_reshape.py b/python/cudf/cudf/tests/test_reshape.py index 7fbe072dde7..eae73e47955 100644 --- a/python/cudf/cudf/tests/test_reshape.py +++ b/python/cudf/cudf/tests/test_reshape.py @@ -798,6 +798,25 @@ def test_dataframe_pivot_table_simple(aggfunc, fill_value): assert_eq(expected, actual, check_dtype=False) +@pytest.mark.parametrize("index", ["A", ["A"]]) +@pytest.mark.parametrize("columns", ["C", ["C"]]) +def test_pivot_table_scalar_index_columns(index, columns): + data = { + "A": ["one", "one", "two", "three"] * 6, + "B": ["A", "B", "C"] * 8, + "C": ["foo", "foo", "foo", "bar", "bar", "bar"] * 4, + "D": range(24), + "E": range(24), + } + result = cudf.DataFrame(data).pivot_table( + values="D", index=index, columns=columns, aggfunc="sum" + ) + expected = pd.DataFrame(data).pivot_table( + values="D", index=index, columns=columns, aggfunc="sum" + ) + assert_eq(result, expected) + + def test_crosstab_simple(): a = np.array( [ From 7713bc1e8a339644815421b442abd6f91e04e15b Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Wed, 26 Feb 2025 19:06:55 -0800 Subject: [PATCH 099/129] Simplify DecimalDtype and DecimalColumn operations (#18111) Broken off (the non-breaking parts) from https://github.com/rapidsai/cudf/pull/18035/ as that PR will probably not move forward since it would require a pyarrow minimum version bump to 19 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18111 --- docs/cudf/source/conf.py | 1 + python/cudf/cudf/core/column/decimal.py | 30 +++++----------- python/cudf/cudf/core/column/timedelta.py | 4 ++- python/cudf/cudf/core/dtypes.py | 43 ++++++++++++----------- python/cudf/cudf/core/scalar.py | 6 ++-- 5 files changed, 37 insertions(+), 47 deletions(-) diff --git a/docs/cudf/source/conf.py b/docs/cudf/source/conf.py index c74da8d0ca9..8eea644363b 100644 --- a/docs/cudf/source/conf.py +++ b/docs/cudf/source/conf.py @@ -585,6 +585,7 @@ def on_missing_reference(app, env, node, contnode): ("py:class", "pd.DataFrame"), ("py:class", "pandas.core.indexes.frozen.FrozenList"), ("py:class", "pa.Array"), + ("py:class", "pa.Decimal128Type"), ("py:class", "ScalarLike"), ("py:class", "ParentType"), ("py:class", "pyarrow.lib.DataType"), diff --git a/python/cudf/cudf/core/column/decimal.py b/python/cudf/cudf/core/column/decimal.py index 3c603c8e6ef..8db6f805bce 100644 --- a/python/cudf/cudf/core/column/decimal.py +++ b/python/cudf/cudf/core/column/decimal.py @@ -13,7 +13,6 @@ import pylibcudf as plc import cudf -from cudf.api.types import is_scalar from cudf.core._internals import binaryop from cudf.core.buffer import acquire_spill_lock, as_buffer from cudf.core.column.column import ColumnBase @@ -73,11 +72,8 @@ def __cuda_array_interface__(self): def as_decimal_column( self, dtype: Dtype, - ) -> "DecimalBaseColumn": - if ( - isinstance(dtype, cudf.core.dtypes.DecimalDtype) - and dtype.scale < self.dtype.scale - ): + ) -> DecimalBaseColumn: + if isinstance(dtype, DecimalDtype) and dtype.scale < self.dtype.scale: warnings.warn( "cuDF truncates when downcasting decimals to a lower scale. " "To round, use Series.round() or DataFrame.round()." @@ -204,22 +200,17 @@ def normalize_binop_value(self, other) -> Self | cudf.Scalar: other = other.astype(self.dtype) return other if isinstance(other, cudf.Scalar) and isinstance( - # TODO: Should it be possible to cast scalars of other numerical - # types to decimal? other.dtype, - cudf.core.dtypes.DecimalDtype, + DecimalDtype, ): + # TODO: Should it be possible to cast scalars of other numerical + # types to decimal? if _same_precision_and_scale(self.dtype, other.dtype): other = other.astype(self.dtype) return other - elif is_scalar(other) and isinstance(other, (int, Decimal)): - other = Decimal(other) - metadata = other.as_tuple() - precision = max(len(metadata.digits), metadata.exponent) - scale = -cast(int, metadata.exponent) - return cudf.Scalar( - other, dtype=self.dtype.__class__(precision, scale) - ) + elif isinstance(other, (int, Decimal)): + dtype = self.dtype._from_decimal(Decimal(other)) + return cudf.Scalar(other, dtype=dtype) return NotImplemented def as_numerical_column( @@ -373,11 +364,6 @@ def __init__( children=children, ) - def __setitem__(self, key, value): - if isinstance(value, np.integer): - value = int(value) - super().__setitem__(key, value) - @classmethod def from_arrow(cls, data: pa.Array): dtype = Decimal64Dtype.from_arrow(data.type) diff --git a/python/cudf/cudf/core/column/timedelta.py b/python/cudf/cudf/core/column/timedelta.py index 8b0ef9f0cc8..d02681d389d 100644 --- a/python/cudf/cudf/core/column/timedelta.py +++ b/python/cudf/cudf/core/column/timedelta.py @@ -309,7 +309,9 @@ def total_seconds(self) -> ColumnBase: # https://github.com/rapidsai/cudf/issues/17664 return ( (self.astype(np.dtype(np.int64)) * conversion) - .astype(cudf.Decimal128Dtype(38, 9)) + .astype( + cudf.Decimal128Dtype(cudf.Decimal128Dtype.MAX_PRECISION, 9) + ) .round(decimals=abs(int(math.log10(conversion)))) .astype(np.dtype(np.float64)) ) diff --git a/python/cudf/cudf/core/dtypes.py b/python/cudf/cudf/core/dtypes.py index 977208f5eb4..ac9c4d23cc2 100644 --- a/python/cudf/cudf/core/dtypes.py +++ b/python/cudf/cudf/core/dtypes.py @@ -776,35 +776,36 @@ def _recursively_replace_fields(self, result: dict) -> dict: class DecimalDtype(_BaseDtype): _metadata = ("precision", "scale") - def __init__(self, precision, scale=0): + def __init__(self, precision: int, scale: int = 0) -> None: self._validate(precision, scale) - self._typ = pa.decimal128(precision, scale) + self._precision = precision + self._scale = scale @property - def str(self): + def str(self) -> str: return f"{self.name!s}({self.precision}, {self.scale})" @property - def precision(self): + def precision(self) -> int: """ The decimal precision, in number of decimal digits (an integer). """ - return self._typ.precision + return self._precision @precision.setter - def precision(self, value): + def precision(self, value: int) -> None: self._validate(value, self.scale) - self._typ = pa.decimal128(precision=value, scale=self.scale) + self._precision = value @property - def scale(self): + def scale(self) -> int: """ The decimal scale (an integer). """ - return self._typ.scale + return self._scale @property - def itemsize(self): + def itemsize(self) -> int: """ Length of one column element in bytes. """ @@ -815,14 +816,14 @@ def type(self): # might need to account for precision and scale here return decimal.Decimal - def to_arrow(self): + def to_arrow(self) -> pa.Decimal128Type: """ Return the equivalent ``pyarrow`` dtype. """ - return self._typ + return pa.decimal128(self.precision, self.scale) @classmethod - def from_arrow(cls, typ): + def from_arrow(cls, typ: pa.Decimal128Type) -> Self: """ Construct a cudf decimal dtype from a ``pyarrow`` dtype @@ -856,23 +857,23 @@ def __repr__(self): ) @classmethod - def _validate(cls, precision, scale=0): + def _validate(cls, precision: int, scale: int) -> None: if precision > cls.MAX_PRECISION: raise ValueError( f"Cannot construct a {cls.__name__}" f" with precision > {cls.MAX_PRECISION}" ) if abs(scale) > precision: - raise ValueError(f"scale={scale} exceeds precision={precision}") + raise ValueError(f"{scale=} cannot exceed {precision=}") @classmethod - def _from_decimal(cls, decimal): + def _from_decimal(cls, decimal: decimal.Decimal) -> Self: """ Create a cudf.DecimalDtype from a decimal.Decimal object """ metadata = decimal.as_tuple() - precision = max(len(metadata.digits), -metadata.exponent) - return cls(precision, -metadata.exponent) + precision = max(len(metadata.digits), -metadata.exponent) # type: ignore[operator] + return cls(precision, -metadata.exponent) # type: ignore[operator] def serialize(self) -> tuple[dict, list]: return ( @@ -885,7 +886,7 @@ def serialize(self) -> tuple[dict, list]: ) @classmethod - def deserialize(cls, header: dict, frames: list): + def deserialize(cls, header: dict, frames: list) -> Self: _check_type(cls, header, frames, is_valid_class=issubclass) return cls(header["precision"], header["scale"]) @@ -896,8 +897,8 @@ def __eq__(self, other: Dtype) -> bool: return False return self.precision == other.precision and self.scale == other.scale - def __hash__(self): - return hash(self._typ) + def __hash__(self) -> int: + return hash(self.to_arrow()) @doc_apply( diff --git a/python/cudf/cudf/core/scalar.py b/python/cudf/cudf/core/scalar.py index cf85282cccb..29139768a36 100644 --- a/python/cudf/cudf/core/scalar.py +++ b/python/cudf/cudf/core/scalar.py @@ -85,9 +85,9 @@ def _preprocess_host_value(value, dtype) -> tuple[ScalarLike, Dtype]: return value.as_py(), dtype if isinstance(dtype, cudf.core.dtypes.DecimalDtype): - value = pa.scalar( - value, type=pa.decimal128(dtype.precision, dtype.scale) - ).as_py() + if isinstance(value, np.integer): + value = int(value) + value = pa.scalar(value, type=dtype.to_arrow()).as_py() if isinstance(value, decimal.Decimal) and dtype is None: dtype = cudf.Decimal128Dtype._from_decimal(value) From 601d0a10c853ef837c948e536a8b5a11f4cd26ab Mon Sep 17 00:00:00 2001 From: GALI PREM SAGAR Date: Wed, 26 Feb 2025 21:34:11 -0600 Subject: [PATCH 100/129] Add `as_proxy_object` API to `cudf.pandas` (#18072) This is a public API to proxify true `pandas` or `cudf` objects. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Bradley Dice (https://github.com/bdice) - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/18072 --- python/cudf/cudf/pandas/__init__.py | 7 +- python/cudf/cudf/pandas/fast_slow_proxy.py | 27 +++++- .../cudf_pandas_tests/test_cudf_pandas.py | 88 +++++++++++++++++++ 3 files changed, 120 insertions(+), 2 deletions(-) diff --git a/python/cudf/cudf/pandas/__init__.py b/python/cudf/cudf/pandas/__init__.py index 52fc945709e..742a6b57e59 100644 --- a/python/cudf/cudf/pandas/__init__.py +++ b/python/cudf/cudf/pandas/__init__.py @@ -8,12 +8,17 @@ import pylibcudf import rmm.mr -from .fast_slow_proxy import is_proxy_instance, is_proxy_object +from .fast_slow_proxy import ( + as_proxy_object, + is_proxy_instance, + is_proxy_object, +) from .magics import load_ipython_extension from .profiler import Profiler __all__ = [ "Profiler", + "as_proxy_object", "install", "is_proxy_instance", "is_proxy_object", diff --git a/python/cudf/cudf/pandas/fast_slow_proxy.py b/python/cudf/cudf/pandas/fast_slow_proxy.py index 45944452c17..147971e8bee 100644 --- a/python/cudf/cudf/pandas/fast_slow_proxy.py +++ b/python/cudf/cudf/pandas/fast_slow_proxy.py @@ -151,7 +151,7 @@ def make_final_proxy_type( additional_attributes Mapping of additional attributes to add to the class (optional), these will override any defaulted attributes (e.g. - ``__init__`). If you want to remove a defaulted attribute + ``__init__``). If you want to remove a defaulted attribute completely, pass the special sentinel ``_DELETE`` as a value. postprocess Optional function called to allow the proxy to postprocess @@ -1335,6 +1335,31 @@ def _get_proxy_base_class(cls): return object +def as_proxy_object(obj: Any) -> Any: + """ + Wraps a cudf or pandas object in a proxy object if applicable. + + There will be no memory transfer, i.e., GPU objects stay on GPU and + CPU objects stay on CPU. The object will be wrapped in a + proxy object. This is useful for ensuring that the object is + compatible with the fast-slow proxy system. + + Parameters + ---------- + obj : Any + The object to wrap. + + Returns + ------- + Any + The wrapped proxy object if applicable, otherwise the original object. + """ + if _is_final_type(obj): + typ = get_final_type_map()[type(obj)] + return typ._fsproxy_wrap(obj, None) + return obj + + def is_proxy_instance(obj, type): return is_proxy_object(obj) and obj.__class__.__name__ == type.__name__ diff --git a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py index 47de8fb1435..d3bfd9298c2 100644 --- a/python/cudf/cudf_pandas_tests/test_cudf_pandas.py +++ b/python/cudf/cudf_pandas_tests/test_cudf_pandas.py @@ -44,6 +44,7 @@ OOMFallbackError, TypeFallbackError, _Unusable, + as_proxy_object, is_proxy_object, ) from cudf.testing import assert_eq @@ -1979,6 +1980,93 @@ def test_numpy_data_access(): assert type(expected) is type(actual) +@pytest.mark.parametrize( + "obj", + [ + pd.DataFrame({"a": [1, 2, 3]}), + pd.Series([1, 2, 3]), + pd.Index([1, 2, 3]), + pd.Categorical([1, 2, 3]), + pd.to_datetime(["2021-01-01", "2021-01-02"]), + pd.to_timedelta(["1 days", "2 days"]), + xpd.DataFrame({"a": [1, 2, 3]}), + xpd.Series([1, 2, 3]), + xpd.Index([1, 2, 3]), + xpd.Categorical([1, 2, 3]), + xpd.to_datetime(["2021-01-01", "2021-01-02"]), + xpd.to_timedelta(["1 days", "2 days"]), + cudf.DataFrame({"a": [1, 2, 3]}), + cudf.Series([1, 2, 3]), + cudf.Index([1, 2, 3]), + cudf.Index([1, 2, 3], dtype="category"), + cudf.to_datetime(["2021-01-01", "2021-01-02"]), + cudf.Index([1, 2, 3], dtype="timedelta64[ns]"), + [1, 2, 3], + {"a": 1, "b": 2}, + (1, 2, 3), + ], +) +def test_as_proxy_object(obj): + proxy_obj = as_proxy_object(obj) + if isinstance( + obj, + ( + pd.DataFrame, + pd.Series, + pd.Index, + pd.Categorical, + xpd.DataFrame, + xpd.Series, + xpd.Index, + xpd.Categorical, + cudf.DataFrame, + cudf.Series, + cudf.Index, + ), + ): + assert is_proxy_object(proxy_obj) + if isinstance(proxy_obj, xpd.DataFrame): + tm.assert_frame_equal(proxy_obj, xpd.DataFrame(obj)) + elif isinstance(proxy_obj, xpd.Series): + tm.assert_series_equal(proxy_obj, xpd.Series(obj)) + elif isinstance(proxy_obj, xpd.Index): + tm.assert_index_equal(proxy_obj, xpd.Index(obj)) + else: + tm.assert_equal(proxy_obj, obj) + else: + assert not is_proxy_object(proxy_obj) + assert proxy_obj == obj + + +def test_as_proxy_object_doesnot_copy_series(): + s = pd.Series([1, 2, 3]) + proxy_obj = as_proxy_object(s) + s[0] = 10 + assert proxy_obj[0] == 10 + tm.assert_series_equal(s, proxy_obj) + + +def test_as_proxy_object_doesnot_copy_dataframe(): + df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]}) + proxy_obj = as_proxy_object(df) + df.iloc[0, 0] = 10 + assert proxy_obj.iloc[0, 0] == 10 + tm.assert_frame_equal(df, proxy_obj) + + +def test_as_proxy_object_doesnot_copy_index(): + idx = pd.Index([1, 2, 3]) + proxy_obj = as_proxy_object(idx) + assert proxy_obj._fsproxy_wrapped is idx + + +def test_as_proxy_object_no_op_for_intermediates(): + s = pd.Series(["abc", "def", "ghi"]) + str_attr = s.str + proxy_obj = as_proxy_object(str_attr) + assert proxy_obj is str_attr + + def test_pickle_round_trip_proxy_numpy_array(array): arr, proxy_arr = array pickled_arr = BytesIO() From 10048b813bc4054c9a092f31194a676e7459e840 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Wed, 26 Feb 2025 20:26:23 -0800 Subject: [PATCH 101/129] Make Column.view/can_cast_safely accept a dtype object (#18066) Partially broken off from https://github.com/rapidsai/cudf/pull/17978 Since Column objects are technically private, not marking this as breaking. Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18066 --- python/cudf/cudf/core/column/categorical.py | 3 +- python/cudf/cudf/core/column/column.py | 10 ++---- python/cudf/cudf/core/column/datetime.py | 3 +- python/cudf/cudf/core/column/string.py | 35 +++++++++------------ python/cudf/cudf/core/column/timedelta.py | 13 +++++--- python/cudf/cudf/tests/test_column.py | 15 ++++++--- python/cudf/cudf/utils/utils.py | 3 +- 7 files changed, 44 insertions(+), 38 deletions(-) diff --git a/python/cudf/cudf/core/column/categorical.py b/python/cudf/cudf/core/column/categorical.py index a57ff9a7817..d41e448254c 100644 --- a/python/cudf/cudf/core/column/categorical.py +++ b/python/cudf/cudf/core/column/categorical.py @@ -36,6 +36,7 @@ ColumnBinaryOperand, ColumnLike, Dtype, + DtypeObj, ScalarLike, SeriesOrIndex, SeriesOrSingleColumnIndex, @@ -1168,7 +1169,7 @@ def _mimic_inplace( self._codes = other_col.codes return out - def view(self, dtype: Dtype) -> ColumnBase: + def view(self, dtype: DtypeObj) -> ColumnBase: raise NotImplementedError( "Categorical column views are not currently supported" ) diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index 89ac39b2be5..61f4f7d52fb 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -950,7 +950,7 @@ def copy(self, deep: bool = True) -> Self: ), ) - def view(self, dtype: Dtype) -> ColumnBase: + def view(self, dtype: DtypeObj) -> ColumnBase: """ View the data underlying a column as different dtype. The source column must divide evenly into the size of @@ -959,13 +959,9 @@ def view(self, dtype: Dtype) -> ColumnBase: Parameters ---------- - dtype : NumPy dtype, string + dtype : Dtype object The dtype to view the data as - """ - - dtype = cudf.dtype(dtype) - if dtype.kind in ("o", "u", "s"): raise TypeError( "Bytes viewed as str without metadata is ambiguous" @@ -1586,7 +1582,7 @@ def distinct_count(self, dropna: bool = True) -> int: self._distinct_count[dropna] = result return self._distinct_count[dropna] - def can_cast_safely(self, to_dtype: Dtype) -> bool: + def can_cast_safely(self, to_dtype: DtypeObj) -> bool: raise NotImplementedError() @acquire_spill_lock() diff --git a/python/cudf/cudf/core/column/datetime.py b/python/cudf/cudf/core/column/datetime.py index 92d5c39e69d..213e91d7b3f 100644 --- a/python/cudf/cudf/core/column/datetime.py +++ b/python/cudf/cudf/core/column/datetime.py @@ -47,6 +47,7 @@ ColumnBinaryOperand, DatetimeLikeScalar, Dtype, + DtypeObj, ScalarLike, ) from cudf.core.column.numerical import NumericalColumn @@ -837,7 +838,7 @@ def is_unique(self) -> bool: def isin(self, values: Sequence) -> ColumnBase: return cudf.core.tools.datetimes._isin_datetimelike(self, values) - def can_cast_safely(self, to_dtype: Dtype) -> bool: + def can_cast_safely(self, to_dtype: DtypeObj) -> bool: if to_dtype.kind == "M": # type: ignore[union-attr] to_res, _ = np.datetime_data(to_dtype) self_res, _ = np.datetime_data(self.dtype) diff --git a/python/cudf/cudf/core/column/string.py b/python/cudf/cudf/core/column/string.py index 28e8b98edfe..944f5cd6d26 100644 --- a/python/cudf/cudf/core/column/string.py +++ b/python/cudf/cudf/core/column/string.py @@ -43,6 +43,7 @@ ColumnBinaryOperand, ColumnLike, Dtype, + DtypeObj, ScalarLike, SeriesOrIndex, ) @@ -5640,7 +5641,7 @@ def __init__( if not isinstance(data, Buffer): raise ValueError("data must be a Buffer") if dtype != CUDF_STRING_DTYPE: - raise ValueError(f"dtypy must be {CUDF_STRING_DTYPE}") + raise ValueError(f"dtype must be {CUDF_STRING_DTYPE}") if len(children) > 1: raise ValueError("StringColumn must have at most 1 offset column.") @@ -5826,23 +5827,22 @@ def __contains__(self, item: ScalarLike) -> bool: other = [item] if is_scalar(item) else item return self.contains(column.as_column(other, dtype=self.dtype)).any() - def as_numerical_column(self, dtype: Dtype) -> NumericalColumn: - out_dtype = cudf.api.types.dtype(dtype) - if out_dtype.kind == "b": + def as_numerical_column(self, dtype: np.dtype) -> NumericalColumn: + if dtype.kind == "b": with acquire_spill_lock(): plc_column = plc.strings.attributes.count_characters( self.to_pylibcudf(mode="read") ) result = ColumnBase.from_pylibcudf(plc_column) return (result > np.int8(0)).fillna(False) - elif out_dtype.kind in {"i", "u"}: + elif dtype.kind in {"i", "u"}: if not self.is_integer().all(): raise ValueError( "Could not convert strings to integer " "type due to presence of non-integer values." ) cast_func = plc.strings.convert.convert_integers.to_integers - elif out_dtype.kind == "f": + elif dtype.kind == "f": if not self.is_float().all(): raise ValueError( "Could not convert strings to float " @@ -5850,10 +5850,8 @@ def as_numerical_column(self, dtype: Dtype) -> NumericalColumn: ) cast_func = plc.strings.convert.convert_floats.to_floats else: - raise ValueError( - f"dtype must be a numerical type, not {out_dtype}" - ) - plc_dtype = dtype_to_pylibcudf_type(out_dtype) + raise ValueError(f"dtype must be a numerical type, not {dtype}") + plc_dtype = dtype_to_pylibcudf_type(dtype) with acquire_spill_lock(): return type(self).from_pylibcudf( # type: ignore[return-value] cast_func(self.to_pylibcudf(mode="read"), plc_dtype) @@ -5973,17 +5971,15 @@ def to_pandas( else: return super().to_pandas(nullable=nullable, arrow_type=arrow_type) - def can_cast_safely(self, to_dtype: Dtype) -> bool: - to_dtype = cudf.api.types.dtype(to_dtype) - + def can_cast_safely(self, to_dtype: DtypeObj) -> bool: if self.dtype == to_dtype: return True - elif to_dtype.kind in {"i", "u"} and not self.is_integer().all(): - return False - elif to_dtype.kind == "f" and not self.is_float().all(): - return False - else: + elif to_dtype.kind in {"i", "u"} and self.is_integer().all(): + return True + elif to_dtype.kind == "f" and self.is_float().all(): return True + else: + return False def find_and_replace( self, @@ -6122,12 +6118,11 @@ def _binaryop( return NotImplemented @copy_docstring(ColumnBase.view) - def view(self, dtype) -> ColumnBase: + def view(self, dtype: DtypeObj) -> ColumnBase: if self.null_count > 0: raise ValueError( "Can not produce a view of a string column with nulls" ) - dtype = cudf.api.types.dtype(dtype) str_byte_offset = self.base_children[0].element_indexing(self.offset) str_end_byte_offset = self.base_children[0].element_indexing( self.offset + self.size diff --git a/python/cudf/cudf/core/column/timedelta.py b/python/cudf/cudf/core/column/timedelta.py index d02681d389d..e4d47f492c2 100644 --- a/python/cudf/cudf/core/column/timedelta.py +++ b/python/cudf/cudf/core/column/timedelta.py @@ -28,7 +28,12 @@ if TYPE_CHECKING: from collections.abc import Sequence - from cudf._typing import ColumnBinaryOperand, DatetimeLikeScalar, Dtype + from cudf._typing import ( + ColumnBinaryOperand, + DatetimeLikeScalar, + Dtype, + DtypeObj, + ) _unit_to_nanoseconds_conversion = { "ns": 1, @@ -380,10 +385,10 @@ def find_and_replace( ), ) - def can_cast_safely(self, to_dtype: Dtype) -> bool: - if to_dtype.kind == "m": # type: ignore[union-attr] + def can_cast_safely(self, to_dtype: DtypeObj) -> bool: + if to_dtype.kind == "m": to_res, _ = np.datetime_data(to_dtype) - self_res, _ = np.datetime_data(self.dtype) + self_res = self.time_unit max_int = np.iinfo(np.int64).max diff --git a/python/cudf/cudf/tests/test_column.py b/python/cudf/cudf/tests/test_column.py index 2996a88c171..b7cd2388f30 100644 --- a/python/cudf/cudf/tests/test_column.py +++ b/python/cudf/cudf/tests/test_column.py @@ -290,6 +290,8 @@ def test_column_chunked_array_creation(): ], ) def test_column_view_valid_numeric_to_numeric(data, from_dtype, to_dtype): + from_dtype = np.dtype(from_dtype) + to_dtype = np.dtype(to_dtype) cpu_data = np.asarray(data, dtype=from_dtype) gpu_data = as_column(data, dtype=from_dtype) @@ -314,6 +316,8 @@ def test_column_view_valid_numeric_to_numeric(data, from_dtype, to_dtype): ], ) def test_column_view_invalid_numeric_to_numeric(data, from_dtype, to_dtype): + from_dtype = np.dtype(from_dtype) + to_dtype = np.dtype(to_dtype) cpu_data = np.asarray(data, dtype=from_dtype) gpu_data = as_column(data, dtype=from_dtype) @@ -337,6 +341,7 @@ def test_column_view_invalid_numeric_to_numeric(data, from_dtype, to_dtype): ], ) def test_column_view_valid_string_to_numeric(data, to_dtype): + to_dtype = np.dtype(to_dtype) expect = cudf.Series._from_column(cudf.Series(data)._column.view(to_dtype)) got = cudf.Series(str_host_view(data, to_dtype)) @@ -352,7 +357,7 @@ def test_column_view_nulls_widths_even(): sr = cudf.Series(data, dtype="int32") expect = cudf.Series(expect_data, dtype="float32") - got = cudf.Series._from_column(sr._column.view("float32")) + got = cudf.Series._from_column(sr._column.view(np.dtype(np.float32))) assert_eq(expect, got) @@ -364,7 +369,7 @@ def test_column_view_nulls_widths_even(): sr = cudf.Series(data, dtype="float64") expect = cudf.Series(expect_data, dtype="int64") - got = cudf.Series._from_column(sr._column.view("int64")) + got = cudf.Series._from_column(sr._column.view(np.dtype(np.int64))) assert_eq(expect, got) @@ -376,7 +381,7 @@ def test_column_view_numeric_slice(slc): expect = cudf.Series(data[slc].view("int64")) got = cudf.Series._from_column( - sr._column.slice(slc.start, slc.stop).view("int64") + sr._column.slice(slc.start, slc.stop).view(np.dtype(np.int64)) ) assert_eq(expect, got) @@ -389,7 +394,9 @@ def test_column_view_string_slice(slc): data = ["a", "bcde", "cd", "efg", "h"] expect = cudf.Series._from_column( - cudf.Series(data)._column.slice(slc.start, slc.stop).view("int8") + cudf.Series(data) + ._column.slice(slc.start, slc.stop) + .view(np.dtype(np.int8)) ) got = cudf.Series(str_host_view(data[slc], "int8")) diff --git a/python/cudf/cudf/utils/utils.py b/python/cudf/cudf/utils/utils.py index fd946937945..2678a4f8116 100644 --- a/python/cudf/cudf/utils/utils.py +++ b/python/cudf/cudf/utils/utils.py @@ -18,9 +18,10 @@ import cudf.api.types from cudf.core import column from cudf.core.buffer import as_buffer +from cudf.utils.dtypes import SIZE_TYPE_DTYPE # The size of the mask in bytes -mask_dtype = cudf.api.types.dtype(np.int32) +mask_dtype = SIZE_TYPE_DTYPE mask_bitsize = mask_dtype.itemsize * 8 # Mapping from ufuncs to the corresponding binary operators. From b8ec71a24b4b8a3e3a997f38881ddfedd698610e Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Thu, 27 Feb 2025 06:56:24 -0500 Subject: [PATCH 102/129] Bump polars version to <1.24 (#18076) The PR upgrades the Polars version to 1.23. Authors: - Matthew Murray (https://github.com/Matt711) - GALI PREM SAGAR (https://github.com/galipremsagar) - Bradley Dice (https://github.com/bdice) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18076 --- ci/test_cudf_polars_polars_tests.sh | 2 ++ .../environments/all_cuda-118_arch-x86_64.yaml | 2 +- .../environments/all_cuda-128_arch-x86_64.yaml | 2 +- conda/recipes/cudf-polars/meta.yaml | 2 +- dependencies.yaml | 2 +- .../cudf_polars/cudf_polars/testing/plugin.py | 1 - python/cudf_polars/cudf_polars/utils/dtypes.py | 18 ++++++++++++++---- python/cudf_polars/pyproject.toml | 2 +- 8 files changed, 21 insertions(+), 10 deletions(-) diff --git a/ci/test_cudf_polars_polars_tests.sh b/ci/test_cudf_polars_polars_tests.sh index 3466edacfc5..1df7bb61834 100755 --- a/ci/test_cudf_polars_polars_tests.sh +++ b/ci/test_cudf_polars_polars_tests.sh @@ -26,6 +26,8 @@ git clone https://github.com/pola-rs/polars.git --branch "${TAG}" --depth 1 # Install requirements for running polars tests rapids-logger "Install polars test requirements" +# TODO: Remove sed command when polars-cloud supports 1.23 +sed -i '/^polars-cloud$/d' polars/py-polars/requirements-dev.txt rapids-pip-retry install -r polars/py-polars/requirements-dev.txt -r polars/py-polars/requirements-ci.txt # shellcheck disable=SC2317 diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index e7dbb765099..a23981b4e72 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -66,7 +66,7 @@ dependencies: - pandas - pandas>=2.0,<2.2.4dev0 - pandoc -- polars>=1.20,<1.23 +- polars>=1.20,<1.24 - pre-commit - ptxcompiler - pyarrow>=14.0.0,<20.0.0a0 diff --git a/conda/environments/all_cuda-128_arch-x86_64.yaml b/conda/environments/all_cuda-128_arch-x86_64.yaml index 342ec8d4b59..e2b9302dc36 100644 --- a/conda/environments/all_cuda-128_arch-x86_64.yaml +++ b/conda/environments/all_cuda-128_arch-x86_64.yaml @@ -64,7 +64,7 @@ dependencies: - pandas - pandas>=2.0,<2.2.4dev0 - pandoc -- polars>=1.20,<1.23 +- polars>=1.20,<1.24 - pre-commit - pyarrow>=14.0.0,<20.0.0a0 - pydata-sphinx-theme>=0.15.4 diff --git a/conda/recipes/cudf-polars/meta.yaml b/conda/recipes/cudf-polars/meta.yaml index 1d36ab2a3e4..64a147d3c63 100644 --- a/conda/recipes/cudf-polars/meta.yaml +++ b/conda/recipes/cudf-polars/meta.yaml @@ -43,7 +43,7 @@ requirements: run: - python - pylibcudf ={{ version }} - - polars >=1.20,<1.23 + - polars >=1.20,<1.24 - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} test: diff --git a/dependencies.yaml b/dependencies.yaml index c7869eee922..1578dadc793 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -813,7 +813,7 @@ dependencies: common: - output_types: [conda, requirements, pyproject] packages: - - polars>=1.20,<1.23 + - polars>=1.20,<1.24 run_cudf_polars_experimental: common: - output_types: [conda, requirements, pyproject] diff --git a/python/cudf_polars/cudf_polars/testing/plugin.py b/python/cudf_polars/cudf_polars/testing/plugin.py index a7b10a6e8fa..9b798688992 100644 --- a/python/cudf_polars/cudf_polars/testing/plugin.py +++ b/python/cudf_polars/cudf_polars/testing/plugin.py @@ -197,7 +197,6 @@ def pytest_configure(config: pytest.Config) -> None: "tests/unit/io/test_multiscan.py::test_include_file_paths[scan_csv-write_csv]": "Need to expose include_file_paths xref: cudf#18012", "tests/unit/streaming/test_streaming_io.py::test_parquet_eq_statistics[False]": "Debug output on stderr doesn't match", # Maybe flaky, order-dependent? - "tests/unit/test_projections.py::test_schema_full_outer_join_projection_pd_13287": "Order-specific result check, query is correct but in different order", "tests/unit/test_queries.py::test_group_by_agg_equals_zero_3535": "libcudf sums all nulls to null, not zero", } diff --git a/python/cudf_polars/cudf_polars/utils/dtypes.py b/python/cudf_polars/cudf_polars/utils/dtypes.py index 6bb5d78c488..85a4f007cf0 100644 --- a/python/cudf_polars/cudf_polars/utils/dtypes.py +++ b/python/cudf_polars/cudf_polars/utils/dtypes.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. # SPDX-License-Identifier: Apache-2.0 """Datatype utilities.""" @@ -71,7 +71,9 @@ def can_cast(from_: plc.DataType, to: plc.DataType) -> bool: ------- True if casting is supported, False otherwise """ - has_empty = from_.id() == plc.TypeId.EMPTY or to.id() == plc.TypeId.EMPTY + to_is_empty = to.id() == plc.TypeId.EMPTY + from_is_empty = from_.id() == plc.TypeId.EMPTY + has_empty = to_is_empty or from_is_empty return ( ( from_ == to @@ -84,8 +86,16 @@ def can_cast(from_: plc.DataType, to: plc.DataType) -> bool: ) ) ) - or (from_.id() == plc.TypeId.STRING and is_numeric_not_bool(to)) - or (to.id() == plc.TypeId.STRING and is_numeric_not_bool(from_)) + or ( + from_.id() == plc.TypeId.STRING + and not to_is_empty + and is_numeric_not_bool(to) + ) + or ( + to.id() == plc.TypeId.STRING + and not from_is_empty + and is_numeric_not_bool(from_) + ) ) diff --git a/python/cudf_polars/pyproject.toml b/python/cudf_polars/pyproject.toml index 9026a0c29ca..e9fc054efc2 100644 --- a/python/cudf_polars/pyproject.toml +++ b/python/cudf_polars/pyproject.toml @@ -19,7 +19,7 @@ authors = [ license = { text = "Apache 2.0" } requires-python = ">=3.10" dependencies = [ - "polars>=1.20,<1.23", + "polars>=1.20,<1.24", "pylibcudf==25.4.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. classifiers = [ From 25f17ad02615afd7cbb9ee2784de392f6e0c7a66 Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Thu, 27 Feb 2025 13:13:23 -0500 Subject: [PATCH 103/129] Make pylibcudf traits raise exceptions gracefully rather than terminating in C++ (#18117) Closes #18110 Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/18117 --- .../pylibcudf/libcudf/utilities/traits.pxd | 40 +++++++++---------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/python/pylibcudf/pylibcudf/libcudf/utilities/traits.pxd b/python/pylibcudf/pylibcudf/libcudf/utilities/traits.pxd index 93f13a7e11f..33749141590 100644 --- a/python/pylibcudf/pylibcudf/libcudf/utilities/traits.pxd +++ b/python/pylibcudf/pylibcudf/libcudf/utilities/traits.pxd @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from libcpp cimport bool from libcpp.vector cimport vector from pylibcudf.exception_handler cimport libcudf_exception_handler @@ -6,22 +6,22 @@ from pylibcudf.libcudf.types cimport data_type cdef extern from "cudf/utilities/traits.hpp" namespace "cudf" nogil: - cdef bool is_relationally_comparable(data_type) - cdef bool is_equality_comparable(data_type) - cdef bool is_numeric(data_type) - cdef bool is_numeric_not_bool(data_type) - cdef bool is_index_type(data_type) - cdef bool is_unsigned(data_type) - cdef bool is_integral(data_type) - cdef bool is_integral_not_bool(data_type) - cdef bool is_floating_point(data_type) - cdef bool is_boolean(data_type) - cdef bool is_timestamp(data_type) - cdef bool is_fixed_point(data_type) - cdef bool is_duration(data_type) - cdef bool is_chrono(data_type) - cdef bool is_dictionary(data_type) - cdef bool is_fixed_width(data_type) - cdef bool is_compound(data_type) - cdef bool is_nested(data_type) - cdef bool is_bit_castable(data_type, data_type) + cdef bool is_relationally_comparable(data_type) except +libcudf_exception_handler + cdef bool is_equality_comparable(data_type) except +libcudf_exception_handler + cdef bool is_numeric(data_type) except +libcudf_exception_handler + cdef bool is_numeric_not_bool(data_type) except +libcudf_exception_handler + cdef bool is_index_type(data_type) except +libcudf_exception_handler + cdef bool is_unsigned(data_type) except +libcudf_exception_handler + cdef bool is_integral(data_type) except +libcudf_exception_handler + cdef bool is_integral_not_bool(data_type) except +libcudf_exception_handler + cdef bool is_floating_point(data_type) except +libcudf_exception_handler + cdef bool is_boolean(data_type) except +libcudf_exception_handler + cdef bool is_timestamp(data_type) except +libcudf_exception_handler + cdef bool is_fixed_point(data_type) except +libcudf_exception_handler + cdef bool is_duration(data_type) except +libcudf_exception_handler + cdef bool is_chrono(data_type) except +libcudf_exception_handler + cdef bool is_dictionary(data_type) except +libcudf_exception_handler + cdef bool is_fixed_width(data_type) except +libcudf_exception_handler + cdef bool is_compound(data_type) except +libcudf_exception_handler + cdef bool is_nested(data_type) except +libcudf_exception_handler + cdef bool is_bit_castable(data_type, data_type) except +libcudf_exception_handler From b92d2c0adcca94a5cd04d9206fc89ca059f50f36 Mon Sep 17 00:00:00 2001 From: Vyas Ramasubramani Date: Thu, 27 Feb 2025 10:14:24 -0800 Subject: [PATCH 104/129] Remove now non-existent job (#18123) This job was removed from PRs in https://github.com/rapidsai/cudf/pull/18091 but I forgot to remove the corresponding nightly test job. --- .github/workflows/test.yaml | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 7046fd0e5dc..8357a12e221 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -46,18 +46,6 @@ jobs: arch: "amd64" container_image: "rapidsai/ci-conda:latest" run_script: "ci/test_cpp_memcheck.sh" - static-configure: - secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 - with: - build_type: ${{ inputs.build_type }} - branch: ${{ inputs.branch }} - date: ${{ inputs.date }} - sha: ${{ inputs.sha }} - # Use the wheel container so we can skip conda solves and since our - # primary static consumers (Spark) are not in conda anyway. - container_image: "rapidsai/ci-wheel:latest" - run_script: "ci/configure_cpp_static.sh" cpp-linters: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04 From 960bb28f426d004ed96ac066e07675d87bb186de Mon Sep 17 00:00:00 2001 From: Bradley Dice Date: Thu, 27 Feb 2025 13:25:54 -0600 Subject: [PATCH 105/129] Use cpu16 for build CI jobs (#18124) We use `cpu16` for PR jobs that build libcudf (conda and wheels). We also need to use `cpu16` for the corresponding jobs in `build.yaml`. --- .github/workflows/build.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 11104037c5e..148861c0fa2 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -34,6 +34,7 @@ jobs: branch: ${{ inputs.branch }} date: ${{ inputs.date }} sha: ${{ inputs.sha }} + node_type: "cpu16" python-build: needs: [cpp-build] secrets: inherit @@ -77,6 +78,7 @@ jobs: branch: ${{ inputs.branch }} sha: ${{ inputs.sha }} date: ${{ inputs.date }} + node_type: "cpu16" script: ci/build_wheel_libcudf.sh wheel-publish-libcudf: needs: wheel-build-libcudf From 08ea13a407f09babe647fef8cf98595c7e710f0b Mon Sep 17 00:00:00 2001 From: Michael Schellenberger Costa Date: Thu, 27 Feb 2025 20:35:44 +0100 Subject: [PATCH 106/129] Add include for `` (#18102) There are some files that use `std::function` and it seems they were relying on transitive includes from CCCL headers because building cudf fails with CCCL 2.8, which is the next CCCL release in line for rapids Authors: - Michael Schellenberger Costa (https://github.com/miscco) Approvers: - David Wendt (https://github.com/davidwendt) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18102 --- cpp/benchmarks/common/random_distribution_factory.cuh | 3 ++- cpp/src/column/column_device_view.cu | 3 ++- cpp/src/io/functions.cpp | 1 + cpp/src/io/json/host_tree_algorithms.cu | 3 ++- cpp/src/io/json/read_json.cu | 1 + cpp/src/io/orc/aggregate_orc_metadata.cpp | 1 + cpp/src/io/orc/writer_impl.cu | 1 + cpp/src/io/parquet/reader_impl_chunking.cu | 1 + cpp/src/io/parquet/writer_impl.cu | 1 + cpp/src/lists/dremel.cu | 4 +++- cpp/src/strings/regex/regex.cuh | 3 ++- cpp/src/strings/replace/multi_re.cu | 3 ++- cpp/src/table/row_operators.cu | 4 +++- cpp/src/text/bpe/load_merge_pairs.cu | 3 ++- cpp/tests/groupby/tdigest_tests.cu | 4 +++- cpp/tests/io/metadata_utilities.cpp | 4 +++- cpp/tests/io/parquet_writer_test.cpp | 1 + cpp/tests/reductions/scan_tests.cpp | 3 ++- 18 files changed, 33 insertions(+), 11 deletions(-) diff --git a/cpp/benchmarks/common/random_distribution_factory.cuh b/cpp/benchmarks/common/random_distribution_factory.cuh index c27616132d0..32424fbaaa3 100644 --- a/cpp/benchmarks/common/random_distribution_factory.cuh +++ b/cpp/benchmarks/common/random_distribution_factory.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -29,6 +29,7 @@ #include #include +#include #include #include diff --git a/cpp/src/column/column_device_view.cu b/cpp/src/column/column_device_view.cu index 9dc39f01ab3..c304d705f9b 100644 --- a/cpp/src/column/column_device_view.cu +++ b/cpp/src/column/column_device_view.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -25,6 +25,7 @@ #include #include +#include #include namespace cudf { diff --git a/cpp/src/io/functions.cpp b/cpp/src/io/functions.cpp index 53c1d335a40..204aca8a69c 100644 --- a/cpp/src/io/functions.cpp +++ b/cpp/src/io/functions.cpp @@ -36,6 +36,7 @@ #include #include +#include #include namespace cudf::io { diff --git a/cpp/src/io/json/host_tree_algorithms.cu b/cpp/src/io/json/host_tree_algorithms.cu index 7b9fc25d1cc..e506d60a2be 100644 --- a/cpp/src/io/json/host_tree_algorithms.cu +++ b/cpp/src/io/json/host_tree_algorithms.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -46,6 +46,7 @@ #include #include +#include namespace cudf::io::json::detail { diff --git a/cpp/src/io/json/read_json.cu b/cpp/src/io/json/read_json.cu index 0c95c2b05e8..c265ac5e316 100644 --- a/cpp/src/io/json/read_json.cu +++ b/cpp/src/io/json/read_json.cu @@ -43,6 +43,7 @@ #include #include +#include #include namespace cudf::io::json::detail { diff --git a/cpp/src/io/orc/aggregate_orc_metadata.cpp b/cpp/src/io/orc/aggregate_orc_metadata.cpp index 050bf692c14..77643d294e8 100644 --- a/cpp/src/io/orc/aggregate_orc_metadata.cpp +++ b/cpp/src/io/orc/aggregate_orc_metadata.cpp @@ -19,6 +19,7 @@ #include "io/utilities/row_selection.hpp" #include +#include #include namespace cudf::io::orc::detail { diff --git a/cpp/src/io/orc/writer_impl.cu b/cpp/src/io/orc/writer_impl.cu index dbf5e293c4e..3a20ffbce19 100644 --- a/cpp/src/io/orc/writer_impl.cu +++ b/cpp/src/io/orc/writer_impl.cu @@ -64,6 +64,7 @@ #include #include +#include #include #include #include diff --git a/cpp/src/io/parquet/reader_impl_chunking.cu b/cpp/src/io/parquet/reader_impl_chunking.cu index 03a37327e9b..be1e7d38fff 100644 --- a/cpp/src/io/parquet/reader_impl_chunking.cu +++ b/cpp/src/io/parquet/reader_impl_chunking.cu @@ -40,6 +40,7 @@ #include #include +#include #include namespace cudf::io::parquet::detail { diff --git a/cpp/src/io/parquet/writer_impl.cu b/cpp/src/io/parquet/writer_impl.cu index 9e50fafa8a7..4a410cec558 100644 --- a/cpp/src/io/parquet/writer_impl.cu +++ b/cpp/src/io/parquet/writer_impl.cu @@ -53,6 +53,7 @@ #include #include +#include #include #include #include diff --git a/cpp/src/lists/dremel.cu b/cpp/src/lists/dremel.cu index 469442d46d4..d7b1bf360fe 100644 --- a/cpp/src/lists/dremel.cu +++ b/cpp/src/lists/dremel.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -36,6 +36,8 @@ #include #include +#include + namespace cudf::detail { namespace { /** diff --git a/cpp/src/strings/regex/regex.cuh b/cpp/src/strings/regex/regex.cuh index d22fb04696c..6071a9fdd2d 100644 --- a/cpp/src/strings/regex/regex.cuh +++ b/cpp/src/strings/regex/regex.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -27,6 +27,7 @@ #include #include +#include #include namespace cudf { diff --git a/cpp/src/strings/replace/multi_re.cu b/cpp/src/strings/replace/multi_re.cu index 0777253bb38..af8b53ccd8c 100644 --- a/cpp/src/strings/replace/multi_re.cu +++ b/cpp/src/strings/replace/multi_re.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -39,6 +39,7 @@ #include #include +#include namespace cudf { namespace strings { diff --git a/cpp/src/table/row_operators.cu b/cpp/src/table/row_operators.cu index 990c4855a14..d77cc0cf17a 100644 --- a/cpp/src/table/row_operators.cu +++ b/cpp/src/table/row_operators.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -33,6 +33,8 @@ #include +#include + namespace cudf { namespace experimental { diff --git a/cpp/src/text/bpe/load_merge_pairs.cu b/cpp/src/text/bpe/load_merge_pairs.cu index a13a435a271..9118fe54ab2 100644 --- a/cpp/src/text/bpe/load_merge_pairs.cu +++ b/cpp/src/text/bpe/load_merge_pairs.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -33,6 +33,7 @@ #include #include +#include #include #include diff --git a/cpp/tests/groupby/tdigest_tests.cu b/cpp/tests/groupby/tdigest_tests.cu index 883a5093bd1..ad92e322ee2 100644 --- a/cpp/tests/groupby/tdigest_tests.cu +++ b/cpp/tests/groupby/tdigest_tests.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -30,6 +30,8 @@ #include #include +#include + namespace { /** * @brief Functor to generate a tdigest by key. diff --git a/cpp/tests/io/metadata_utilities.cpp b/cpp/tests/io/metadata_utilities.cpp index 380d66c53f9..980d8d8b3d1 100644 --- a/cpp/tests/io/metadata_utilities.cpp +++ b/cpp/tests/io/metadata_utilities.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -17,6 +17,8 @@ #include #include +#include + namespace cudf::test { void expect_metadata_equal(cudf::io::table_input_metadata in_meta, diff --git a/cpp/tests/io/parquet_writer_test.cpp b/cpp/tests/io/parquet_writer_test.cpp index e201dc0565c..d99e19822c0 100644 --- a/cpp/tests/io/parquet_writer_test.cpp +++ b/cpp/tests/io/parquet_writer_test.cpp @@ -33,6 +33,7 @@ #include #include +#include using cudf::test::iterators::no_nulls; diff --git a/cpp/tests/reductions/scan_tests.cpp b/cpp/tests/reductions/scan_tests.cpp index 5f911597b02..c6c419706e0 100644 --- a/cpp/tests/reductions/scan_tests.cpp +++ b/cpp/tests/reductions/scan_tests.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -28,6 +28,7 @@ #include #include +#include #include using aggregation = cudf::aggregation; From 4fda491e84bf212e16ab8d6ee5cf97da6d67362b Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Thu, 27 Feb 2025 15:42:40 -0500 Subject: [PATCH 107/129] Add new nvtext tokenized minhash API (#17944) Creates a new minhash API that works on ngrams of row elements given a list column of strings. ``` std::unique_ptr minhash_ngrams( cudf::lists_column_view const& input, cudf::size_type ngrams, uint32_t seed, cudf::device_span parameter_a, cudf::device_span parameter_b, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr); ``` The input column is expected to be rows of words (strings) and each row is hashed using a sliding window of words (ngrams) and then the permuted algorithm is re-used to produce the minhash values. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Ayush Dattagupta (https://github.com/ayushdg) - Matthew Murray (https://github.com/Matt711) - Yunsong Wang (https://github.com/PointKernel) - Muhammad Haseeb (https://github.com/mhaseeb123) - Shruti Shivakumar (https://github.com/shrshi) URL: https://github.com/rapidsai/cudf/pull/17944 --- cpp/include/nvtext/minhash.hpp | 94 +++++ cpp/src/text/minhash.cu | 392 +++++++++++++++--- cpp/tests/text/minhash_tests.cpp | 173 +++++++- python/cudf/cudf/core/column/string.py | 114 +++++ .../cudf/cudf/tests/text/test_text_methods.py | 42 ++ .../pylibcudf/libcudf/nvtext/minhash.pxd | 18 +- python/pylibcudf/pylibcudf/nvtext/minhash.pxd | 18 +- python/pylibcudf/pylibcudf/nvtext/minhash.pyi | 8 +- python/pylibcudf/pylibcudf/nvtext/minhash.pyx | 96 ++++- .../pylibcudf/tests/test_nvtext_minhash.py | 48 ++- 10 files changed, 911 insertions(+), 92 deletions(-) diff --git a/cpp/include/nvtext/minhash.hpp b/cpp/include/nvtext/minhash.hpp index 43f060fdafa..5f978a0d8ec 100644 --- a/cpp/include/nvtext/minhash.hpp +++ b/cpp/include/nvtext/minhash.hpp @@ -125,5 +125,99 @@ std::unique_ptr minhash64( rmm::cuda_stream_view stream = cudf::get_default_stream(), rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); +/** + * @brief Returns the minhash values for each input row + * + * This function uses MurmurHash3_x86_32 for the hash algorithm. + * + * The input row is first hashed using the given `seed` over a sliding window + * of `ngrams` of strings. These hash values are then combined with the `a` + * and `b` parameter values using the following formula: + * ``` + * max_hash = max of uint32 + * mp = (1 << 61) - 1 + * hv[i] = hash value of a ngrams at i + * pv[i] = ((hv[i] * a[i] + b[i]) % mp) & max_hash + * ``` + * + * This calculation is performed on each set of ngrams and the minimum value + * is computed as follows: + * ``` + * mh[j,i] = min(pv[i]) for all ngrams in row j + * and where i=[0,a.size()) + * ``` + * + * Any null row entries result in corresponding null output rows. + * + * @throw std::invalid_argument if the ngrams < 2 + * @throw std::invalid_argument if parameter_a is empty + * @throw std::invalid_argument if `parameter_b.size() != parameter_a.size()` + * @throw std::overflow_error if `parameter_a.size() * input.size()` exceeds the column size limit + * + * @param input Strings column to compute minhash + * @param ngrams The number of strings to hash within each row + * @param seed Seed value used for the hash algorithm + * @param parameter_a Values used for the permuted calculation + * @param parameter_b Values used for the permuted calculation + * @param stream CUDA stream used for device memory operations and kernel launches + * @param mr Device memory resource used to allocate the returned column's device memory + * @return List column of minhash values for each string per seed + */ +std::unique_ptr minhash_ngrams( + cudf::lists_column_view const& input, + cudf::size_type ngrams, + uint32_t seed, + cudf::device_span parameter_a, + cudf::device_span parameter_b, + rmm::cuda_stream_view stream = cudf::get_default_stream(), + rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); + +/** + * @brief Returns the minhash values for each input row + * + * This function uses MurmurHash3_x64_128 for the hash algorithm. + * + * The input row is first hashed using the given `seed` over a sliding window + * of `ngrams` of strings. These hash values are then combined with the `a` + * and `b` parameter values using the following formula: + * ``` + * max_hash = max of uint64 + * mp = (1 << 61) - 1 + * hv[i] = hash value of a ngrams at i + * pv[i] = ((hv[i] * a[i] + b[i]) % mp) & max_hash + * ``` + * + * This calculation is performed on each set of ngrams and the minimum value + * is computed as follows: + * ``` + * mh[j,i] = min(pv[i]) for all ngrams in row j + * and where i=[0,a.size()) + * ``` + * + * Any null row entries result in corresponding null output rows. + * + * @throw std::invalid_argument if the ngrams < 2 + * @throw std::invalid_argument if parameter_a is empty + * @throw std::invalid_argument if `parameter_b.size() != parameter_a.size()` + * @throw std::overflow_error if `parameter_a.size() * input.size()` exceeds the column size limit + * + * @param input List strings column to compute minhash + * @param ngrams The number of strings to hash within each row + * @param seed Seed value used for the hash algorithm + * @param parameter_a Values used for the permuted calculation + * @param parameter_b Values used for the permuted calculation + * @param stream CUDA stream used for device memory operations and kernel launches + * @param mr Device memory resource used to allocate the returned column's device memory + * @return List column of minhash values for each string per seed + */ +std::unique_ptr minhash64_ngrams( + cudf::lists_column_view const& input, + cudf::size_type ngrams, + uint64_t seed, + cudf::device_span parameter_a, + cudf::device_span parameter_b, + rmm::cuda_stream_view stream = cudf::get_default_stream(), + rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); + /** @} */ // end of group } // namespace CUDF_EXPORT nvtext diff --git a/cpp/src/text/minhash.cu b/cpp/src/text/minhash.cu index 50c16c8ba6c..663595af5df 100644 --- a/cpp/src/text/minhash.cu +++ b/cpp/src/text/minhash.cu @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -62,19 +63,20 @@ constexpr cudf::thread_index_type tile_size = block_size; constexpr cuda::std::size_t params_per_thread = 16; // Separate kernels are used to process strings above and below this value (in bytes). -constexpr cudf::size_type wide_string_threshold = 1 << 18; // 256K +constexpr cudf::size_type wide_row_threshold = 1 << 18; // 256K // The number of blocks per string for the above-threshold kernel processing. -constexpr cudf::size_type blocks_per_string = 64; +constexpr cudf::size_type blocks_per_row = 64; // The above values were determined using the redpajama and books_sample datasets /** * @brief Hashing kernel launched as a thread per tile-size (block or warp) + * for strings column * * This kernel computes the hashes for each string using the seed and the specified * hash function. The width is used to compute rolling substrings to hash over. * The hashes are stored in d_hashes to be used in the minhash_kernel. * - * This kernel also counts the number of strings above the wide_string_threshold + * This kernel also counts the number of strings above the wide_row_threshold * and proactively initializes the output values for those strings. * * @tparam HashFunction The hash function to use for this kernel @@ -84,7 +86,7 @@ constexpr cudf::size_type blocks_per_string = 64; * @param seed The seed used for the hash function * @param width Width in characters used for determining substrings to hash * @param d_hashes The resulting hash values are stored here - * @param threshold_count Stores the number of strings above wide_string_threshold + * @param threshold_count Stores the number of strings above wide_row_threshold * @param param_count Number of parameters (used for the proactive initialize) * @param d_results Final results vector (used for the proactive initialize) */ @@ -146,7 +148,7 @@ CUDF_KERNEL void minhash_seed_kernel(cudf::column_device_view const d_strings, } // logic appended here so an extra kernel is not required - if (size_bytes >= wide_string_threshold) { + if (size_bytes >= wide_row_threshold) { if (lane_idx == 0) { // count the number of wide strings cuda::atomic_ref ref{*threshold_count}; @@ -160,31 +162,130 @@ CUDF_KERNEL void minhash_seed_kernel(cudf::column_device_view const d_strings, } } +/** + * @brief Hashing kernel launched as a thread per tile-size (block or warp) + * for a lists column + * + * This kernel computes the hashes for each row using the seed and the specified + * hash function. The ngrams identifies consecutive strings to hash over in + * sliding window formation. The hashes are stored in d_hashes and used as input + * to the minhash_kernel. + * + * This kernel also counts the number of rows above the wide_row_threshold + * and proactively initializes the output values for those rows. + * + * @tparam HashFunction The hash function to use for this kernel + * @tparam hash_value_type Derived from HashFunction result_type + * + * @param d_input The input column to hash + * @param seed The seed used for the hash function + * @param ngrams Number of strings in each row to hash + * @param d_hashes The resulting hash values are stored here + * @param threshold_count Stores the number of rows above wide_row_threshold + * @param param_count Number of parameters (used for the proactive initialize) + * @param d_results Final results vector (used for the proactive initialize) + */ +template +CUDF_KERNEL void minhash_ngrams_kernel(cudf::detail::lists_column_device_view const d_input, + hash_value_type seed, + cudf::size_type ngrams, + hash_value_type* d_hashes, + cudf::size_type* threshold_count, + cudf::size_type param_count, + hash_value_type* d_results) +{ + auto const tid = cudf::detail::grid_1d::global_thread_id(); + auto const row_idx = tid / tile_size; + if (row_idx >= d_input.size()) { return; } + if (d_input.is_null(row_idx)) { return; } + + // retrieve this row's offset to locate the output position in d_hashes + auto const offsets_itr = d_input.offsets().data() + d_input.offset(); + auto const offset = offsets_itr[row_idx]; + auto const size_row = offsets_itr[row_idx + 1] - offset; + if (size_row == 0) { return; } + + auto const d_row = cudf::list_device_view(d_input, row_idx); + auto const lane_idx = static_cast(tid % tile_size); + + // hashes for this row/thread are stored here + auto seed_hashes = d_hashes + offset - offsets_itr[0] + lane_idx; + auto const hasher = HashFunction(seed); + + for (auto idx = lane_idx; idx < size_row; idx += tile_size, seed_hashes += tile_size) { + if (d_row.is_null(idx)) { + *seed_hashes = 0; + continue; + } + + auto next_idx = cuda::std::min(idx + ngrams, size_row - 1); + if ((idx != 0) && ((next_idx - idx) < ngrams)) { + *seed_hashes = 0; + continue; + } + + auto const first_str = d_row.element(idx); + auto const last_str = d_row.element(next_idx); + // build super-string since adjacent strings are contiguous in memory + auto const size = static_cast( + thrust::distance(first_str.data(), last_str.data()) + last_str.size_bytes()); + auto const hash_str = cudf::string_view(first_str.data(), size); + hash_value_type hv; + if constexpr (std::is_same_v) { + hv = hasher(hash_str); + } else { + hv = cuda::std::get<0>(hasher(hash_str)); + } + // disallowing hash to zero case + *seed_hashes = cuda::std::max(hv, hash_value_type{1}); + } + + // logic appended here to count long rows so an extra kernel is not required + if (size_row >= wide_row_threshold) { + if (lane_idx == 0) { + // count the number of wide rows + cuda::atomic_ref ref{*threshold_count}; + ref.fetch_add(1, cuda::std::memory_order_relaxed); + } + // initialize the output -- only needed for wider rows + auto d_output = d_results + (row_idx * param_count); + for (auto i = lane_idx; i < param_count; i += tile_size) { + d_output[i] = cuda::std::numeric_limits::max(); + } + } +} + /** * @brief Permutation calculation kernel * - * This kernel uses the hashes from the minhash_seed_kernel and the parameter_a and - * parameter_b values to compute the final output results. + * This kernel uses the hashes from the minhash_seed_kernel or minhash_ngrams_kernel + * and the 'parameter_a' and 'parameter_b' values to compute the final output. * The output is the number of input rows (N) by the number of parameter values (M). - * Each output[i] is the calculated result for parameter_a/b[0:M]. + * Each row output[i] is the calculated result for parameter_a/b[0:M]. + * + * This kernel is launched with either blocks per row of 1 for rows + * below the wide_row_threshold or blocks per row = blocks_per_rows + * for rows above wide_row_threshold. * - * This kernel is launched with either blocks per strings of 1 for strings - * below the wide_strings_threshold or blocks per string = blocks_per_strings - * for strings above wide_strings_threshold. + * Note that this was refactored to accommodate lists of strings which is possible + * since there is no need here to access the characters, only the hash values. + * The offsets and width are used to locate and count the hash values produced by + * kernels above for each input row. * + * @tparam offsets_type Type for the offsets iterator for the input column * @tparam hash_value_type Derived from HashFunction result_type - * @tparam blocks_per_string Number of blocks used to process each string + * @tparam blocks_per_row Number of blocks used to process each row * - * @param d_strings The input strings to hash - * @param indices The indices of the strings in d_strings to process + * @param offsets_itr The offsets are used to address the d_hashes + * @param indices The indices of the rows in the input column * @param parameter_a 1st set of parameters for the calculation result * @param parameter_b 2nd set of parameters for the calculation result - * @param width Used for calculating the number of available hashes in each string - * @param d_hashes The hash values computed in minhash_seed_kernel + * @param width Used for calculating the number of available hashes in each row + * @param d_hashes The hash values computed in one of the hash kernels * @param d_results Final results vector of calculate values */ -template -CUDF_KERNEL void minhash_kernel(cudf::column_device_view const d_strings, +template +CUDF_KERNEL void minhash_kernel(offsets_type offsets_itr, cudf::device_span indices, cudf::device_span parameter_a, cudf::device_span parameter_b, @@ -193,41 +294,36 @@ CUDF_KERNEL void minhash_kernel(cudf::column_device_view const d_strings, hash_value_type* d_results) { auto const tid = cudf::detail::grid_1d::global_thread_id(); - auto const idx = (tid / blocks_per_string) / block_size; + auto const idx = (tid / blocks_per_row) / block_size; if (idx >= indices.size()) { return; } - auto const str_idx = indices[idx]; - if (d_strings.is_null(str_idx)) { return; } + auto const row_idx = indices[idx]; auto const block = cooperative_groups::this_thread_block(); - int const section_idx = block.group_index().x % blocks_per_string; + int const section_idx = block.group_index().x % blocks_per_row; - auto const offsets = d_strings.child(cudf::strings_column_view::offsets_column_index); - auto const offsets_itr = - cudf::detail::input_offsetalator(offsets.head(), offsets.type(), d_strings.offset()); - auto const offset = offsets_itr[str_idx]; - auto const size_bytes = static_cast(offsets_itr[str_idx + 1] - offset); + auto const offset = offsets_itr[row_idx]; + auto const row_size = static_cast(offsets_itr[row_idx + 1] - offset); // number of items to process in this block; - // last block also includes any remainder values from the size_bytes/blocks_per_string truncation + // last block also includes any remainder values from the row_size/blocks_per_row truncation // example: - // each section_size for string with size 588090 and blocks_per_string=64 is 9188 + // each section_size for string with size 588090 and blocks_per_row=64 is 9188 // except the last section which is 9188 + (588090 % 64) = 9246 - auto const section_size = - (size_bytes / blocks_per_string) + - (section_idx < (blocks_per_string - 1) ? 0 : size_bytes % blocks_per_string); - auto const section_offset = section_idx * (size_bytes / blocks_per_string); + auto const section_size = (row_size / blocks_per_row) + + (section_idx < (blocks_per_row - 1) ? 0 : row_size % blocks_per_row); + auto const section_offset = section_idx * (row_size / blocks_per_row); // hash values for this block/section auto const seed_hashes = d_hashes + offset - offsets_itr[0] + section_offset; // width used here as a max value since a string's char-count <= byte-count auto const hashes_size = - section_idx < (blocks_per_string - 1) + section_idx < (blocks_per_row - 1) ? section_size - : cuda::std::max(static_cast(size_bytes > 0), section_size - width + 1); + : cuda::std::max(static_cast(row_size > 0), section_size - width + 1); - auto const init = size_bytes == 0 ? 0 : cuda::std::numeric_limits::max(); + auto const init = row_size == 0 ? 0 : cuda::std::numeric_limits::max(); auto const lane_idx = block.thread_rank(); - auto const d_output = d_results + (str_idx * parameter_a.size()); + auto const d_output = d_results + (row_idx * parameter_a.size()); auto const begin = seed_hashes + lane_idx; auto const end = seed_hashes + hashes_size; @@ -273,7 +369,7 @@ CUDF_KERNEL void minhash_kernel(cudf::column_device_view const d_strings, // cooperative groups does not have a min function and cub::BlockReduce was slower auto const minv = thrust::reduce(thrust::seq, values, values + block_size, init, thrust::minimum{}); - if constexpr (blocks_per_string > 1) { + if constexpr (blocks_per_row > 1) { // accumulates mins for each block into d_output cuda::atomic_ref ref{d_output[lane_idx + i]}; ref.fetch_min(minv, cuda::std::memory_order_relaxed); @@ -285,6 +381,46 @@ CUDF_KERNEL void minhash_kernel(cudf::column_device_view const d_strings, } } +/** + * @brief Partition input rows by row size + * + * The returned index is the first row above the wide_row_threshold size. + * The returned vector are the indices partitioned above and below the + * wide_row_threshold size. + * + * @param size Number of rows in the input column + * @param threshold_count Number of rows above wide_row_threshold + * @param tfn Transform function returns the size of each row + * @param stream Stream used for allocation and kernel launches + */ +template +std::pair> partition_input( + cudf::size_type size, + cudf::size_type threshold_count, + transform_fn tfn, + rmm::cuda_stream_view stream) +{ + auto indices = rmm::device_uvector(size, stream); + thrust::sequence(rmm::exec_policy(stream), indices.begin(), indices.end()); + cudf::size_type threshold_index = threshold_count < size ? size : 0; + + // if we counted a split of above/below threshold then + // compute partitions based on the size of each string + if ((threshold_count > 0) && (threshold_count < size)) { + auto sizes = rmm::device_uvector(size, stream); + auto begin = thrust::counting_iterator(0); + auto end = begin + size; + thrust::transform(rmm::exec_policy_nosync(stream), begin, end, sizes.data(), tfn); + // these 2 are slightly faster than using partition() + thrust::sort_by_key( + rmm::exec_policy_nosync(stream), sizes.begin(), sizes.end(), indices.begin()); + auto const lb = thrust::lower_bound( + rmm::exec_policy_nosync(stream), sizes.begin(), sizes.end(), wide_row_threshold); + threshold_index = static_cast(thrust::distance(sizes.begin(), lb)); + } + return {threshold_index, std::move(indices)}; +} + template std::unique_ptr minhash_fn(cudf::strings_column_view const& input, hash_value_type seed, @@ -334,40 +470,112 @@ std::unique_ptr minhash_fn(cudf::strings_column_view const& input, d_threshold_count.data(), parameter_a.size(), d_results); - auto const threshold_count = d_threshold_count.value(stream); - auto indices = rmm::device_uvector(input.size(), stream); - thrust::sequence(rmm::exec_policy(stream), indices.begin(), indices.end()); - cudf::size_type threshold_index = threshold_count < input.size() ? input.size() : 0; + auto transform_fn = [d_strings = *d_strings] __device__(auto idx) -> cudf::size_type { + if (d_strings.is_null(idx)) { return 0; } + return d_strings.element(idx).size_bytes(); + }; + auto [threshold_index, indices] = + partition_input(input.size(), d_threshold_count.value(stream), transform_fn, stream); - // if we counted a split of above/below threshold then - // compute partitions based on the size of each string - if ((threshold_count > 0) && (threshold_count < input.size())) { - auto sizes = rmm::device_uvector(input.size(), stream); - thrust::transform(rmm::exec_policy_nosync(stream), - thrust::counting_iterator(0), - thrust::counting_iterator(input.size()), - sizes.data(), - cuda::proclaim_return_type( - [d_strings = *d_strings] __device__(auto idx) -> cudf::size_type { - if (d_strings.is_null(idx)) { return 0; } - return d_strings.element(idx).size_bytes(); - })); - thrust::sort_by_key( - rmm::exec_policy_nosync(stream), sizes.begin(), sizes.end(), indices.begin()); - auto const lb = thrust::lower_bound( - rmm::exec_policy_nosync(stream), sizes.begin(), sizes.end(), wide_string_threshold); - threshold_index = static_cast(thrust::distance(sizes.begin(), lb)); + auto input_offsets = + cudf::detail::offsetalator_factory::make_input_iterator(input.offsets(), input.offset()); + using offsets_type = decltype(input_offsets); + + // handle the strings below the threshold width + if (threshold_index > 0) { + auto d_indices = cudf::device_span(indices.data(), threshold_index); + cudf::detail::grid_1d grid{static_cast(d_indices.size()) * block_size, + block_size}; + minhash_kernel + <<>>( + input_offsets, d_indices, parameter_a, parameter_b, width, d_hashes.data(), d_results); + } + + // handle the strings above the threshold width + if (threshold_index < input.size()) { + auto const count = static_cast(input.size() - threshold_index); + auto d_indices = + cudf::device_span(indices.data() + threshold_index, count); + cudf::detail::grid_1d grid{count * block_size * blocks_per_row, block_size}; + minhash_kernel + <<>>( + input_offsets, d_indices, parameter_a, parameter_b, width, d_hashes.data(), d_results); } + return results; +} + +template +std::unique_ptr minhash_ngrams_fn( + cudf::lists_column_view const& input, + cudf::size_type ngrams, + hash_value_type seed, + cudf::device_span parameter_a, + cudf::device_span parameter_b, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) +{ + CUDF_EXPECTS(ngrams >= 2, + "Parameter ngrams should be an integer value of 2 or greater", + std::invalid_argument); + CUDF_EXPECTS(!parameter_a.empty(), "Parameters A and B cannot be empty", std::invalid_argument); + CUDF_EXPECTS(parameter_a.size() == parameter_b.size(), + "Parameters A and B should have the same number of elements", + std::invalid_argument); + CUDF_EXPECTS( + (static_cast(input.size()) * parameter_a.size()) < + static_cast(std::numeric_limits::max()), + "The number of parameters times the number of input rows exceeds the column size limit", + std::overflow_error); + + auto const output_type = cudf::data_type{cudf::type_to_id()}; + if (input.is_empty()) { return cudf::make_empty_column(output_type); } + + auto const d_input = cudf::column_device_view::create(input.parent(), stream); + + auto results = + cudf::make_numeric_column(output_type, + input.size() * static_cast(parameter_a.size()), + cudf::mask_state::UNALLOCATED, + stream, + mr); + auto d_results = results->mutable_view().data(); + + cudf::detail::grid_1d grid{static_cast(input.size()) * block_size, + block_size}; + auto const hashes_size = input.child().size(); + auto d_hashes = rmm::device_uvector(hashes_size, stream); + auto d_threshold_count = cudf::detail::device_scalar(0, stream); + + auto d_list = cudf::detail::lists_column_device_view(*d_input); + minhash_ngrams_kernel + <<>>(d_list, + seed, + ngrams, + d_hashes.data(), + d_threshold_count.data(), + parameter_a.size(), + d_results); + + auto sizes_fn = [d_list] __device__(auto idx) -> cudf::size_type { + if (d_list.is_null(idx)) { return 0; } + return cudf::list_device_view(d_list, idx).size(); + }; + auto [threshold_index, indices] = + partition_input(input.size(), d_threshold_count.value(stream), sizes_fn, stream); + + auto input_offsets = input.offsets_begin(); // already includes input.offset() + using offset_type = decltype(input_offsets); + // handle the strings below the threshold width if (threshold_index > 0) { auto d_indices = cudf::device_span(indices.data(), threshold_index); cudf::detail::grid_1d grid{static_cast(d_indices.size()) * block_size, block_size}; - minhash_kernel + minhash_kernel <<>>( - *d_strings, d_indices, parameter_a, parameter_b, width, d_hashes.data(), d_results); + input_offsets, d_indices, parameter_a, parameter_b, ngrams, d_hashes.data(), d_results); } // handle the strings above the threshold width @@ -375,10 +583,10 @@ std::unique_ptr minhash_fn(cudf::strings_column_view const& input, auto const count = static_cast(input.size() - threshold_index); auto d_indices = cudf::device_span(indices.data() + threshold_index, count); - cudf::detail::grid_1d grid{count * block_size * blocks_per_string, block_size}; - minhash_kernel + cudf::detail::grid_1d grid{count * block_size * blocks_per_row, block_size}; + minhash_kernel <<>>( - *d_strings, d_indices, parameter_a, parameter_b, width, d_hashes.data(), d_results); + input_offsets, d_indices, parameter_a, parameter_b, ngrams, d_hashes.data(), d_results); } return results; @@ -426,6 +634,20 @@ std::unique_ptr minhash(cudf::strings_column_view const& input, return build_list_result(input.parent(), std::move(hashes), parameter_a.size(), stream, mr); } +std::unique_ptr minhash_ngrams(cudf::lists_column_view const& input, + cudf::size_type ngrams, + uint32_t seed, + cudf::device_span parameter_a, + cudf::device_span parameter_b, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) +{ + using HashFunction = cudf::hashing::detail::MurmurHash3_x86_32; + auto hashes = detail::minhash_ngrams_fn( + input, ngrams, seed, parameter_a, parameter_b, stream, mr); + return build_list_result(input.parent(), std::move(hashes), parameter_a.size(), stream, mr); +} + std::unique_ptr minhash64(cudf::strings_column_view const& input, uint64_t seed, cudf::device_span parameter_a, @@ -440,6 +662,20 @@ std::unique_ptr minhash64(cudf::strings_column_view const& input, return build_list_result(input.parent(), std::move(hashes), parameter_a.size(), stream, mr); } +std::unique_ptr minhash64_ngrams(cudf::lists_column_view const& input, + cudf::size_type ngrams, + uint64_t seed, + cudf::device_span parameter_a, + cudf::device_span parameter_b, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) +{ + using HashFunction = cudf::hashing::detail::MurmurHash3_x64_128; + auto hashes = detail::minhash_ngrams_fn( + input, ngrams, seed, parameter_a, parameter_b, stream, mr); + return build_list_result(input.parent(), std::move(hashes), parameter_a.size(), stream, mr); +} + } // namespace detail std::unique_ptr minhash(cudf::strings_column_view const& input, @@ -454,6 +690,19 @@ std::unique_ptr minhash(cudf::strings_column_view const& input, return detail::minhash(input, seed, parameter_a, parameter_b, width, stream, mr); } +std::unique_ptr minhash_ngrams(cudf::lists_column_view const& input, + cudf::size_type ngrams, + uint32_t seed, + cudf::device_span parameter_a, + cudf::device_span parameter_b, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) + +{ + CUDF_FUNC_RANGE(); + return detail::minhash_ngrams(input, ngrams, seed, parameter_a, parameter_b, stream, mr); +} + std::unique_ptr minhash64(cudf::strings_column_view const& input, uint64_t seed, cudf::device_span parameter_a, @@ -466,4 +715,17 @@ std::unique_ptr minhash64(cudf::strings_column_view const& input, return detail::minhash64(input, seed, parameter_a, parameter_b, width, stream, mr); } +std::unique_ptr minhash64_ngrams(cudf::lists_column_view const& input, + cudf::size_type ngrams, + uint64_t seed, + cudf::device_span parameter_a, + cudf::device_span parameter_b, + rmm::cuda_stream_view stream, + rmm::device_async_resource_ref mr) + +{ + CUDF_FUNC_RANGE(); + return detail::minhash64_ngrams(input, ngrams, seed, parameter_a, parameter_b, stream, mr); +} + } // namespace nvtext diff --git a/cpp/tests/text/minhash_tests.cpp b/cpp/tests/text/minhash_tests.cpp index 8bfb17e0efd..db43484ab09 100644 --- a/cpp/tests/text/minhash_tests.cpp +++ b/cpp/tests/text/minhash_tests.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2023-2024, NVIDIA CORPORATION. + * Copyright (c) 2023-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -187,6 +187,15 @@ TEST_F(MinHashTest, EmptyTest) auto params64 = cudf::test::fixed_width_column_wrapper({1, 2, 3}); results = nvtext::minhash64(view, 0, cudf::column_view(params64), cudf::column_view(params64), 4); EXPECT_EQ(results->size(), 0); + + auto empty = cudf::test::lists_column_wrapper(); + auto lview = cudf::lists_column_view(empty); + results = + nvtext::minhash_ngrams(lview, 4, 0, cudf::column_view(params), cudf::column_view(params)); + EXPECT_EQ(results->size(), 0); + results = + nvtext::minhash64_ngrams(lview, 4, 0, cudf::column_view(params64), cudf::column_view(params64)); + EXPECT_EQ(results->size(), 0); } TEST_F(MinHashTest, ErrorsTest) @@ -194,17 +203,20 @@ TEST_F(MinHashTest, ErrorsTest) auto input = cudf::test::strings_column_wrapper({"this string intentionally left blank"}); auto view = cudf::strings_column_view(input); auto empty = cudf::test::fixed_width_column_wrapper(); - EXPECT_THROW(nvtext::minhash(view, 0, cudf::column_view(empty), cudf::column_view(empty), 0), - std::invalid_argument); + auto eview = cudf::column_view(empty); + EXPECT_THROW(nvtext::minhash(view, 0, eview, eview, 0), std::invalid_argument); auto empty64 = cudf::test::fixed_width_column_wrapper(); - EXPECT_THROW( - nvtext::minhash64(view, 0, cudf::column_view(empty64), cudf::column_view(empty64), 0), - std::invalid_argument); - EXPECT_THROW(nvtext::minhash(view, 0, cudf::column_view(empty), cudf::column_view(empty), 4), - std::invalid_argument); - EXPECT_THROW( - nvtext::minhash64(view, 0, cudf::column_view(empty64), cudf::column_view(empty64), 4), - std::invalid_argument); + auto eview64 = cudf::column_view(empty64); + EXPECT_THROW(nvtext::minhash64(view, 0, eview64, eview64, 0), std::invalid_argument); + EXPECT_THROW(nvtext::minhash(view, 0, eview, eview, 4), std::invalid_argument); + EXPECT_THROW(nvtext::minhash64(view, 0, eview64, eview64, 4), std::invalid_argument); + + auto empty_list = cudf::test::lists_column_wrapper(); + auto lview = cudf::lists_column_view(empty_list); + EXPECT_THROW(nvtext::minhash_ngrams(lview, 0, 0, eview, eview), std::invalid_argument); + EXPECT_THROW(nvtext::minhash64_ngrams(lview, 0, 0, eview64, eview64), std::invalid_argument); + EXPECT_THROW(nvtext::minhash_ngrams(lview, 4, 0, eview, eview), std::invalid_argument); + EXPECT_THROW(nvtext::minhash64_ngrams(lview, 4, 0, eview64, eview64), std::invalid_argument); std::vector h_input(50000, ""); input = cudf::test::strings_column_wrapper(h_input.begin(), h_input.end()); @@ -212,16 +224,133 @@ TEST_F(MinHashTest, ErrorsTest) auto const zeroes = thrust::constant_iterator(0); auto params = cudf::test::fixed_width_column_wrapper(zeroes, zeroes + 50000); - EXPECT_THROW(nvtext::minhash(view, 0, cudf::column_view(params), cudf::column_view(params), 4), - std::overflow_error); + auto pview = cudf::column_view(params); + EXPECT_THROW(nvtext::minhash(view, 0, pview, pview, 4), std::overflow_error); auto params64 = cudf::test::fixed_width_column_wrapper(zeroes, zeroes + 50000); - EXPECT_THROW( - nvtext::minhash64(view, 0, cudf::column_view(params64), cudf::column_view(params64), 4), - std::overflow_error); - - EXPECT_THROW(nvtext::minhash(view, 0, cudf::column_view(params), cudf::column_view(empty), 4), - std::invalid_argument); - EXPECT_THROW( - nvtext::minhash64(view, 0, cudf::column_view(params64), cudf::column_view(empty64), 4), - std::invalid_argument); + auto pview64 = cudf::column_view(params64); + EXPECT_THROW(nvtext::minhash64(view, 0, pview64, pview64, 4), std::overflow_error); + + auto offsets = cudf::test::fixed_width_column_wrapper( + thrust::counting_iterator(0), + thrust::counting_iterator(h_input.size() + 1)); + auto input_ngrams = + cudf::make_lists_column(h_input.size(), offsets.release(), input.release(), 0, {}); + lview = cudf::lists_column_view(input_ngrams->view()); + EXPECT_THROW(nvtext::minhash_ngrams(lview, 4, 0, pview, pview), std::overflow_error); + EXPECT_THROW(nvtext::minhash64_ngrams(lview, 4, 0, pview64, pview64), std::overflow_error); +} + +TEST_F(MinHashTest, Ngrams) +{ + using LCWS = cudf::test::lists_column_wrapper; + auto input = + LCWS({LCWS{"The", "quick", "brown", "fox", "jumpéd", "over", "the", "lazy", "brown", "dog."}, + LCWS{"The", "quick", "brown", "fox", "jumpéd", "over", "the", "lazy", "", "dog."}, + LCWS{"short", "row"}}); + + auto view = cudf::lists_column_view(input); + + auto first = thrust::counting_iterator(10); + auto params = cudf::test::fixed_width_column_wrapper(first, first + 3); + auto results = + nvtext::minhash_ngrams(view, 4, 0, cudf::column_view(params), cudf::column_view(params)); + using LCW32 = cudf::test::lists_column_wrapper; + // clang-format off + LCW32 expected({ + LCW32{ 230924604u, 55492793u, 963436400u}, + LCW32{ 230924604u, 367515795u, 963436400u}, + LCW32{2380648568u, 1330223236u, 279797904u} + }); + // clang-format on + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + + auto params64 = cudf::test::fixed_width_column_wrapper(first, first + 3); + auto results64 = + nvtext::minhash64_ngrams(view, 4, 0, cudf::column_view(params64), cudf::column_view(params64)); + using LCW64 = cudf::test::lists_column_wrapper; + // clang-format off + LCW64 expected64({ + LCW64{ 208926840193078200ul, 576399628675212695ul, 312927673584437419ul}, + LCW64{ 677038498284219393ul, 326338087730412201ul, 298455901014050223ul}, + LCW64{1493265692486268500ul, 720255058049417768ul, 2253087432826260995ul} + }); + // clang-format on + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results64, expected64); +} + +TEST_F(MinHashTest, NgramsWide) +{ + auto many = std::vector(1024, "hello"); + auto str_data = cudf::test::strings_column_wrapper(many.begin(), many.end()); + auto offsets = + cudf::test::fixed_width_column_wrapper({0ul, many.size() / 2, many.size()}); + auto input = cudf::make_lists_column(2, offsets.release(), str_data.release(), 0, {}); + + auto view = cudf::lists_column_view(input->view()); + + auto first = thrust::counting_iterator(10); + auto params = cudf::test::fixed_width_column_wrapper(first, first + 3); + auto results = + nvtext::minhash_ngrams(view, 4, 0, cudf::column_view(params), cudf::column_view(params)); + using LCW32 = cudf::test::lists_column_wrapper; + // clang-format off + LCW32 expected({ + LCW32{ 571536396u, 2346676954u, 4121817512u}, + LCW32{ 571536396u, 2346676954u, 4121817512u} + }); + // clang-format on + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + + auto params64 = cudf::test::fixed_width_column_wrapper(first, first + 3); + auto results64 = + nvtext::minhash64_ngrams(view, 4, 0, cudf::column_view(params64), cudf::column_view(params64)); + using LCW64 = cudf::test::lists_column_wrapper; + // clang-format off + LCW64 expected64({ + LCW64{ 1947142336021414174ul, 1219519365938078011ul, 491896395854741840ul}, + LCW64{ 1947142336021414174ul, 1219519365938078011ul, 491896395854741840ul} + }); + // clang-format on + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results64, expected64); +} + +TEST_F(MinHashTest, NgramsSliced) +{ + using LCWS = cudf::test::lists_column_wrapper; + auto input = + LCWS({LCWS{"ignored", "row"}, + LCWS{"The", "quick", "brown", "fox", "jumpéd", "over", "the", "lazy", "brown", "dog."}, + LCWS{"The", "quick", "brown", "fox", "jumpéd", "over", "the", "lazy", "", "dog."}, + LCWS{"short", "row"}, + LCWS{"ignored", "row"}}); + + auto view = cudf::lists_column_view(cudf::slice(input, {1, 4}).front()); + auto first = thrust::counting_iterator(10); + + auto params = cudf::test::fixed_width_column_wrapper(first, first + 3); + auto results = + nvtext::minhash_ngrams(view, 4, 0, cudf::column_view(params), cudf::column_view(params)); + + using LCW32 = cudf::test::lists_column_wrapper; + // clang-format off + LCW32 expected({ + LCW32{ 230924604u, 55492793u, 963436400u}, + LCW32{ 230924604u, 367515795u, 963436400u}, + LCW32{2380648568u, 1330223236u, 279797904u} + }); + // clang-format on + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results, expected); + + auto params64 = cudf::test::fixed_width_column_wrapper(first, first + 3); + auto results64 = + nvtext::minhash64_ngrams(view, 4, 0, cudf::column_view(params64), cudf::column_view(params64)); + using LCW64 = cudf::test::lists_column_wrapper; + // clang-format off + LCW64 expected64({ + LCW64{ 208926840193078200ul, 576399628675212695ul, 312927673584437419ul}, + LCW64{ 677038498284219393ul, 326338087730412201ul, 298455901014050223ul}, + LCW64{1493265692486268500ul, 720255058049417768ul, 2253087432826260995ul} + }); + // clang-format on + CUDF_TEST_EXPECT_COLUMNS_EQUAL(*results64, expected64); } diff --git a/python/cudf/cudf/core/column/string.py b/python/cudf/cudf/core/column/string.py index 944f5cd6d26..b82ec1958fb 100644 --- a/python/cudf/cudf/core/column/string.py +++ b/python/cudf/cudf/core/column/string.py @@ -5533,6 +5533,120 @@ def minhash64( self._column.minhash64(seed, a_column, b_column, width) # type: ignore[arg-type] ) + def minhash_ngrams( + self, ngrams: int, seed: np.uint32, a: ColumnLike, b: ColumnLike + ) -> SeriesOrIndex: + """ + Compute the minhash of a list column of strings. + + This uses the MurmurHash3_x86_32 algorithm for the hash function. + + Calculation uses the formula (hv * a + b) % mersenne_prime + where hv is the hash of a ngrams of strings within each row, + a and b are provided values and mersenne_prime is 2^61-1. + + Parameters + ---------- + ngrams : int + Number of strings to hash within each row. + seed : uint32 + The seed used for the hash algorithm. + a : ColumnLike + Values for minhash calculation. + Must be of type uint32. + b : ColumnLike + Values for minhash calculation. + Must be of type uint32. + + Examples + -------- + >>> import cudf + >>> import numpy as np + >>> s = cudf.Series([['this', 'is', 'my'], ['favorite', 'book']]) + >>> a = cudf.Series([1, 2, 3], dtype=np.uint32) + >>> b = cudf.Series([4, 5, 6], dtype=np.uint32) + >>> s.str.minhash_ngrams(ngrams=2, seed=0, a=a, b=b) + 0 [416367551, 832735099, 1249102647] + 1 [1906668704, 3813337405, 1425038810] + dtype: list + """ + a_column = column.as_column(a) + if a_column.dtype != np.uint32: + raise ValueError( + f"Expecting a Series with dtype uint32, got {type(a)}" + ) + b_column = column.as_column(b) + if b_column.dtype != np.uint32: + raise ValueError( + f"Expecting a Series with dtype uint32, got {type(b)}" + ) + plc_column = plc.nvtext.minhash.minhash_ngrams( + self._column.to_pylibcudf(mode="read"), + ngrams, + seed, + a._column.to_pylibcudf(mode="read"), + b._column.to_pylibcudf(mode="read"), + ) + result = ColumnBase.from_pylibcudf(plc_column) + return self._return_or_inplace(result) + + def minhash64_ngrams( + self, ngrams: int, seed: np.uint64, a: ColumnLike, b: ColumnLike + ) -> SeriesOrIndex: + """ + Compute the minhash of a list column of strings. + + This uses the MurmurHash3_x64_128 algorithm for the hash function. + + Calculation uses the formula (hv * a + b) % mersenne_prime + where hv is the hash of a ngrams of strings within each row, + a and b are provided values and mersenne_prime is 2^61-1. + + Parameters + ---------- + ngrams : int + Number of strings to hash within each row. + seed : uint64 + The seed used for the hash algorithm. + a : ColumnLike + Values for minhash calculation. + Must be of type uint64. + b : ColumnLike + Values for minhash calculation. + Must be of type uint64. + + Examples + -------- + >>> import cudf + >>> import numpy as np + >>> s = cudf.Series([['this', 'is', 'my'], ['favorite', 'book']]) + >>> a = cudf.Series([2, 3], dtype=np.uint64) + >>> b = cudf.Series([5, 6], dtype=np.uint64) + >>> s.str.minhash64_ngrams(ngrams=2, seed=0, a=a, b=b) + 0 [1304293339825194559, 1956440009737791829] + 1 [472203876238918632, 1861227318965224922] + dtype: list + """ + a_column = column.as_column(a) + if a_column.dtype != np.uint64: + raise ValueError( + f"Expecting a Series with dtype uint64, got {type(a)}" + ) + b_column = column.as_column(b) + if b_column.dtype != np.uint64: + raise ValueError( + f"Expecting a Series with dtype uint64, got {type(b)}" + ) + plc_column = plc.nvtext.minhash.minhash64_ngrams( + self._column.to_pylibcudf(mode="read"), + ngrams, + seed, + a._column.to_pylibcudf(mode="read"), + b._column.to_pylibcudf(mode="read"), + ) + result = ColumnBase.from_pylibcudf(plc_column) + return self._return_or_inplace(result) + def jaccard_index(self, input: cudf.Series, width: int) -> SeriesOrIndex: """ Compute the Jaccard index between this column and the given diff --git a/python/cudf/cudf/tests/text/test_text_methods.py b/python/cudf/cudf/tests/text/test_text_methods.py index dc45827d2e8..47b41bd1e39 100644 --- a/python/cudf/cudf/tests/text/test_text_methods.py +++ b/python/cudf/cudf/tests/text/test_text_methods.py @@ -930,6 +930,48 @@ def test_minhash(): strings.str.minhash64(1, a=params, b=params, width=8) +def test_minhash_ngrams(): + strings = cudf.Series( + [["this", "is", "my"], ["favorite", "book", "today"]] + ) + + params = cudf.Series([1, 2, 3], dtype=np.uint32) + expected = cudf.Series( + [ + cudf.Series([416367548, 832735096, 1249102644], dtype=np.uint32), + cudf.Series([1408797893, 2817595786, 4226393679], dtype=np.uint32), + ] + ) + actual = strings.str.minhash_ngrams(ngrams=2, seed=0, a=params, b=params) + assert_eq(expected, actual) + + params = cudf.Series([1, 2, 3], dtype=np.uint64) + expected = cudf.Series( + [ + cudf.Series( + [652146669912597278, 1304293339825194556, 1956440009737791826], + dtype=np.uint64, + ), + cudf.Series( + [1776622609581023632, 1247402209948353305, 718181810315682986], + dtype=np.uint64, + ), + ] + ) + actual = strings.str.minhash64_ngrams(ngrams=2, seed=0, a=params, b=params) + assert_eq(expected, actual) + + # test wrong input types + with pytest.raises(ValueError): + strings.str.minhash_ngrams(ngrams=7, seed=1, a="a", b="b") + with pytest.raises(ValueError): + params = cudf.Series([0, 1, 2], dtype=np.int32) + strings.str.minhash_ngrams(ngrams=6, seed=1, a=params, b=params) + with pytest.raises(ValueError): + params = cudf.Series([0, 1, 2], dtype=np.uint32) + strings.str.minhash64_ngrams(ngrams=8, seed=1, a=params, b=params) + + def test_jaccard_index(): str1 = cudf.Series(["the brown dog", "jumped about"]) str2 = cudf.Series(["the black cat", "jumped around"]) diff --git a/python/pylibcudf/pylibcudf/libcudf/nvtext/minhash.pxd b/python/pylibcudf/pylibcudf/libcudf/nvtext/minhash.pxd index 9d1e8cba425..bfbb99e8eb0 100644 --- a/python/pylibcudf/pylibcudf/libcudf/nvtext/minhash.pxd +++ b/python/pylibcudf/pylibcudf/libcudf/nvtext/minhash.pxd @@ -1,4 +1,4 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. from libc.stdint cimport uint32_t, uint64_t from libcpp.memory cimport unique_ptr from pylibcudf.exception_handler cimport libcudf_exception_handler @@ -25,3 +25,19 @@ cdef extern from "nvtext/minhash.hpp" namespace "nvtext" nogil: const column_view &b, const size_type width, ) except + + + cdef unique_ptr[column] minhash_ngrams( + const column_view &strings, + const size_type ngrams, + const uint32_t seed, + const column_view &a, + const column_view &b, + ) except + + + cdef unique_ptr[column] minhash64_ngrams( + const column_view &strings, + const size_type ngrams, + const uint64_t seed, + const column_view &a, + const column_view &b, + ) except + diff --git a/python/pylibcudf/pylibcudf/nvtext/minhash.pxd b/python/pylibcudf/pylibcudf/nvtext/minhash.pxd index 0af53748cdc..f1e099ca7da 100644 --- a/python/pylibcudf/pylibcudf/nvtext/minhash.pxd +++ b/python/pylibcudf/pylibcudf/nvtext/minhash.pxd @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from libc.stdint cimport uint32_t, uint64_t from pylibcudf.column cimport Column @@ -24,3 +24,19 @@ cpdef Column minhash64( Column b, size_type width ) + +cpdef Column minhash_ngrams( + Column input, + size_type width, + uint32_t seed, + Column a, + Column b +) + +cpdef Column minhash64_ngrams( + Column input, + size_type width, + uint64_t seed, + Column a, + Column b +) diff --git a/python/pylibcudf/pylibcudf/nvtext/minhash.pyi b/python/pylibcudf/pylibcudf/nvtext/minhash.pyi index 5d88cfbbea0..bb50a150798 100644 --- a/python/pylibcudf/pylibcudf/nvtext/minhash.pyi +++ b/python/pylibcudf/pylibcudf/nvtext/minhash.pyi @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from pylibcudf.column import Column @@ -8,3 +8,9 @@ def minhash( def minhash64( input: Column, seed: int, a: Column, b: Column, width: int ) -> Column: ... +def minhash_ngrams( + input: Column, ngrams: int, seed: int, a: Column, b: Column +) -> Column: ... +def minhash64_ngrams( + input: Column, ngrams: int, seed: int, a: Column, b: Column +) -> Column: ... diff --git a/python/pylibcudf/pylibcudf/nvtext/minhash.pyx b/python/pylibcudf/pylibcudf/nvtext/minhash.pyx index 84811cda867..cdc4a4f3ac8 100644 --- a/python/pylibcudf/pylibcudf/nvtext/minhash.pyx +++ b/python/pylibcudf/pylibcudf/nvtext/minhash.pyx @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. from libc.stdint cimport uint32_t, uint64_t from libcpp.memory cimport unique_ptr @@ -8,12 +8,16 @@ from pylibcudf.libcudf.column.column cimport column from pylibcudf.libcudf.nvtext.minhash cimport ( minhash as cpp_minhash, minhash64 as cpp_minhash64, + minhash_ngrams as cpp_minhash_ngrams, + minhash64_ngrams as cpp_minhash64_ngrams, ) from pylibcudf.libcudf.types cimport size_type __all__ = [ "minhash", "minhash64", + "minhash_ngrams", + "minhash64_ngrams", ] cpdef Column minhash( @@ -103,3 +107,93 @@ cpdef Column minhash64( ) return Column.from_libcudf(move(c_result)) + +cpdef Column minhash_ngrams( + Column input, + size_type ngrams, + uint32_t seed, + Column a, + Column b +): + """ + Returns the minhash values for each input row of strings. + This function uses MurmurHash3_x86_32 for the hash algorithm. + + For details, see :cpp:func:`minhash_ngrams`. + + Parameters + ---------- + input : Column + List column of strings to compute minhash + ngrams : size_type + Number of consecutive strings to hash in each row + seed : uint32_t + Seed used for the hash function + a : Column + 1st parameter value used for the minhash algorithm. + b : Column + 2nd parameter value used for the minhash algorithm. + + Returns + ------- + Column + List column of minhash values for each row per + value in columns a and b. + """ + cdef unique_ptr[column] c_result + + with nogil: + c_result = cpp_minhash_ngrams( + input.view(), + ngrams, + seed, + a.view(), + b.view() + ) + + return Column.from_libcudf(move(c_result)) + +cpdef Column minhash64_ngrams( + Column input, + size_type ngrams, + uint64_t seed, + Column a, + Column b +): + """ + Returns the minhash values for each input row of strings. + This function uses MurmurHash3_x64_128 for the hash algorithm. + + For details, see :cpp:func:`minhash64_ngrams`. + + Parameters + ---------- + input : Column + Strings column to compute minhash + ngrams : size_type + Number of consecutive strings to hash in each row + seed : uint64_t + Seed used for the hash function + a : Column + 1st parameter value used for the minhash algorithm. + b : Column + 2nd parameter value used for the minhash algorithm. + + Returns + ------- + Column + List column of minhash values for each row per + value in columns a and b. + """ + cdef unique_ptr[column] c_result + + with nogil: + c_result = cpp_minhash64_ngrams( + input.view(), + ngrams, + seed, + a.view(), + b.view() + ) + + return Column.from_libcudf(move(c_result)) diff --git a/python/pylibcudf/pylibcudf/tests/test_nvtext_minhash.py b/python/pylibcudf/pylibcudf/tests/test_nvtext_minhash.py index ad7a6f7a762..ff8545f0617 100644 --- a/python/pylibcudf/pylibcudf/tests/test_nvtext_minhash.py +++ b/python/pylibcudf/pylibcudf/tests/test_nvtext_minhash.py @@ -1,4 +1,4 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. +# Copyright (c) 2024-2025, NVIDIA CORPORATION. import pyarrow as pa import pytest @@ -33,3 +33,49 @@ def test_minhash(minhash_input_data, width): assert pa_result.type == pa.list_( pa.field("element", seed_type, nullable=False) ) + + +@pytest.fixture(scope="module", params=[pa.uint32(), pa.uint64()]) +def minhash_ngrams_input_data(request): + input_arr = pa.array( + [ + ["foo", "bar", "foo foo", "bar bar", "foo bar", "bar foo"], + [ + "one", + "two", + "three", + "four", + "five", + "six", + "seven", + "eight", + "nine", + "ten", + "eleven", + ], + ] + ) + ab = pa.array([2, 3, 4, 5], request.param) + return input_arr, ab, request.param + + +@pytest.mark.parametrize("ngrams", [5, 10]) +def test_minhash_ngrams(minhash_ngrams_input_data, ngrams): + input_arr, ab, seed_type = minhash_ngrams_input_data + minhash_func = ( + plc.nvtext.minhash.minhash_ngrams + if seed_type == pa.uint32() + else plc.nvtext.minhash.minhash64_ngrams + ) + result = minhash_func( + plc.interop.from_arrow(input_arr), + ngrams, + 0, + plc.interop.from_arrow(ab), + plc.interop.from_arrow(ab), + ) + pa_result = plc.interop.to_arrow(result) + assert all(len(got) == len(ab) for got, s in zip(pa_result, input_arr)) + assert pa_result.type == pa.list_( + pa.field("element", seed_type, nullable=False) + ) From cf8938bc6b11de35337f6d4a04c73559420f3f4b Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Thu, 27 Feb 2025 16:42:46 -0500 Subject: [PATCH 108/129] Add a list of expected failures to narwhals tests (#18097) ## Description Adds an xfail list to the narwhals tests we run using cudf. Note: We can update/replace the dict when running Narwhals with cudf.pandas. xref #18031 ## Checklist - [ ] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [ ] New or existing tests cover these changes. - [ ] The documentation is up to date with these changes. --------- Co-authored-by: Vyas Ramasubramani Co-authored-by: GALI PREM SAGAR Co-authored-by: Bradley Dice --- ci/test_narwhals.sh | 1 + docs/cudf/source/conf.py | 1 + python/cudf/cudf/testing/__init__.py | 3 ++- .../cudf/cudf/testing/narwhals_test_plugin.py | 25 +++++++++++++++++++ 4 files changed, 29 insertions(+), 1 deletion(-) create mode 100644 python/cudf/cudf/testing/narwhals_test_plugin.py diff --git a/ci/test_narwhals.sh b/ci/test_narwhals.sh index 4a32ff0b0fd..28eceff2f80 100755 --- a/ci/test_narwhals.sh +++ b/ci/test_narwhals.sh @@ -26,6 +26,7 @@ rapids-logger "Run narwhals tests for cuDF" python -m pytest \ --cache-clear \ --junitxml="${RAPIDS_TESTS_DIR}/junit-cudf-narwhals.xml" \ + -p cudf.testing.narwhals_test_plugin \ --numprocesses=8 \ --dist=worksteal \ --constructors=cudf diff --git a/docs/cudf/source/conf.py b/docs/cudf/source/conf.py index 8eea644363b..92b37c4b3f2 100644 --- a/docs/cudf/source/conf.py +++ b/docs/cudf/source/conf.py @@ -207,6 +207,7 @@ def clean_all_xml_files(path): exclude_patterns = [ "venv", "**/includes/**", + "narwhals_test_plugin", ] # The name of the Pygments (syntax highlighting) style to use. diff --git a/python/cudf/cudf/testing/__init__.py b/python/cudf/cudf/testing/__init__.py index 4e92b43b9f9..a4afa54f754 100644 --- a/python/cudf/cudf/testing/__init__.py +++ b/python/cudf/cudf/testing/__init__.py @@ -1,5 +1,6 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. +from cudf.testing import narwhals_test_plugin from cudf.testing.testing import ( assert_eq, assert_frame_equal, diff --git a/python/cudf/cudf/testing/narwhals_test_plugin.py b/python/cudf/cudf/testing/narwhals_test_plugin.py new file mode 100644 index 00000000000..d794bd0120a --- /dev/null +++ b/python/cudf/cudf/testing/narwhals_test_plugin.py @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. +# SPDX-License-Identifier: Apache-2.0 + +"""Plugin for running narwhals test suite with cudf.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from collections.abc import Mapping + +EXPECTED_FAILURES: Mapping[str, str] = { + "tests/frame/select_test.py::test_select_duplicates[cudf]": "cuDF doesn't support having multiple columns with same names", +} + + +def pytest_collection_modifyitems(session, config, items) -> None: + """Mark known failing tests.""" + import pytest + + for item in items: + if item.nodeid in EXPECTED_FAILURES: + exp_val = EXPECTED_FAILURES[item.nodeid] + item.add_marker(pytest.mark.xfail(reason=exp_val)) From 83a29ce1e99221436e6d7a8ac06d87ee0982bf20 Mon Sep 17 00:00:00 2001 From: Lawrence Mitchell Date: Fri, 28 Feb 2025 14:41:49 +0000 Subject: [PATCH 109/129] Minor improvements in arrow interop (#18053) When ingesting data from an arrow stream, if the stream contains only a single chunk we can avoid the concatenation. Additionally, explicitly raise exceptions if the arrow-side column length would exceed cudf column size limits. Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - Devavret Makkar (https://github.com/devavret) - David Wendt (https://github.com/davidwendt) - Bradley Dice (https://github.com/bdice) - Basit Ayantunde (https://github.com/lamarrr) URL: https://github.com/rapidsai/cudf/pull/18053 --- cpp/include/cudf/interop.hpp | 12 +++++++++++- cpp/src/interop/from_arrow_device.cu | 9 +++++++++ cpp/src/interop/from_arrow_host.cu | 9 +++++++++ cpp/src/interop/from_arrow_stream.cu | 3 ++- 4 files changed, 31 insertions(+), 2 deletions(-) diff --git a/cpp/include/cudf/interop.hpp b/cpp/include/cudf/interop.hpp index 810f0377597..276a1ea77e2 100644 --- a/cpp/include/cudf/interop.hpp +++ b/cpp/include/cudf/interop.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -327,6 +327,8 @@ unique_device_array_t to_arrow_host( * * @throws cudf::data_type_error if the input array is not a struct array. * + * @throws std::overflow_error if the input arrow object exceeds the column size limit. + * * The conversion will not call release on the input Array. * * @param schema `ArrowSchema` pointer to describe the type of the data @@ -367,6 +369,8 @@ std::unique_ptr from_arrow_column( * * @throws std::invalid_argument if the device_type is not `ARROW_DEVICE_CPU` * + * @throws std::overflow_error if the input arrow object exceeds the column size limit. + * * @throws cudf::data_type_error if the input array is not a struct array, * non-struct arrays should be passed to `from_arrow_host_column` instead. * @@ -411,6 +415,8 @@ std::unique_ptr
from_arrow_stream( * * @throws cudf::data_type_error if input arrow data type is not supported in cudf. * + * @throws std::overflow_error if the input arrow object exceeds the column size limit. + * * The conversion will not call release on the input Array. * * @param schema `ArrowSchema` pointer to describe the type of the data @@ -483,6 +489,8 @@ using unique_table_view_t = * * @throws cudf::data_type_error if the input arrow data type is not supported. * + * @throws std::overflow_error if the input arrow object exceeds the column size limit. + * * Each child of the input struct will be the columns of the resulting table_view. * * @note The custom deleter used for the unique_ptr to the table_view maintains ownership @@ -528,6 +536,8 @@ using unique_column_view_t = * * @throws cudf::data_type_error input arrow data type is not supported. * + * @throws std::overflow_error if the input arrow object exceeds the column size limit. + * * @note The custom deleter used for the unique_ptr to the table_view maintains ownership * over any memory which is allocated, such as converting boolean columns from the bitmap * used by Arrow to the 1-byte per value for cudf. diff --git a/cpp/src/interop/from_arrow_device.cu b/cpp/src/interop/from_arrow_device.cu index 29c4dfd35ac..836da2987e2 100644 --- a/cpp/src/interop/from_arrow_device.cu +++ b/cpp/src/interop/from_arrow_device.cu @@ -40,6 +40,10 @@ #include #include +#include +#include +#include + namespace cudf { namespace detail { @@ -317,6 +321,11 @@ dispatch_tuple_t get_column(ArrowSchemaView* schema, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) { + CUDF_EXPECTS( + input->length <= static_cast(std::numeric_limits::max()), + "Total number of rows in Arrow column exceeds the column size limit.", + std::overflow_error); + return type.id() != type_id::EMPTY ? std::move(type_dispatcher( type, dispatch_from_arrow_device{}, schema, input, type, skip_mask, stream, mr)) diff --git a/cpp/src/interop/from_arrow_host.cu b/cpp/src/interop/from_arrow_host.cu index ea5487a2960..0be1557faaf 100644 --- a/cpp/src/interop/from_arrow_host.cu +++ b/cpp/src/interop/from_arrow_host.cu @@ -43,6 +43,10 @@ #include #include +#include +#include +#include + namespace cudf { namespace detail { @@ -381,6 +385,11 @@ std::unique_ptr get_column_copy(ArrowSchemaView* schema, rmm::cuda_stream_view stream, rmm::device_async_resource_ref mr) { + CUDF_EXPECTS( + input->length <= static_cast(std::numeric_limits::max()), + "Total number of rows in Arrow column exceeds the column size limit.", + std::overflow_error); + return type.id() != type_id::EMPTY ? std::move(type_dispatcher( type, dispatch_copy_from_arrow_host{stream, mr}, schema, input, type, skip_mask)) diff --git a/cpp/src/interop/from_arrow_stream.cu b/cpp/src/interop/from_arrow_stream.cu index deff62be576..ce1db96ca43 100644 --- a/cpp/src/interop/from_arrow_stream.cu +++ b/cpp/src/interop/from_arrow_stream.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -121,6 +121,7 @@ std::unique_ptr
from_arrow_stream(ArrowArrayStream* input, schema.release(&schema); + if (chunks.size() == 1) { return std::move(chunks[0]); } auto chunk_views = std::vector{}; chunk_views.reserve(chunks.size()); std::transform( From 09ebf31011f27d343c32ef406b90c3ecc12b0107 Mon Sep 17 00:00:00 2001 From: Vyas Ramasubramani Date: Fri, 28 Feb 2025 16:36:46 -0800 Subject: [PATCH 110/129] Use protocol for dlpack instead of deprecated function (#18134) This PR adapts cudf's dlpack tests for compatibility with cupy 13.4, which was just released yesterday on PyPI and containers https://github.com/cupy/cupy/pull/8722 that breaks the legacy toDlpack functionality. --- python/cudf/cudf/core/df_protocol.py | 2 +- python/cudf/cudf/core/subword_tokenizer.py | 2 +- python/cudf/cudf/tests/test_dlpack.py | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/python/cudf/cudf/core/df_protocol.py b/python/cudf/cudf/core/df_protocol.py index cc9f39d70ef..5f2dfe98a3e 100644 --- a/python/cudf/cudf/core/df_protocol.py +++ b/python/cudf/cudf/core/df_protocol.py @@ -105,7 +105,7 @@ def __dlpack__(self): # DLPack not implemented in NumPy yet, so leave it out here. try: cuda_array = as_cuda_array(self._buf).view(self._dtype) - return cp.asarray(cuda_array).toDlpack() + return cp.asarray(cuda_array).__dlpack__() except ValueError: raise TypeError(f"dtype {self._dtype} unsupported by `dlpack`") diff --git a/python/cudf/cudf/core/subword_tokenizer.py b/python/cudf/cudf/core/subword_tokenizer.py index 50d1a11c39b..24e6aa40de0 100644 --- a/python/cudf/cudf/core/subword_tokenizer.py +++ b/python/cudf/cudf/core/subword_tokenizer.py @@ -19,7 +19,7 @@ def _cast_to_appropriate_type(ar, cast_type): elif cast_type == "tf": from tensorflow.experimental.dlpack import from_dlpack - return from_dlpack(ar.astype("int32").toDlpack()) + return from_dlpack(ar.astype("int32").__dlpack__()) class SubwordTokenizer: diff --git a/python/cudf/cudf/tests/test_dlpack.py b/python/cudf/cudf/tests/test_dlpack.py index 20c24bd7564..187a5524e8e 100644 --- a/python/cudf/cudf/tests/test_dlpack.py +++ b/python/cudf/cudf/tests/test_dlpack.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019-2024, NVIDIA CORPORATION. +# Copyright (c) 2019-2025, NVIDIA CORPORATION. import itertools from contextlib import ExitStack as does_not_raise @@ -140,7 +140,7 @@ def test_to_dlpack_cupy_2d(data_2d): def test_from_dlpack_cupy_1d(data_1d): cupy_array = cupy.array(data_1d) cupy_host_array = cupy_array.get() - dlt = cupy_array.toDlpack() + dlt = cupy_array.__dlpack__() gs = cudf.from_dlpack(dlt) cudf_host_array = gs.to_numpy(na_value=np.nan) @@ -151,7 +151,7 @@ def test_from_dlpack_cupy_1d(data_1d): def test_from_dlpack_cupy_2d(data_2d): cupy_array = cupy.array(data_2d, order="F") cupy_host_array = cupy_array.get().flatten() - dlt = cupy_array.toDlpack() + dlt = cupy_array.__dlpack__() gdf = cudf.from_dlpack(dlt) cudf_host_array = np.array(gdf.to_pandas()).flatten() From 0cf66982df885513921372f0dcbcc32b6d4cd243 Mon Sep 17 00:00:00 2001 From: Tianyu Liu Date: Mon, 3 Mar 2025 12:03:39 -0500 Subject: [PATCH 111/129] Update calls to KvikIO's config setter (#18144) ## Description KvikIO has changed the function names of the config setters to improve clarity (https://github.com/rapidsai/kvikio/pull/644). This PR updates the setter calls in cuDF accordingly. ## Checklist - [x] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [x] New or existing tests cover these changes. - [x] The documentation is up to date with these changes. --- cpp/src/io/utilities/config_utils.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cpp/src/io/utilities/config_utils.cpp b/cpp/src/io/utilities/config_utils.cpp index 46816604918..fa6f04eed73 100644 --- a/cpp/src/io/utilities/config_utils.cpp +++ b/cpp/src/io/utilities/config_utils.cpp @@ -36,10 +36,10 @@ void set_up_kvikio() cudaFree(nullptr); auto const compat_mode = kvikio::getenv_or("KVIKIO_COMPAT_MODE", kvikio::CompatMode::ON); - kvikio::defaults::compat_mode_reset(compat_mode); + kvikio::defaults::set_compat_mode(compat_mode); auto const nthreads = getenv_or("KVIKIO_NTHREADS", 4u); - kvikio::defaults::thread_pool_nthreads_reset(nthreads); + kvikio::defaults::set_thread_pool_nthreads(nthreads); }); } From 1c0ea5e7f7968fbeb6852a533df30795ad754b2b Mon Sep 17 00:00:00 2001 From: Vukasin Milovanovic Date: Mon, 3 Mar 2025 11:18:37 -0800 Subject: [PATCH 112/129] Reduce memory use when writing tables with very short columns to ORC (#18136) Closes #18059 To avoid estimating the maximum compressed size for each actual block in the file, ORC writer uses the estimate for the (uncompressed) block size limit, which defaults to 256KB. However, when we write many small blocks, this compressed block size estimate is much larger than what is needed, leading to high memory use for wide/short tables. This PR adds logic to take the actual block size into account, and to use the size of the actual largest block in the file, not the largest possible block. This changes the memory usage by orders of magnitude in some tests. --------- Co-authored-by: Bradley Dice --- cpp/src/io/orc/writer_impl.cu | 20 +++++++++++++++++++- cpp/src/utilities/host_memory.cpp | 1 + cpp/tests/CMakeLists.txt | 4 ++-- 3 files changed, 22 insertions(+), 3 deletions(-) diff --git a/cpp/src/io/orc/writer_impl.cu b/cpp/src/io/orc/writer_impl.cu index 3a20ffbce19..217aff48d5e 100644 --- a/cpp/src/io/orc/writer_impl.cu +++ b/cpp/src/io/orc/writer_impl.cu @@ -2226,6 +2226,22 @@ stripe_dictionaries build_dictionaries(orc_table_view& orc_table, std::move(dict_order_owner)}; } +[[nodiscard]] uint32_t find_largest_stream_size(device_2dspan ss, + rmm::cuda_stream_view stream) +{ + auto const longest_stream = thrust::max_element( + rmm::exec_policy(stream), + ss.data(), + ss.data() + ss.count(), + cuda::proclaim_return_type([] __device__(auto const& lhs, auto const& rhs) { + return lhs.stream_size < rhs.stream_size; + })); + + auto const h_longest_stream = cudf::detail::make_host_vector_sync( + device_span{longest_stream, 1}, stream); + return h_longest_stream[0].stream_size; +} + /** * @brief Perform the processing steps needed to convert the input table into the output ORC data * for writing, such as compression and ORC encoding. @@ -2319,7 +2335,9 @@ auto convert_table_to_orc_data(table_view const& input, size_t compressed_bfr_size = 0; size_t num_compressed_blocks = 0; - auto const max_compressed_block_size = max_compressed_size(compression, compression_blocksize); + auto const largest_stream_size = find_largest_stream_size(strm_descs, stream); + auto const max_compressed_block_size = + max_compressed_size(compression, std::min(largest_stream_size, compression_blocksize)); auto const padded_max_compressed_block_size = util::round_up_unsafe(max_compressed_block_size, block_align); auto const padded_block_header_size = diff --git a/cpp/src/utilities/host_memory.cpp b/cpp/src/utilities/host_memory.cpp index 94d27d976c3..e41d772a479 100644 --- a/cpp/src/utilities/host_memory.cpp +++ b/cpp/src/utilities/host_memory.cpp @@ -29,6 +29,7 @@ namespace cudf { namespace { + class fixed_pinned_pool_memory_resource { using upstream_mr = rmm::mr::pinned_host_memory_resource; using host_pooled_mr = rmm::mr::pool_memory_resource; diff --git a/cpp/tests/CMakeLists.txt b/cpp/tests/CMakeLists.txt index cfc6a0dc425..e3ca8b70b87 100644 --- a/cpp/tests/CMakeLists.txt +++ b/cpp/tests/CMakeLists.txt @@ -309,7 +309,7 @@ ConfigureTest( ConfigureTest( ORC_TEST io/orc_chunked_reader_test.cu io/orc_test.cpp GPUS 1 - PERCENT 30 + PERCENT 100 ) ConfigureTest( PARQUET_TEST @@ -340,7 +340,7 @@ ConfigureTest(JSON_TREE_CSR io/json/json_tree_csr.cu) ConfigureTest( DATA_CHUNK_SOURCE_TEST io/text/data_chunk_source_test.cpp GPUS 1 - PERCENT 30 + PERCENT 100 ) target_link_libraries(DATA_CHUNK_SOURCE_TEST PRIVATE ZLIB::ZLIB) ConfigureTest(LOGICAL_STACK_TEST io/fst/logical_stack_test.cu) From 34235f4ebacd5982aad4c42d6886706761ac862c Mon Sep 17 00:00:00 2001 From: Matthew Murray <41342305+Matt711@users.noreply.github.com> Date: Mon, 3 Mar 2025 17:06:30 -0500 Subject: [PATCH 113/129] Use protocol for dlpack instead of deprecated function in cupy notebook (#18147) Follow up to #18134 --- docs/cudf/source/user_guide/cupy-interop.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/cudf/source/user_guide/cupy-interop.ipynb b/docs/cudf/source/user_guide/cupy-interop.ipynb index 112f0bcfca6..93e62d90c0f 100644 --- a/docs/cudf/source/user_guide/cupy-interop.ipynb +++ b/docs/cudf/source/user_guide/cupy-interop.ipynb @@ -566,7 +566,7 @@ "%%timeit\n", "\n", "fortran_arr = cp.asfortranarray(reshaped_arr)\n", - "reshaped_df = cudf.from_dlpack(fortran_arr.toDlpack())" + "reshaped_df = cudf.from_dlpack(fortran_arr.__dlpack__())" ] }, { @@ -1418,7 +1418,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.12" + "version": "3.12.9" } }, "nbformat": 4, From b6a6d390f92080481606e91f40450cc4e140fa97 Mon Sep 17 00:00:00 2001 From: Vyas Ramasubramani Date: Mon, 3 Mar 2025 14:22:56 -0800 Subject: [PATCH 114/129] Skip failing test (#18146) This test is failing in multiple places right now, such as [this run](https://github.com/rapidsai/cudf/actions/runs/13595690128/job/38014725800) on https://github.com/rapidsai/cudf/pull/18133 and [this run](https://github.com/rapidsai/cudf/actions/runs/13636334843/job/38118996773?pr=18136) on https://github.com/rapidsai/cudf/pull/18136. Let's skip it until we can debug why so that we unblock other CI. --------- Co-authored-by: Peter Andreas Entschev --- ci/run_cudf_polars_pytests.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ci/run_cudf_polars_pytests.sh b/ci/run_cudf_polars_pytests.sh index e881055e9e3..5a1d5f56bf0 100755 --- a/ci/run_cudf_polars_pytests.sh +++ b/ci/run_cudf_polars_pytests.sh @@ -17,5 +17,5 @@ python -m pytest --cache-clear "$@" tests --executor dask-experimental # Test the "dask-experimental" executor with Distributed cluster # Not all tests pass yet, deselecting by name those that are failing. python -m pytest --cache-clear "$@" tests --executor dask-experimental --dask-cluster \ - -k "not test_groupby_maintain_order_random and not test_scan_csv_multi and not test_select_literal_series" \ - --cov-fail-under=89 # Override coverage, Distributed cluster coverage not yet 100% + -k "not test_groupby_maintain_order_random and not test_scan_csv_multi and not test_select_literal_series and not test_can_convert_lists and not test_executor_basics and not test_replace_literal and not test_hconcat_different_heights and not test_join and not test_dataframescan and not test_strip_chars" \ + --cov-fail-under=80 # Override coverage, Distributed cluster coverage not yet 100% From 93d98af8450d466705062ca23f58f6082fca3e98 Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Mon, 3 Mar 2025 19:02:23 -0500 Subject: [PATCH 115/129] Optimization improvement for substr in cudf::string_view (#18062) Slight optimization improvement sets the character count in the `cudf::string_view` produced by `cudf::string_view::substr` when the number of output characters is known. This can save redundant character counting in downstream usage of the new string. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Devavret Makkar (https://github.com/devavret) - Shruti Shivakumar (https://github.com/shrshi) URL: https://github.com/rapidsai/cudf/pull/18062 --- cpp/include/cudf/strings/string_view.cuh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/cpp/include/cudf/strings/string_view.cuh b/cpp/include/cudf/strings/string_view.cuh index b91748cfc7d..15539c50da9 100644 --- a/cpp/include/cudf/strings/string_view.cuh +++ b/cpp/include/cudf/strings/string_view.cuh @@ -443,10 +443,12 @@ __device__ inline size_type string_view::rfind(char_utf8 chr, size_type pos, siz __device__ inline string_view string_view::substr(size_type pos, size_type count) const { if (pos < 0 || pos >= length()) { return string_view{}; } - auto const itr = begin() + pos; - auto const spos = itr.byte_offset(); - auto const epos = count >= 0 ? (itr + count).byte_offset() : size_bytes(); - return {data() + spos, epos - spos}; + auto const spos = begin() + pos; + auto const epos = count >= 0 ? (spos + count) : const_iterator{*this, _length, size_bytes()}; + auto ss = string_view{data() + spos.byte_offset(), epos.byte_offset() - spos.byte_offset()}; + // this potentially saves redundant character counting downstream + if (_length != UNKNOWN_STRING_LENGTH) { ss._length = epos.position() - spos.position(); } + return ss; } __device__ inline size_type string_view::character_offset(size_type bytepos) const From 08f536a602d288f3c31abf7f2a22a8538b13f62d Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Mon, 3 Mar 2025 19:33:19 -0800 Subject: [PATCH 116/129] Preserve DataFrame.column subclass and type during binop (#18113) closes https://github.com/rapidsai/cudf/issues/11148 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18113 --- python/cudf/cudf/core/dataframe.py | 41 ++++++++++++++++-------- python/cudf/cudf/core/indexed_frame.py | 10 ++---- python/cudf/cudf/core/series.py | 15 ++++----- python/cudf/cudf/tests/test_dataframe.py | 15 +++++++++ 4 files changed, 53 insertions(+), 28 deletions(-) diff --git a/python/cudf/cudf/core/dataframe.py b/python/cudf/cudf/core/dataframe.py index 69db055fe87..3cc42dbe982 100644 --- a/python/cudf/cudf/core/dataframe.py +++ b/python/cudf/cudf/core/dataframe.py @@ -2055,18 +2055,28 @@ def _make_operands_and_index_for_binop( dict[str | None, tuple[ColumnBase, Any, bool, Any]] | NotImplementedType, BaseIndex | None, - bool, + dict[str, Any], ]: lhs, rhs = self._data, other index = self.index fill_requires_key = False left_default: Any = False equal_columns = False - can_use_self_column_name = True + ca_attributes: dict[str, Any] = {} + + def _fill_same_ca_attributes( + attrs: dict[str, Any], ca: ColumnAccessor + ) -> dict[str, Any]: + attrs["rangeindex"] = ca.rangeindex + attrs["multiindex"] = ca.multiindex + attrs["label_dtype"] = ca.label_dtype + attrs["level_names"] = ca.level_names + return attrs if _is_scalar_or_zero_d_array(other): rhs = {name: other for name in self._data} equal_columns = True + ca_attributes = _fill_same_ca_attributes(ca_attributes, self._data) elif isinstance(other, Series): if ( not (self_pd_columns := self._data.to_pandas_index).equals( @@ -2085,9 +2095,12 @@ def _make_operands_and_index_for_binop( # NULL!) and the right value (result is NaN). left_default = as_column(np.nan, length=len(self)) equal_columns = other_pd_index.equals(self_pd_columns) - can_use_self_column_name = ( - equal_columns or other_pd_index.names == self_pd_columns.names - ) + if equal_columns: + ca_attributes = _fill_same_ca_attributes( + ca_attributes, self._data + ) + elif other_pd_index.names == self_pd_columns.names: + ca_attributes["level_names"] = self._data.level_names elif isinstance(other, DataFrame): if ( not can_reindex @@ -2110,17 +2123,19 @@ def _make_operands_and_index_for_binop( # the fill value. left_default = fill_value equal_columns = self._column_names == other._column_names - can_use_self_column_name = ( - equal_columns - or self._data._level_names == other._data._level_names - ) + if self._data.to_pandas_index.equals(other._data.to_pandas_index): + ca_attributes = _fill_same_ca_attributes( + ca_attributes, self._data + ) + elif self._data._level_names == other._data._level_names: + ca_attributes["level_names"] = self._data.level_names elif isinstance(other, (dict, abc.Mapping)): # Need to fail early on host mapping types because we ultimately # convert everything to a dict. - return NotImplemented, None, True + return NotImplemented, None, ca_attributes if not isinstance(rhs, (dict, abc.Mapping)): - return NotImplemented, None, True + return NotImplemented, None, ca_attributes operands = { k: ( @@ -2150,8 +2165,8 @@ def _make_operands_and_index_for_binop( raise ValueError("other must be a DataFrame or Series.") sorted_dict = {key: operands[key] for key in column_names_list} - return sorted_dict, index, can_use_self_column_name - return operands, index, can_use_self_column_name + return sorted_dict, index, ca_attributes + return operands, index, ca_attributes @classmethod @_performance_tracking diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index 9d426ad6bf7..8a625dc9225 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -4888,20 +4888,16 @@ def _binaryop( ( operands, out_index, - can_use_self_column_name, + ca_attributes, ) = self._make_operands_and_index_for_binop( other, op, fill_value, reflect, can_reindex ) if operands is NotImplemented: return NotImplemented - - level_names = ( - self._data._level_names if can_use_self_column_name else None - ) return self._from_data( ColumnAccessor( type(self)._colwise_binop(operands, op), - level_names=level_names, + **ca_attributes, ), index=out_index, ) @@ -4917,7 +4913,7 @@ def _make_operands_and_index_for_binop( dict[str | None, tuple[ColumnBase, Any, bool, Any]] | NotImplementedType, cudf.BaseIndex | None, - bool, + dict[str, Any], ]: raise NotImplementedError( f"Binary operations are not supported for {self.__class__}" diff --git a/python/cudf/cudf/core/series.py b/python/cudf/cudf/core/series.py index f6f1b31dc43..d25550553b1 100644 --- a/python/cudf/cudf/core/series.py +++ b/python/cudf/cudf/core/series.py @@ -1531,7 +1531,7 @@ def _make_operands_and_index_for_binop( dict[str | None, tuple[ColumnBase, Any, bool, Any]] | NotImplementedType, BaseIndex | None, - bool, + dict[str, Any], ]: # Specialize binops to align indices. if isinstance(other, Series): @@ -1547,15 +1547,14 @@ def _make_operands_and_index_for_binop( else: lhs = self - try: - can_use_self_column_name = cudf.utils.utils._is_same_name( - self.name, other.name - ) - except AttributeError: - can_use_self_column_name = False + ca_attributes = {} + if hasattr(other, "name") and cudf.utils.utils._is_same_name( + self.name, other.name + ): + ca_attributes["level_names"] = self._data._level_names operands = lhs._make_operands_for_binop(other, fill_value, reflect) - return operands, lhs.index, can_use_self_column_name + return operands, lhs.index, ca_attributes @copy_docstring(CategoricalAccessor) # type: ignore @property diff --git a/python/cudf/cudf/tests/test_dataframe.py b/python/cudf/cudf/tests/test_dataframe.py index 15c11db5a84..d6bbbf601be 100644 --- a/python/cudf/cudf/tests/test_dataframe.py +++ b/python/cudf/cudf/tests/test_dataframe.py @@ -11083,6 +11083,21 @@ def test_dataframe_columns_set_preserve_type(klass): pd.testing.assert_index_equal(result, expected) +@pytest.mark.parametrize( + "expected", + [ + pd.RangeIndex(1, 2, name="a"), + pd.Index([1], dtype=np.int8, name="a"), + pd.MultiIndex.from_arrays([[1]], names=["a"]), + ], +) +@pytest.mark.parametrize("binop", [lambda df: df == df, lambda df: df - 1]) +def test_dataframe_binop_preserves_column_metadata(expected, binop): + df = cudf.DataFrame([1], columns=expected) + result = binop(df).columns + pd.testing.assert_index_equal(result, expected, exact=True) + + @pytest.mark.parametrize( "scalar", [ From 43bbd7f0fcafd0f29db80f9b57913f8c63e74fd9 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Mon, 3 Mar 2025 19:44:05 -0800 Subject: [PATCH 117/129] Remove some unnecessary module imports (#18143) Noticed while working on https://github.com/rapidsai/cudf/pull/18141. Also made some imports more specific to make it easier to see what we need Authors: - Matthew Roeschke (https://github.com/mroeschke) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18143 --- python/cudf/cudf/core/column/methods.py | 2 -- python/cudf/cudf/core/column/string.py | 1 - python/cudf/cudf/core/dataframe.py | 1 - python/cudf/cudf/core/indexed_frame.py | 4 ++-- python/cudf/cudf/core/udf/groupby_utils.py | 8 +++----- python/cudf/cudf/utils/utils.py | 1 - 6 files changed, 5 insertions(+), 12 deletions(-) diff --git a/python/cudf/cudf/core/column/methods.py b/python/cudf/cudf/core/column/methods.py index b42e4419d72..e545bb4bc5e 100644 --- a/python/cudf/cudf/core/column/methods.py +++ b/python/cudf/cudf/core/column/methods.py @@ -5,8 +5,6 @@ from typing import Literal, Union, overload import cudf -import cudf.core.column -import cudf.core.column_accessor from cudf.utils.utils import NotIterable ParentType = Union["cudf.Series", "cudf.core.index.Index"] diff --git a/python/cudf/cudf/core/column/string.py b/python/cudf/cudf/core/column/string.py index b82ec1958fb..97ec41f4c39 100644 --- a/python/cudf/cudf/core/column/string.py +++ b/python/cudf/cudf/core/column/string.py @@ -16,7 +16,6 @@ import pylibcudf as plc import cudf -import cudf.api.types import cudf.core.column.column as column import cudf.core.column.datetime as datetime from cudf.api.types import is_integer, is_scalar, is_string_dtype diff --git a/python/cudf/cudf/core/dataframe.py b/python/cudf/cudf/core/dataframe.py index 3cc42dbe982..f909d72687c 100644 --- a/python/cudf/cudf/core/dataframe.py +++ b/python/cudf/cudf/core/dataframe.py @@ -35,7 +35,6 @@ import pylibcudf as plc import cudf -import cudf.core.common from cudf.api.extensions import no_default from cudf.api.types import ( _is_scalar_or_zero_d_array, diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index 8a625dc9225..2f4ad360d8b 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -26,8 +26,8 @@ import pylibcudf as plc import cudf -import cudf.core import cudf.core.algorithms +import cudf.core.common from cudf.api.extensions import no_default from cudf.api.types import ( _is_non_decimal_numeric_dtype, @@ -3908,7 +3908,7 @@ def _reindex( } result = self.__class__._from_data( - data=cudf.core.column_accessor.ColumnAccessor( + data=ColumnAccessor( cols, multiindex=multiindex, level_names=level_names, diff --git a/python/cudf/cudf/core/udf/groupby_utils.py b/python/cudf/cudf/core/udf/groupby_utils.py index 814d3e9fc85..943b6ebfd1c 100644 --- a/python/cudf/cudf/core/udf/groupby_utils.py +++ b/python/cudf/cudf/core/udf/groupby_utils.py @@ -1,4 +1,4 @@ -# Copyright (c) 2022-2024, NVIDIA CORPORATION. +# Copyright (c) 2022-2025, NVIDIA CORPORATION. import cupy as cp @@ -8,7 +8,7 @@ from numba.cuda.cudadrv.devices import get_context from numba.np import numpy_support -import cudf.core.udf.utils +from cudf.core.column import column_empty from cudf.core.udf.groupby_typing import ( SUPPORTED_GROUPBY_NUMPY_TYPES, Group, @@ -154,9 +154,7 @@ def jit_groupby_apply(offsets, grouped_values, function, *args): offsets = cp.asarray(offsets) ngroups = len(offsets) - 1 - output = cudf.core.column.column_empty( - ngroups, dtype=return_type, for_numba=True - ) + output = column_empty(ngroups, dtype=return_type, for_numba=True) launch_args = [ offsets, output, diff --git a/python/cudf/cudf/utils/utils.py b/python/cudf/cudf/utils/utils.py index 2678a4f8116..601a7a369e8 100644 --- a/python/cudf/cudf/utils/utils.py +++ b/python/cudf/cudf/utils/utils.py @@ -15,7 +15,6 @@ import rmm import cudf -import cudf.api.types from cudf.core import column from cudf.core.buffer import as_buffer from cudf.utils.dtypes import SIZE_TYPE_DTYPE From 3636040c366c0af2a6bd95e9beff167665a45b86 Mon Sep 17 00:00:00 2001 From: Michael Schellenberger Costa Date: Tue, 4 Mar 2025 05:14:19 +0100 Subject: [PATCH 118/129] Replace more deprecated `CUB` functors (#18119) They will be removed in a future CCCL release Authors: - Michael Schellenberger Costa (https://github.com/miscco) - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - David Wendt (https://github.com/davidwendt) - Nghia Truong (https://github.com/ttnghia) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18119 --- cpp/benchmarks/common/generate_input.cu | 2 +- .../cudf/detail/utilities/functional.hpp | 31 +++++++++++++++++++ .../cudf/table/experimental/row_operators.cuh | 7 +++-- cpp/src/binaryop/compiled/binary_ops.cu | 7 +++-- cpp/src/filling/repeat.cu | 5 +-- cpp/src/groupby/sort/group_rank_scan.cu | 5 +-- cpp/src/groupby/sort/group_replace_nulls.cu | 5 +-- cpp/src/groupby/sort/group_scan_util.cuh | 7 +++-- .../sort/group_single_pass_reduction_util.cuh | 8 ++--- cpp/src/io/avro/reader_impl.cu | 7 +++-- cpp/src/io/comp/nvcomp_adapter.cu | 5 +-- cpp/src/io/fst/logical_stack.cuh | 5 +-- cpp/src/io/json/column_tree_construction.cu | 5 +-- cpp/src/io/json/host_tree_algorithms.cu | 5 +-- cpp/src/io/json/json_column.cu | 9 +++--- cpp/src/io/json/json_tree.cu | 7 +++-- cpp/src/io/json/write_json.cu | 4 +-- cpp/src/io/orc/stripe_data.cu | 4 ++- cpp/src/io/orc/stripe_enc.cu | 10 +++--- cpp/src/io/parquet/delta_enc.cuh | 6 ++-- cpp/src/io/parquet/page_string_decode.cu | 5 ++- cpp/src/io/parquet/reader_impl_chunking.cu | 6 ++-- cpp/src/io/parquet/reader_impl_preprocess.cu | 3 +- .../io/statistics/typed_statistics_chunk.cuh | 12 ++++--- cpp/src/io/utilities/data_casting.cu | 3 +- cpp/src/lists/set_operations.cu | 7 +++-- cpp/src/quantiles/tdigest/tdigest.cu | 5 +-- .../quantiles/tdigest/tdigest_aggregation.cu | 11 ++++--- cpp/src/reductions/segmented/simple.cuh | 5 +-- .../rolling/detail/rolling_collect_list.cu | 6 ++-- cpp/src/sort/rank.cu | 7 +++-- cpp/src/strings/split/split.cu | 7 +++-- cpp/src/strings/split/split_re.cu | 5 +-- cpp/src/text/bpe/byte_pair_encoding.cu | 9 ++++-- cpp/src/text/minhash.cu | 3 +- cpp/tests/iterator/iterator_tests.cuh | 9 +++--- 36 files changed, 160 insertions(+), 87 deletions(-) create mode 100644 cpp/include/cudf/detail/utilities/functional.hpp diff --git a/cpp/benchmarks/common/generate_input.cu b/cpp/benchmarks/common/generate_input.cu index 8d6aacd2ef1..f1af62eaa87 100644 --- a/cpp/benchmarks/common/generate_input.cu +++ b/cpp/benchmarks/common/generate_input.cu @@ -580,7 +580,7 @@ std::unique_ptr create_random_utf8_string_column(data_profile cons null_mask.begin(), lengths.begin(), cuda::proclaim_return_type([] __device__(auto) { return 0; }), - thrust::logical_not{}); + cuda::std::logical_not{}); auto valid_lengths = thrust::make_transform_iterator( thrust::make_zip_iterator(thrust::make_tuple(lengths.begin(), null_mask.begin())), valid_or_zero{}); diff --git a/cpp/include/cudf/detail/utilities/functional.hpp b/cpp/include/cudf/detail/utilities/functional.hpp new file mode 100644 index 00000000000..114c69bbe46 --- /dev/null +++ b/cpp/include/cudf/detail/utilities/functional.hpp @@ -0,0 +1,31 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include +#include + +namespace cudf::detail { + +#if CCCL_MAJOR_VERSION >= 3 +using cuda::maximum; +using cuda::minimum; +#else +using thrust::maximum; +using thrust::minimum; +#endif + +} // namespace cudf::detail diff --git a/cpp/include/cudf/table/experimental/row_operators.cuh b/cpp/include/cudf/table/experimental/row_operators.cuh index 8214ea6e83b..6ace930c1fe 100644 --- a/cpp/include/cudf/table/experimental/row_operators.cuh +++ b/cpp/include/cudf/table/experimental/row_operators.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -33,6 +33,7 @@ #include #include +#include #include #include #include @@ -1466,9 +1467,9 @@ class device_row_comparator { auto rvalid = detail::make_validity_iterator(rcol); if (nulls_are_equal == null_equality::UNEQUAL) { if (thrust::any_of( - thrust::seq, lvalid, lvalid + lcol.size(), thrust::logical_not()) or + thrust::seq, lvalid, lvalid + lcol.size(), cuda::std::logical_not()) or thrust::any_of( - thrust::seq, rvalid, rvalid + rcol.size(), thrust::logical_not())) { + thrust::seq, rvalid, rvalid + rcol.size(), cuda::std::logical_not())) { return false; } } else { diff --git a/cpp/src/binaryop/compiled/binary_ops.cu b/cpp/src/binaryop/compiled/binary_ops.cu index 3c558f1e264..70e26ae4285 100644 --- a/cpp/src/binaryop/compiled/binary_ops.cu +++ b/cpp/src/binaryop/compiled/binary_ops.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -241,8 +242,8 @@ struct null_considering_binop { return invalid_str; else if (lhs_valid && rhs_valid) { return (op == binary_operator::NULL_MAX) - ? thrust::maximum()(lhs_value, rhs_value) - : thrust::minimum()(lhs_value, rhs_value); + ? cudf::detail::maximum()(lhs_value, rhs_value) + : cudf::detail::minimum()(lhs_value, rhs_value); } else if (lhs_valid) return lhs_value; else diff --git a/cpp/src/filling/repeat.cu b/cpp/src/filling/repeat.cu index 2e78954d78a..2695288af64 100644 --- a/cpp/src/filling/repeat.cu +++ b/cpp/src/filling/repeat.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -81,7 +82,7 @@ struct count_checker { if (static_cast(std::numeric_limits::max()) > std::numeric_limits::max()) { auto max = thrust::reduce( - rmm::exec_policy(stream), count.begin(), count.end(), 0, thrust::maximum()); + rmm::exec_policy(stream), count.begin(), count.end(), 0, cudf::detail::maximum()); CUDF_EXPECTS(max <= std::numeric_limits::max(), "count exceeds the column size limit", std::overflow_error); diff --git a/cpp/src/groupby/sort/group_rank_scan.cu b/cpp/src/groupby/sort/group_rank_scan.cu index 583357d9090..a0ba81bccb2 100644 --- a/cpp/src/groupby/sort/group_rank_scan.cu +++ b/cpp/src/groupby/sort/group_rank_scan.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -29,6 +29,7 @@ #include #include +#include #include #include #include @@ -146,7 +147,7 @@ std::unique_ptr rank_generator(column_view const& grouped_values, group_labels_begin + group_labels.size(), mutable_rank_begin, mutable_rank_begin, - thrust::equal_to{}, + cuda::std::equal_to{}, scan_op); return ranks; } diff --git a/cpp/src/groupby/sort/group_replace_nulls.cu b/cpp/src/groupby/sort/group_replace_nulls.cu index 088ed05e5eb..f94ae71a23c 100644 --- a/cpp/src/groupby/sort/group_replace_nulls.cu +++ b/cpp/src/groupby/sort/group_replace_nulls.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -23,6 +23,7 @@ #include +#include #include #include #include @@ -55,7 +56,7 @@ std::unique_ptr group_replace_nulls(cudf::column_view const& grouped_val thrust::make_tuple(gather_map.begin(), thrust::make_discard_iterator())); auto func = cudf::detail::replace_policy_functor(); - thrust::equal_to eq; + cuda::std::equal_to eq; if (replace_policy == cudf::replace_policy::PRECEDING) { thrust::inclusive_scan_by_key(rmm::exec_policy(stream), group_labels.begin(), diff --git a/cpp/src/groupby/sort/group_scan_util.cuh b/cpp/src/groupby/sort/group_scan_util.cuh index a90445fabe1..160d0a3b276 100644 --- a/cpp/src/groupby/sort/group_scan_util.cuh +++ b/cpp/src/groupby/sort/group_scan_util.cuh @@ -37,6 +37,7 @@ #include #include +#include #include #include #include @@ -122,7 +123,7 @@ struct group_scan_functor() group_labels.end(), inp_iter, out_iter, - thrust::equal_to{}, + cuda::std::equal_to{}, binop); }; @@ -167,7 +168,7 @@ struct group_scan_functor(0), gather_map.begin(), - thrust::equal_to{}, + cuda::std::equal_to{}, binop_generator.binop()); // diff --git a/cpp/src/groupby/sort/group_single_pass_reduction_util.cuh b/cpp/src/groupby/sort/group_single_pass_reduction_util.cuh index 662c380eff5..9dba468bf14 100644 --- a/cpp/src/groupby/sort/group_single_pass_reduction_util.cuh +++ b/cpp/src/groupby/sort/group_single_pass_reduction_util.cuh @@ -175,7 +175,7 @@ struct group_reduction_functor< inp_iter, thrust::make_discard_iterator(), out_iter, - thrust::equal_to{}, + cuda::std::equal_to{}, binop); }; @@ -201,7 +201,7 @@ struct group_reduction_functor< rmm::device_uvector validity(num_groups, stream); do_reduction(cudf::detail::make_validity_iterator(*d_values_ptr), validity.begin(), - thrust::logical_or{}); + cuda::std::logical_or{}); auto [null_mask, null_count] = cudf::detail::valid_if(validity.begin(), validity.end(), cuda::std::identity{}, stream, mr); @@ -238,7 +238,7 @@ struct group_reduction_functor< inp_iter, thrust::make_discard_iterator(), out_iter, - thrust::equal_to{}, + cuda::std::equal_to{}, binop); }; @@ -254,7 +254,7 @@ struct group_reduction_functor< auto validity = rmm::device_uvector(num_groups, stream); do_reduction(cudf::detail::make_validity_iterator(*d_values_ptr), validity.begin(), - thrust::logical_or{}); + cuda::std::logical_or{}); auto [null_mask, null_count] = cudf::detail::valid_if(validity.begin(), validity.end(), cuda::std::identity{}, stream, mr); diff --git a/cpp/src/io/avro/reader_impl.cu b/cpp/src/io/avro/reader_impl.cu index 11d5749ee38..2be2e42c2b3 100644 --- a/cpp/src/io/avro/reader_impl.cu +++ b/cpp/src/io/avro/reader_impl.cu @@ -21,6 +21,7 @@ #include "io/utilities/hostdevice_vector.hpp" #include +#include #include #include #include @@ -300,8 +301,10 @@ rmm::device_buffer decompress_data(datasource& source, size_t const uncompressed_data_size = std::reduce(uncompressed_data_sizes.begin(), uncompressed_data_sizes.end()); - size_t const max_uncomp_block_size = std::reduce( - uncompressed_data_sizes.begin(), uncompressed_data_sizes.end(), 0, thrust::maximum()); + size_t const max_uncomp_block_size = std::reduce(uncompressed_data_sizes.begin(), + uncompressed_data_sizes.end(), + 0, + cudf::detail::maximum()); size_t temp_size = 0; status = diff --git a/cpp/src/io/comp/nvcomp_adapter.cu b/cpp/src/io/comp/nvcomp_adapter.cu index cf5996dfd93..30501c3f2e2 100644 --- a/cpp/src/io/comp/nvcomp_adapter.cu +++ b/cpp/src/io/comp/nvcomp_adapter.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -15,6 +15,7 @@ */ #include "nvcomp_adapter.cuh" +#include #include #include @@ -122,7 +123,7 @@ std::pair max_chunk_and_total_input_size(device_span()); + cudf::detail::maximum()); auto const sum = thrust::reduce(rmm::exec_policy(stream), input_sizes.begin(), input_sizes.end()); return {max, sum}; } diff --git a/cpp/src/io/fst/logical_stack.cuh b/cpp/src/io/fst/logical_stack.cuh index 7b217d08da3..4b80b981030 100644 --- a/cpp/src/io/fst/logical_stack.cuh +++ b/cpp/src/io/fst/logical_stack.cuh @@ -27,6 +27,7 @@ #include #include +#include #include #include #include @@ -400,7 +401,7 @@ void sparse_stack_op_to_top_of_stack(StackSymbolItT d_symbols, d_kv_operations.Current(), detail::AddStackLevelFromStackOp{symbol_to_stack_op}, num_symbols_in, - cub::Equality{}, + cuda::std::equal_to{}, stream)); stack_level_scan_bytes = std::max(gen_segments_scan_bytes, scan_by_key_bytes); } else { @@ -499,7 +500,7 @@ void sparse_stack_op_to_top_of_stack(StackSymbolItT d_symbols, d_kv_operations.Current(), detail::AddStackLevelFromStackOp{symbol_to_stack_op}, num_symbols_in, - cub::Equality{}, + cuda::std::equal_to{}, stream)); } else { CUDF_CUDA_TRY(cub::DeviceScan::InclusiveScan( diff --git a/cpp/src/io/json/column_tree_construction.cu b/cpp/src/io/json/column_tree_construction.cu index c4fe7926706..13d1751e03d 100644 --- a/cpp/src/io/json/column_tree_construction.cu +++ b/cpp/src/io/json/column_tree_construction.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -17,6 +17,7 @@ #include "nested_json.hpp" #include +#include #include #include #include @@ -208,7 +209,7 @@ std::tuple reduce_to_column_tree( thrust::make_constant_iterator(1), non_leaf_nodes.begin(), non_leaf_nodes_children.begin(), - thrust::equal_to()); + cuda::std::equal_to()); thrust::scatter(rmm::exec_policy_nosync(stream), non_leaf_nodes_children.begin(), diff --git a/cpp/src/io/json/host_tree_algorithms.cu b/cpp/src/io/json/host_tree_algorithms.cu index e506d60a2be..712d280c11f 100644 --- a/cpp/src/io/json/host_tree_algorithms.cu +++ b/cpp/src/io/json/host_tree_algorithms.cu @@ -20,6 +20,7 @@ #include #include +#include #include #include #include @@ -1007,13 +1008,13 @@ void scatter_offsets(tree_meta_t const& tree, col.string_offsets.begin(), col.string_offsets.end(), col.string_offsets.begin(), - thrust::maximum{}); + cudf::detail::maximum{}); } else if (col.type == json_col_t::ListColumn) { thrust::inclusive_scan(rmm::exec_policy_nosync(stream), col.child_offsets.begin(), col.child_offsets.end(), col.child_offsets.begin(), - thrust::maximum{}); + cudf::detail::maximum{}); } } stream.synchronize(); diff --git a/cpp/src/io/json/json_column.cu b/cpp/src/io/json/json_column.cu index 1fe58a0449f..c0790c2f73d 100644 --- a/cpp/src/io/json/json_column.cu +++ b/cpp/src/io/json/json_column.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -130,8 +131,8 @@ reduce_to_column_tree(tree_meta_t const& tree, ordered_row_offsets, unique_col_ids.begin(), max_row_offsets.begin(), - thrust::equal_to(), - thrust::maximum()); + cuda::std::equal_to(), + cudf::detail::maximum()); // 3. reduce_by_key {col_id}, {node_categories} - custom opp (*+v=*, v+v=v, *+#=E) rmm::device_uvector column_categories(num_columns, stream); @@ -142,7 +143,7 @@ reduce_to_column_tree(tree_meta_t const& tree, thrust::make_permutation_iterator(tree.node_categories.begin(), ordered_node_ids.begin()), unique_col_ids.begin(), column_categories.begin(), - thrust::equal_to(), + cuda::std::equal_to(), [] __device__(NodeT type_a, NodeT type_b) -> NodeT { auto is_a_leaf = (type_a == NC_VAL || type_a == NC_STR); auto is_b_leaf = (type_b == NC_VAL || type_b == NC_STR); diff --git a/cpp/src/io/json/json_tree.cu b/cpp/src/io/json/json_tree.cu index e2fe926ea19..e0d6f51aad9 100644 --- a/cpp/src/io/json/json_tree.cu +++ b/cpp/src/io/json/json_tree.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -213,8 +214,8 @@ void propagate_first_sibling_to_other(cudf::device_span node_l sorted_node_levels.end(), thrust::make_permutation_iterator(parent_node_ids.begin(), sorted_order.begin()), thrust::make_permutation_iterator(parent_node_ids.begin(), sorted_order.begin()), - thrust::equal_to{}, - thrust::maximum{}); + cuda::std::equal_to{}, + cudf::detail::maximum{}); } // Generates a tree representation of the given tokens, token_indices. diff --git a/cpp/src/io/json/write_json.cu b/cpp/src/io/json/write_json.cu index 1587c4da9c8..b8f0fe7cb07 100644 --- a/cpp/src/io/json/write_json.cu +++ b/cpp/src/io/json/write_json.cu @@ -333,8 +333,8 @@ std::unique_ptr struct_to_strings(table_view const& strings_columns, validity_iterator, d_str_separator.begin(), false, - thrust::equal_to{}, - thrust::logical_or{}); + cuda::std::equal_to{}, + cuda::std::logical_or{}); thrust::for_each(rmm::exec_policy_nosync(stream), thrust::make_counting_iterator(0), thrust::make_counting_iterator(total_rows), diff --git a/cpp/src/io/orc/stripe_data.cu b/cpp/src/io/orc/stripe_data.cu index c0887304db9..426e470a151 100644 --- a/cpp/src/io/orc/stripe_data.cu +++ b/cpp/src/io/orc/stripe_data.cu @@ -18,6 +18,7 @@ #include "io/utilities/column_buffer.hpp" #include "orc_gpu.hpp" +#include #include #include @@ -1511,10 +1512,11 @@ static __device__ void DecodeRowPositions(orcdec_state_s* s, } if (t == nrows - 1) { s->u.rowdec.nz_count = min(nz_count, s->top.data.max_vals); } __syncthreads(); + // TBD: Brute-forcing this, there might be a more efficient way to find the thread with the // last row last_row = (nz_count == s->u.rowdec.nz_count) ? row_plus1 : 0; - last_row = block_reduce(temp_storage).Reduce(last_row, cub::Max()); + last_row = block_reduce(temp_storage).Reduce(last_row, cudf::detail::maximum{}); nz_pos = (valid) ? nz_count : 0; if (t == 0) { s->top.data.nrows = last_row; } if (valid && nz_pos - 1 < s->u.rowdec.nz_count) { s->u.rowdec.row[nz_pos - 1] = row_plus1; } diff --git a/cpp/src/io/orc/stripe_enc.cu b/cpp/src/io/orc/stripe_enc.cu index 3a1f3a88da4..2ccf3f5d284 100644 --- a/cpp/src/io/orc/stripe_enc.cu +++ b/cpp/src/io/orc/stripe_enc.cu @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -366,8 +367,9 @@ static __device__ uint32_t IntegerRLE( orcenc_state_s* s, T const* inbuf, uint32_t inpos, uint32_t numvals, int t, Storage& temp_storage) { using block_reduce = cub::BlockReduce; - uint8_t* dst = s->stream.data_ptrs[cid] + s->strm_pos[cid]; - uint32_t out_cnt = 0; + + uint8_t* dst = s->stream.data_ptrs[cid] + s->strm_pos[cid]; + uint32_t out_cnt = 0; __shared__ uint64_t block_vmin; while (numvals > 0) { @@ -413,9 +415,9 @@ static __device__ uint32_t IntegerRLE( T vmin = (t < literal_run) ? v0 : cuda::std::numeric_limits::max(); T vmax = (t < literal_run) ? v0 : cuda::std::numeric_limits::min(); uint32_t literal_mode, literal_w; - vmin = block_reduce(temp_storage).Reduce(vmin, cub::Min()); + vmin = block_reduce(temp_storage).Reduce(vmin, cudf::detail::minimum{}); __syncthreads(); - vmax = block_reduce(temp_storage).Reduce(vmax, cub::Max()); + vmax = block_reduce(temp_storage).Reduce(vmax, cudf::detail::maximum{}); if (t == 0) { uint32_t mode1_w, mode2_w; typename std::make_unsigned::type vrange_mode1, vrange_mode2; diff --git a/cpp/src/io/parquet/delta_enc.cuh b/cpp/src/io/parquet/delta_enc.cuh index 56b7c8065ee..8dba755b73a 100644 --- a/cpp/src/io/parquet/delta_enc.cuh +++ b/cpp/src/io/parquet/delta_enc.cuh @@ -19,6 +19,7 @@ #include "parquet_gpu.hpp" #include +#include #include #include @@ -221,6 +222,7 @@ class delta_binary_packer { inline __device__ uint8_t* flush() { using cudf::detail::warp_size; + __shared__ T block_min; int const t = threadIdx.x; @@ -240,7 +242,7 @@ class delta_binary_packer { : cuda::std::numeric_limits::max(); // Find min delta for the block. - auto const min_delta = block_reduce(*_block_tmp).Reduce(delta, cub::Min()); + auto const min_delta = block_reduce(*_block_tmp).Reduce(delta, cudf::detail::minimum{}); if (t == 0) { block_min = min_delta; } __syncthreads(); @@ -250,7 +252,7 @@ class delta_binary_packer { // Get max normalized delta for each warp, and use that to determine how many bits to use // for the bitpacking of this warp. - U const warp_max = warp_reduce(_warp_tmp[warp_id]).Reduce(norm_delta, cub::Max()); + U const warp_max = warp_reduce(_warp_tmp[warp_id]).Reduce(norm_delta, cudf::detail::maximum{}); __syncwarp(); if (lane_id == 0) { _mb_bits[warp_id] = sizeof(long long) * 8 - __clzll(warp_max); } diff --git a/cpp/src/io/parquet/page_string_decode.cu b/cpp/src/io/parquet/page_string_decode.cu index 7d670057cf9..fe9b05c8054 100644 --- a/cpp/src/io/parquet/page_string_decode.cu +++ b/cpp/src/io/parquet/page_string_decode.cu @@ -21,6 +21,7 @@ #include "rle_stream.cuh" #include +#include #include #include @@ -498,6 +499,7 @@ __device__ thrust::pair totalDeltaByteArraySize(uint8_t const* d { using cudf::detail::warp_size; using WarpReduce = cub::WarpReduce; + __shared__ typename WarpReduce::TempStorage temp_storage[2]; __shared__ __align__(16) delta_binary_decoder prefixes; @@ -550,7 +552,8 @@ __device__ thrust::pair totalDeltaByteArraySize(uint8_t const* d // note: warp_sum will only be valid on lane 0. auto const warp_sum = WarpReduce(temp_storage[warp_id]).Sum(lane_sum); __syncwarp(); - auto const warp_max = WarpReduce(temp_storage[warp_id]).Reduce(lane_max, cub::Max()); + auto const warp_max = + WarpReduce(temp_storage[warp_id]).Reduce(lane_max, cudf::detail::maximum{}); if (lane_id == 0) { total_bytes += warp_sum; diff --git a/cpp/src/io/parquet/reader_impl_chunking.cu b/cpp/src/io/parquet/reader_impl_chunking.cu index be1e7d38fff..5242b18b574 100644 --- a/cpp/src/io/parquet/reader_impl_chunking.cu +++ b/cpp/src/io/parquet/reader_impl_chunking.cu @@ -1149,7 +1149,7 @@ void include_decompression_scratch_size(device_span chunk page_keys + pages.size(), decomp_iter, decomp_info.begin(), - thrust::equal_to{}, + cuda::std::equal_to{}, decomp_sum{}); // retrieve to host so we can call nvcomp to get compression scratch sizes @@ -1388,7 +1388,7 @@ void reader::impl::setup_next_subpass(read_mode mode) page_keys + pass.pages.size(), page_size, c_info.begin(), - thrust::equal_to{}, + cuda::std::equal_to{}, cumulative_page_sum{}); // include scratch space needed for decompression. for certain codecs (eg ZSTD) this @@ -1703,7 +1703,7 @@ void reader::impl::compute_output_chunks_for_subpass() page_keys + subpass.pages.size(), page_input, c_info.begin(), - thrust::equal_to{}, + cuda::std::equal_to{}, cumulative_page_sum{}); auto iter = thrust::make_counting_iterator(0); // cap the max row in all pages by the max row we expect in the subpass. input chunking diff --git a/cpp/src/io/parquet/reader_impl_preprocess.cu b/cpp/src/io/parquet/reader_impl_preprocess.cu index e1e9bac5a07..052ed80bc14 100644 --- a/cpp/src/io/parquet/reader_impl_preprocess.cu +++ b/cpp/src/io/parquet/reader_impl_preprocess.cu @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -608,7 +609,7 @@ void decode_page_headers(pass_intermediate_data& pass, level_bit_size, level_bit_size + pass.chunks.size(), 0, - thrust::maximum()); + cudf::detail::maximum()); pass.level_type_size = std::max(1, cudf::util::div_rounding_up_safe(max_level_bits, 8)); // sort the pages in chunk/schema order. diff --git a/cpp/src/io/statistics/typed_statistics_chunk.cuh b/cpp/src/io/statistics/typed_statistics_chunk.cuh index dc023e69423..34e663447e3 100644 --- a/cpp/src/io/statistics/typed_statistics_chunk.cuh +++ b/cpp/src/io/statistics/typed_statistics_chunk.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -27,6 +27,7 @@ #include "statistics_type_identification.cuh" #include "temp_storage_wrapper.cuh" +#include #include #include @@ -202,11 +203,12 @@ __inline__ __device__ typed_statistics_chunk block_reduce( using E = typename detail::extrema_type::type; using extrema_reduce = cub::BlockReduce; using count_reduce = cub::BlockReduce; - output_chunk.minimum_value = - extrema_reduce(storage.template get()).Reduce(output_chunk.minimum_value, cub::Min()); + + output_chunk.minimum_value = extrema_reduce(storage.template get()) + .Reduce(output_chunk.minimum_value, cudf::detail::minimum{}); __syncthreads(); - output_chunk.maximum_value = - extrema_reduce(storage.template get()).Reduce(output_chunk.maximum_value, cub::Max()); + output_chunk.maximum_value = extrema_reduce(storage.template get()) + .Reduce(output_chunk.maximum_value, cudf::detail::maximum{}); __syncthreads(); output_chunk.non_nulls = count_reduce(storage.template get()).Sum(output_chunk.non_nulls); diff --git a/cpp/src/io/utilities/data_casting.cu b/cpp/src/io/utilities/data_casting.cu index 2750a17d328..c6391d49294 100644 --- a/cpp/src/io/utilities/data_casting.cu +++ b/cpp/src/io/utilities/data_casting.cu @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -814,7 +815,7 @@ static std::unique_ptr parse_string(string_view_pair_it str_tuples, str_tuples + col_size, cuda::proclaim_return_type([] __device__(auto t) { return t.second; }), size_type{0}, - thrust::maximum{}); + cudf::detail::maximum{}); auto sizes = rmm::device_uvector(col_size, stream); auto d_sizes = sizes.data(); diff --git a/cpp/src/lists/set_operations.cu b/cpp/src/lists/set_operations.cu index 6f2acbb0712..0ed4b5193b7 100644 --- a/cpp/src/lists/set_operations.cu +++ b/cpp/src/lists/set_operations.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -34,6 +34,7 @@ #include #include +#include #include #include #include @@ -103,8 +104,8 @@ std::unique_ptr have_overlap(lists_column_view const& lhs, contained.begin(), // values to reduce list_indices.begin(), // out keys overlap_results.begin(), // out values - thrust::equal_to{}, // comp for keys - thrust::logical_or{}); // reduction op for values + cuda::std::equal_to{}, // comp for keys + cuda::std::logical_or{}); // reduction op for values auto const num_non_empty_segments = thrust::distance(overlap_results.begin(), end.second); auto [null_mask, null_count] = diff --git a/cpp/src/quantiles/tdigest/tdigest.cu b/cpp/src/quantiles/tdigest/tdigest.cu index 3a365477366..83423649507 100644 --- a/cpp/src/quantiles/tdigest/tdigest.cu +++ b/cpp/src/quantiles/tdigest/tdigest.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -395,7 +396,7 @@ std::unique_ptr percentile_approx(tdigest_column_view const& input, return std::pair{rmm::device_buffer{}, null_count}; } return cudf::detail::valid_if( - tdigest_is_empty, tdigest_is_empty + tdv.size(), thrust::logical_not{}, stream, mr); + tdigest_is_empty, tdigest_is_empty + tdv.size(), cuda::std::logical_not{}, stream, mr); }(); return cudf::make_lists_column(input.size(), diff --git a/cpp/src/quantiles/tdigest/tdigest_aggregation.cu b/cpp/src/quantiles/tdigest/tdigest_aggregation.cu index fd98d262154..f07b8695024 100644 --- a/cpp/src/quantiles/tdigest/tdigest_aggregation.cu +++ b/cpp/src/quantiles/tdigest/tdigest_aggregation.cu @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -793,7 +794,7 @@ std::unique_ptr compute_tdigests(int delta, centroids_begin, // values thrust::make_discard_iterator(), // key output output, // output - thrust::equal_to{}, // key equality check + cuda::std::equal_to{}, // key equality check merge_centroids{}); // create final tdigest column @@ -1161,8 +1162,8 @@ std::unique_ptr merge_tdigests(tdigest_column_view const& tdv, min_iter, thrust::make_discard_iterator(), merged_min_col->mutable_view().begin(), - thrust::equal_to{}, // key equality check - thrust::minimum{}); + cuda::std::equal_to{}, // key equality check + cudf::detail::minimum{}); auto merged_max_col = cudf::make_numeric_column( data_type{type_id::FLOAT64}, num_groups, mask_state::UNALLOCATED, stream, mr); @@ -1176,8 +1177,8 @@ std::unique_ptr merge_tdigests(tdigest_column_view const& tdv, max_iter, thrust::make_discard_iterator(), merged_max_col->mutable_view().begin(), - thrust::equal_to{}, // key equality check - thrust::maximum{}); + cuda::std::equal_to{}, // key equality check + cudf::detail::maximum{}); auto tdigest_offsets = tdv.centroids().offsets(); diff --git a/cpp/src/reductions/segmented/simple.cuh b/cpp/src/reductions/segmented/simple.cuh index 6c35e750e6b..d9b1fefe09a 100644 --- a/cpp/src/reductions/segmented/simple.cuh +++ b/cpp/src/reductions/segmented/simple.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -25,6 +25,7 @@ #include #include #include +#include #include #include #include @@ -249,7 +250,7 @@ std::unique_ptr fixed_point_segmented_reduction( counts.begin(), counts.end(), size_type{0}, - thrust::maximum{}); + cudf::detail::maximum{}); auto const new_scale = numeric::scale_type{col.type().scale() * max_count}; diff --git a/cpp/src/rolling/detail/rolling_collect_list.cu b/cpp/src/rolling/detail/rolling_collect_list.cu index 8a98b65b406..d189b397afd 100644 --- a/cpp/src/rolling/detail/rolling_collect_list.cu +++ b/cpp/src/rolling/detail/rolling_collect_list.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021-2024, NVIDIA CORPORATION. + * Copyright (c) 2021-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -18,6 +18,7 @@ #include #include +#include #include #include @@ -53,6 +54,7 @@ std::unique_ptr get_list_child_to_list_row_mapping(cudf::column_view con // offsets == [0, 2, 5, 5, 8, 11, 13] // scatter result == [0, 0, 1, 0, 0, 2, 0, 0, 1, 0, 0, 1, 0] // + auto const num_child_rows{ cudf::detail::get_value(offsets, offsets.size() - 1, stream)}; auto per_row_mapping = make_fixed_width_column( @@ -83,7 +85,7 @@ std::unique_ptr get_list_child_to_list_row_mapping(cudf::column_view con per_row_mapping_begin, per_row_mapping_begin + num_child_rows, per_row_mapping_begin, - thrust::maximum{}); + cudf::detail::maximum{}); return per_row_mapping; } diff --git a/cpp/src/sort/rank.cu b/cpp/src/sort/rank.cu index e7dca2277ec..35a9a3ec38d 100644 --- a/cpp/src/sort/rank.cu +++ b/cpp/src/sort/rank.cu @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -145,7 +146,7 @@ void tie_break_ranks_transform(cudf::device_span dense_rank_sor tie_iter, thrust::make_discard_iterator(), tie_sorted.begin(), - thrust::equal_to{}, + cuda::std::equal_to{}, tie_breaker); using TransformerReturnType = cuda::std::decay_t>; @@ -202,7 +203,7 @@ void rank_min(cudf::device_span group_keys, thrust::make_counting_iterator(1), sorted_order_view, rank_mutable_view.begin(), - thrust::minimum{}, + cudf::detail::minimum{}, cuda::std::identity{}, stream); } @@ -220,7 +221,7 @@ void rank_max(cudf::device_span group_keys, thrust::make_counting_iterator(1), sorted_order_view, rank_mutable_view.begin(), - thrust::maximum{}, + cudf::detail::maximum{}, cuda::std::identity{}, stream); } diff --git a/cpp/src/strings/split/split.cu b/cpp/src/strings/split/split.cu index 352ca83c8b2..9d30e3d0026 100644 --- a/cpp/src/strings/split/split.cu +++ b/cpp/src/strings/split/split.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. + * Copyright (c) 2019-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -135,7 +136,7 @@ std::unique_ptr
split_fn(strings_column_view const& input, return static_cast(d_offsets[idx + 1] - d_offsets[idx]); }), 0, - thrust::maximum{}); + cudf::detail::maximum{}); // build strings columns for each token position for (size_type col = 0; col < columns_count; ++col) { @@ -346,7 +347,7 @@ std::unique_ptr
whitespace_split_fn(size_type strings_count, // column count is the maximum number of tokens for any string size_type const columns_count = thrust::reduce( - rmm::exec_policy(stream), token_counts.begin(), token_counts.end(), 0, thrust::maximum{}); + rmm::exec_policy(stream), token_counts.begin(), token_counts.end(), 0, cudf::detail::maximum{}); std::vector> results; // boundary case: if no columns, return one null column (issue #119) diff --git a/cpp/src/strings/split/split_re.cu b/cpp/src/strings/split/split_re.cu index ef96b9d3f36..68b610bcb93 100644 --- a/cpp/src/strings/split/split_re.cu +++ b/cpp/src/strings/split/split_re.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -227,7 +228,7 @@ std::unique_ptr
split_re(strings_column_view const& input, return static_cast(d_offsets[idx + 1] - d_offsets[idx]); }), 0, - thrust::maximum{}); + cudf::detail::maximum{}); // boundary case: if no columns, return one all-null column (custrings issue #119) if (columns_count == 0) { diff --git a/cpp/src/text/bpe/byte_pair_encoding.cu b/cpp/src/text/bpe/byte_pair_encoding.cu index 0aacfd16f67..972bcc32077 100644 --- a/cpp/src/text/bpe/byte_pair_encoding.cu +++ b/cpp/src/text/bpe/byte_pair_encoding.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. + * Copyright (c) 2022-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -25,6 +25,7 @@ #include #include #include +#include #include #include #include @@ -212,7 +213,8 @@ CUDF_KERNEL void bpe_parallel_fn(cudf::column_device_view const d_strings, } } // compute the min rank across the block - auto const reduce_rank = block_reduce(temp_storage).Reduce(min_rank, cub::Min(), num_valid); + auto const reduce_rank = + block_reduce(temp_storage).Reduce(min_rank, cudf::detail::minimum{}, num_valid); if (lane_idx == 0) { block_min_rank = reduce_rank; } __syncthreads(); @@ -277,7 +279,8 @@ CUDF_KERNEL void bpe_parallel_fn(cudf::column_device_view const d_strings, } // re-compute the minimum rank across the block (since new pairs are created above) - auto const reduce_rank = block_reduce(temp_storage).Reduce(min_rank, cub::Min(), num_valid); + auto const reduce_rank = + block_reduce(temp_storage).Reduce(min_rank, cudf::detail::minimum{}, num_valid); if (lane_idx == 0) { block_min_rank = reduce_rank; } __syncthreads(); } // if no min ranks are found we are done, otherwise start again diff --git a/cpp/src/text/minhash.cu b/cpp/src/text/minhash.cu index 663595af5df..61a7375772b 100644 --- a/cpp/src/text/minhash.cu +++ b/cpp/src/text/minhash.cu @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -368,7 +369,7 @@ CUDF_KERNEL void minhash_kernel(offsets_type offsets_itr, auto const values = block_values + (lane_idx * block_size); // cooperative groups does not have a min function and cub::BlockReduce was slower auto const minv = - thrust::reduce(thrust::seq, values, values + block_size, init, thrust::minimum{}); + thrust::reduce(thrust::seq, values, values + block_size, init, cudf::detail::minimum{}); if constexpr (blocks_per_row > 1) { // accumulates mins for each block into d_output cuda::atomic_ref ref{d_output[lane_idx + i]}; diff --git a/cpp/tests/iterator/iterator_tests.cuh b/cpp/tests/iterator/iterator_tests.cuh index 119d8e7b138..d6a991f675c 100644 --- a/cpp/tests/iterator/iterator_tests.cuh +++ b/cpp/tests/iterator/iterator_tests.cuh @@ -19,6 +19,7 @@ #include #include +#include #include // for meanvar #include #include @@ -28,7 +29,7 @@ #include #include -#include +#include #include #include #include @@ -59,7 +60,7 @@ struct IteratorTest : public cudf::test::BaseFixture { d_in, dev_result.begin(), num_items, - thrust::minimum{}, + cudf::detail::minimum{}, init, cudf::get_default_stream().value()); @@ -72,7 +73,7 @@ struct IteratorTest : public cudf::test::BaseFixture { d_in, dev_result.begin(), num_items, - thrust::minimum{}, + cudf::detail::minimum{}, init, cudf::get_default_stream().value()); @@ -98,7 +99,7 @@ struct IteratorTest : public cudf::test::BaseFixture { d_in_last, dev_expected.begin(), dev_results.begin(), - thrust::equal_to{}); + cuda::std::equal_to{}); auto result = thrust::all_of(rmm::exec_policy(cudf::get_default_stream()), dev_results.begin(), dev_results.end(), From 45bd05d51435fe4b50ee48a256b3eb4772c5b086 Mon Sep 17 00:00:00 2001 From: Gil Forsyth Date: Mon, 3 Mar 2025 23:27:38 -0500 Subject: [PATCH 119/129] Port all conda recipes to `rattler-build` (#18054) Port all condabuild recipes over to use `rattler-build` instead. Contributes to rapidsai/build-planning#47 - To satisfy `rattler`, this changes all the licenses in the `pyproject.toml` files to the SPDX-compliant `Apache-2.0` instead of `Apache 2.0` Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18054 --- ci/build_cpp.sh | 20 +- ci/build_python.sh | 115 +++++-- conda/recipes/cudf-polars/build.sh | 4 - conda/recipes/cudf-polars/meta.yaml | 61 ---- conda/recipes/cudf-polars/recipe.yaml | 67 ++++ conda/recipes/cudf/build.sh | 4 - conda/recipes/cudf/meta.yaml | 119 ------- conda/recipes/cudf/recipe.yaml | 126 +++++++ conda/recipes/cudf_kafka/build.sh | 3 - conda/recipes/cudf_kafka/meta.yaml | 86 ----- conda/recipes/cudf_kafka/recipe.yaml | 85 +++++ conda/recipes/custreamz/build.sh | 4 - conda/recipes/custreamz/meta.yaml | 65 ---- conda/recipes/custreamz/recipe.yaml | 54 +++ conda/recipes/dask-cudf/build.sh | 4 - conda/recipes/dask-cudf/meta.yaml | 62 ---- conda/recipes/dask-cudf/recipe.yaml | 50 +++ conda/recipes/libcudf/build.sh | 9 - conda/recipes/libcudf/install_libcudf.sh | 4 - .../libcudf/install_libcudf_example.sh | 5 - .../recipes/libcudf/install_libcudf_kafka.sh | 4 - .../recipes/libcudf/install_libcudf_tests.sh | 5 - conda/recipes/libcudf/meta.yaml | 220 ------------ conda/recipes/libcudf/recipe.yaml | 323 ++++++++++++++++++ conda/recipes/pylibcudf/build.sh | 4 - conda/recipes/pylibcudf/meta.yaml | 100 ------ conda/recipes/pylibcudf/recipe.yaml | 106 ++++++ python/cudf/pyproject.toml | 2 +- python/cudf_kafka/pyproject.toml | 2 +- python/cudf_polars/pyproject.toml | 2 +- python/custreamz/pyproject.toml | 2 +- python/dask_cudf/pyproject.toml | 2 +- python/libcudf/pyproject.toml | 2 +- python/pylibcudf/pyproject.toml | 2 +- 34 files changed, 915 insertions(+), 808 deletions(-) delete mode 100644 conda/recipes/cudf-polars/build.sh delete mode 100644 conda/recipes/cudf-polars/meta.yaml create mode 100644 conda/recipes/cudf-polars/recipe.yaml delete mode 100644 conda/recipes/cudf/build.sh delete mode 100644 conda/recipes/cudf/meta.yaml create mode 100644 conda/recipes/cudf/recipe.yaml delete mode 100644 conda/recipes/cudf_kafka/build.sh delete mode 100644 conda/recipes/cudf_kafka/meta.yaml create mode 100644 conda/recipes/cudf_kafka/recipe.yaml delete mode 100644 conda/recipes/custreamz/build.sh delete mode 100644 conda/recipes/custreamz/meta.yaml create mode 100644 conda/recipes/custreamz/recipe.yaml delete mode 100644 conda/recipes/dask-cudf/build.sh delete mode 100644 conda/recipes/dask-cudf/meta.yaml create mode 100644 conda/recipes/dask-cudf/recipe.yaml delete mode 100644 conda/recipes/libcudf/build.sh delete mode 100644 conda/recipes/libcudf/install_libcudf.sh delete mode 100644 conda/recipes/libcudf/install_libcudf_example.sh delete mode 100644 conda/recipes/libcudf/install_libcudf_kafka.sh delete mode 100644 conda/recipes/libcudf/install_libcudf_tests.sh delete mode 100644 conda/recipes/libcudf/meta.yaml create mode 100644 conda/recipes/libcudf/recipe.yaml delete mode 100644 conda/recipes/pylibcudf/build.sh delete mode 100644 conda/recipes/pylibcudf/meta.yaml create mode 100644 conda/recipes/pylibcudf/recipe.yaml diff --git a/ci/build_cpp.sh b/ci/build_cpp.sh index 0c324d01cdf..78a15bc8092 100755 --- a/ci/build_cpp.sh +++ b/ci/build_cpp.sh @@ -17,10 +17,24 @@ rapids-logger "Begin cpp build" sccache --zero-stats -# With boa installed conda build forward to boa -RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) rapids-conda-retry build \ - conda/recipes/libcudf +RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) +export RAPIDS_PACKAGE_VERSION + +source rapids-rattler-channel-string + +# --no-build-id allows for caching with `sccache` +# more info is available at +# https://rattler.build/latest/tips_and_tricks/#using-sccache-or-ccache-with-rattler-build +rattler-build build --recipe conda/recipes/libcudf \ + --experimental \ + --no-build-id \ + --channel-priority disabled \ + --output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \ + "${RATTLER_CHANNELS[@]}" sccache --show-adv-stats +# remove build_cache directory +rm -rf "$RAPIDS_CONDA_BLD_OUTPUT_DIR"/build_cache + rapids-upload-conda-to-s3 cpp diff --git a/ci/build_python.sh b/ci/build_python.sh index abbdc3f3a3b..1dd8b67dfbb 100755 --- a/ci/build_python.sh +++ b/ci/build_python.sh @@ -3,8 +3,6 @@ set -euo pipefail -rapids-configure-conda-channels - source rapids-configure-sccache source rapids-date-string @@ -19,53 +17,100 @@ rapids-logger "Begin py build" CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) +RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) +export RAPIDS_PACKAGE_VERSION + +# populates `RATTLER_CHANNELS` array +source rapids-rattler-channel-string + +rapids-logger "Prepending channel ${CPP_CHANNEL} to RATTLER_CHANNELS" + +RATTLER_CHANNELS=("--channel" "${CPP_CHANNEL}" "${RATTLER_CHANNELS[@]}") + sccache --zero-stats -# TODO: Remove `--no-test` flag once importing on a CPU -# node works correctly -# With boa installed conda build forwards to the boa builder +rapids-logger "Building pylibcudf" -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ - --no-test \ - --channel "${CPP_CHANNEL}" \ - conda/recipes/pylibcudf +# TODO: Remove `--test skip` flag once importing on a CPU node works correctly +# --no-build-id allows for caching with `sccache` +# more info is available at +# https://rattler.build/latest/tips_and_tricks/#using-sccache-or-ccache-with-rattler-build +rattler-build build --recipe conda/recipes/pylibcudf \ + --experimental \ + --no-build-id \ + --channel-priority disabled \ + --output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \ + --test skip \ + "${RATTLER_CHANNELS[@]}" sccache --show-adv-stats sccache --zero-stats -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ - --no-test \ - --channel "${CPP_CHANNEL}" \ - --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ - conda/recipes/cudf +rapids-logger "Building cudf" + +rattler-build build --recipe conda/recipes/cudf \ + --experimental \ + --no-build-id \ + --channel-priority disabled \ + --output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \ + --test skip \ + "${RATTLER_CHANNELS[@]}" + +sccache --show-adv-stats +sccache --zero-stats + +rapids-logger "Building dask-cudf" + +rattler-build build --recipe conda/recipes/dask-cudf \ + --experimental \ + --no-build-id \ + --channel-priority disabled \ + --output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \ + --test skip \ + "${RATTLER_CHANNELS[@]}" + +sccache --show-adv-stats +sccache --zero-stats + +rapids-logger "Building cudf_kafka" + +rattler-build build --recipe conda/recipes/cudf_kafka \ + --experimental \ + --no-build-id \ + --channel-priority disabled \ + --output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \ + --test skip \ + "${RATTLER_CHANNELS[@]}" + +sccache --show-adv-stats +sccache --zero-stats + +rapids-logger "Building custreamz" + +rattler-build build --recipe conda/recipes/custreamz \ + --experimental \ + --no-build-id \ + --channel-priority disabled \ + --output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \ + --test skip \ + "${RATTLER_CHANNELS[@]}" sccache --show-adv-stats sccache --zero-stats -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ - --no-test \ - --channel "${CPP_CHANNEL}" \ - --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ - conda/recipes/dask-cudf +rapids-logger "Building cudf-polars" -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ - --no-test \ - --channel "${CPP_CHANNEL}" \ - --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ - conda/recipes/cudf_kafka +rattler-build build --recipe conda/recipes/cudf-polars \ + --experimental \ + --no-build-id \ + --channel-priority disabled \ + --output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \ + --test skip \ + "${RATTLER_CHANNELS[@]}" sccache --show-adv-stats -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ - --no-test \ - --channel "${CPP_CHANNEL}" \ - --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ - conda/recipes/custreamz - -RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \ - --no-test \ - --channel "${CPP_CHANNEL}" \ - --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \ - conda/recipes/cudf-polars +# remove build_cache directory +rm -rf "$RAPIDS_CONDA_BLD_OUTPUT_DIR"/build_cache rapids-upload-conda-to-s3 python diff --git a/conda/recipes/cudf-polars/build.sh b/conda/recipes/cudf-polars/build.sh deleted file mode 100644 index 06e2f1bcb99..00000000000 --- a/conda/recipes/cudf-polars/build.sh +++ /dev/null @@ -1,4 +0,0 @@ -# Copyright (c) 2024, NVIDIA CORPORATION. - -# This assumes the script is executed from the root of the repo directory -./build.sh cudf_polars diff --git a/conda/recipes/cudf-polars/meta.yaml b/conda/recipes/cudf-polars/meta.yaml deleted file mode 100644 index 64a147d3c63..00000000000 --- a/conda/recipes/cudf-polars/meta.yaml +++ /dev/null @@ -1,61 +0,0 @@ -# Copyright (c) 2024-2025, NVIDIA CORPORATION. - -{% set version = environ['RAPIDS_PACKAGE_VERSION'].lstrip('v') %} -{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %} -{% set py_version = environ['CONDA_PY'] %} -{% set cuda_version = '.'.join(environ['RAPIDS_CUDA_VERSION'].split('.')[:2]) %} -{% set cuda_major = cuda_version.split('.')[0] %} -{% set date_string = environ['RAPIDS_DATE_STRING'] %} - -package: - name: cudf-polars - version: {{ version }} - -source: - path: ../../.. - -build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - script_env: - - AWS_ACCESS_KEY_ID - - AWS_SECRET_ACCESS_KEY - - AWS_SESSION_TOKEN - - CMAKE_C_COMPILER_LAUNCHER - - CMAKE_CUDA_COMPILER_LAUNCHER - - CMAKE_CXX_COMPILER_LAUNCHER - - CMAKE_GENERATOR - - PARALLEL_LEVEL - - SCCACHE_BUCKET - - SCCACHE_IDLE_TIMEOUT - - SCCACHE_REGION - - SCCACHE_S3_KEY_PREFIX=cudf-polars-aarch64 # [aarch64] - - SCCACHE_S3_KEY_PREFIX=cudf-polars-linux64 # [linux64] - - SCCACHE_S3_USE_SSL - - SCCACHE_S3_NO_CREDENTIALS - -requirements: - host: - - python - - rapids-build-backend >=0.3.0,<0.4.0.dev0 - - setuptools - - cuda-version ={{ cuda_version }} - run: - - python - - pylibcudf ={{ version }} - - polars >=1.20,<1.24 - - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - -test: - requires: - - cuda-version ={{ cuda_version }} - imports: - - cudf_polars - - -about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: cudf-polars library diff --git a/conda/recipes/cudf-polars/recipe.yaml b/conda/recipes/cudf-polars/recipe.yaml new file mode 100644 index 00000000000..8eaf7e4f843 --- /dev/null +++ b/conda/recipes/cudf-polars/recipe.yaml @@ -0,0 +1,67 @@ +# Copyright (c) 2018-2025, NVIDIA CORPORATION. +schema_version: 1 + +context: + version: ${{ env.get("RAPIDS_PACKAGE_VERSION") }} + minor_version: ${{ (version | split("."))[:2] | join(".") }} + cuda_version: ${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[:2] | join(".") }} + cuda_major: '${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[0] }}' + date_string: '${{ env.get("RAPIDS_DATE_STRING") }}' + py_version: ${{ env.get("RAPIDS_PY_VERSION") }} + py_buildstring: ${{ py_version | version_to_buildstring }} + head_rev: ${{ git.head_rev(".")[:8] }} + +package: + name: cudf-polars + version: ${{ version }} + +source: + path: ../../.. + +build: + string: cuda${{ cuda_major }}_py${{ py_buildstring }}_${{ date_string }}_${{ head_rev }} + script: + content: | + ./build.sh cudf_polars + secrets: + - AWS_ACCESS_KEY_ID + - AWS_SECRET_ACCESS_KEY + - AWS_SESSION_TOKEN + env: + CMAKE_C_COMPILER_LAUNCHER: ${{ env.get("CMAKE_C_COMPILER_LAUNCHER") }} + CMAKE_CUDA_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CUDA_COMPILER_LAUNCHER") }} + CMAKE_CXX_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CXX_COMPILER_LAUNCHER") }} + CMAKE_GENERATOR: ${{ env.get("CMAKE_GENERATOR") }} + SCCACHE_BUCKET: ${{ env.get("SCCACHE_BUCKET") }} + SCCACHE_IDLE_TIMEOUT: ${{ env.get("SCCACHE_IDLE_TIMEOUT") }} + SCCACHE_REGION: ${{ env.get("SCCACHE_REGION") }} + SCCACHE_S3_USE_SSL: ${{ env.get("SCCACHE_S3_USE_SSL") }} + SCCACHE_S3_NO_CREDENTIALS: ${{ env.get("SCCACHE_S3_NO_CREDENTIALS") }} + SCCACHE_S3_KEY_PREFIX: cudf-polars-${{ env.get("RAPIDS_CONDA_ARCH") }} + +requirements: + host: + - python =${{ py_version }} + - pip + - rapids-build-backend >=0.3.0,<0.4.0.dev0 + - setuptools + - cuda-version =${{ cuda_version }} + run: + - python + - pylibcudf =${{ version }} + - polars >=1.20,<1.24 + - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }} + ignore_run_exports: + by_name: + - cuda-version + +tests: + - python: + imports: + - cudf_polars + pip_check: false + +about: + homepage: ${{ load_from_file("python/cudf_polars/pyproject.toml").project.urls.Homepage }} + license: ${{ load_from_file("python/cudf_polars/pyproject.toml").project.license.text }} + summary: ${{ load_from_file("python/cudf_polars/pyproject.toml").project.description }} diff --git a/conda/recipes/cudf/build.sh b/conda/recipes/cudf/build.sh deleted file mode 100644 index 43d046402c7..00000000000 --- a/conda/recipes/cudf/build.sh +++ /dev/null @@ -1,4 +0,0 @@ -# Copyright (c) 2018-2022, NVIDIA CORPORATION. - -# This assumes the script is executed from the root of the repo directory -./build.sh cudf diff --git a/conda/recipes/cudf/meta.yaml b/conda/recipes/cudf/meta.yaml deleted file mode 100644 index 43060ef1c87..00000000000 --- a/conda/recipes/cudf/meta.yaml +++ /dev/null @@ -1,119 +0,0 @@ -# Copyright (c) 2018-2025, NVIDIA CORPORATION. - -{% set version = environ['RAPIDS_PACKAGE_VERSION'].lstrip('v') %} -{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %} -{% set py_version = environ['CONDA_PY'] %} -{% set cuda_version = '.'.join(environ['RAPIDS_CUDA_VERSION'].split('.')[:2]) %} -{% set cuda_major = cuda_version.split('.')[0] %} -{% set date_string = environ['RAPIDS_DATE_STRING'] %} - -package: - name: cudf - version: {{ version }} - -source: - path: ../../.. - -build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - script_env: - - AWS_ACCESS_KEY_ID - - AWS_SECRET_ACCESS_KEY - - AWS_SESSION_TOKEN - - CMAKE_C_COMPILER_LAUNCHER - - CMAKE_CUDA_COMPILER_LAUNCHER - - CMAKE_CXX_COMPILER_LAUNCHER - - CMAKE_GENERATOR - - PARALLEL_LEVEL - - SCCACHE_BUCKET - - SCCACHE_IDLE_TIMEOUT - - SCCACHE_REGION - - SCCACHE_S3_KEY_PREFIX=cudf-aarch64 # [aarch64] - - SCCACHE_S3_KEY_PREFIX=cudf-linux64 # [linux64] - - SCCACHE_S3_USE_SSL - - SCCACHE_S3_NO_CREDENTIALS - ignore_run_exports_from: - - {{ compiler('cuda') }} - {% if cuda_major != "11" %} - - cuda-cudart-dev - - libcufile-dev # [linux64] - {% endif %} - -requirements: - build: - - cmake {{ cmake_version }} - - ninja - - {{ compiler('c') }} - - {{ compiler('cxx') }} - {% if cuda_major == "11" %} - - {{ compiler('cuda') }} ={{ cuda_version }} - {% else %} - - {{ compiler('cuda') }} - {% endif %} - - cuda-version ={{ cuda_version }} - - {{ stdlib("c") }} - host: - - python - - cython >=3.0.3 - - rapids-build-backend >=0.3.0,<0.4.0.dev0 - - scikit-build-core >=0.10.0 - - dlpack >=0.8,<1.0 - - libcudf ={{ version }} - - pylibcudf ={{ version }} - - rmm ={{ minor_version }} - {% if cuda_major == "11" %} - - cudatoolkit - {% else %} - - cuda-cudart-dev - - cuda-nvrtc - - libcufile-dev # [linux64] - {% endif %} - - cuda-version ={{ cuda_version }} - run: - - python - - typing_extensions >=4.0.0 - - pandas >=2.0,<2.2.4dev0 - - cupy >=12.0.0 - - numba-cuda >=0.4.0,<0.5.0a0 - - numba >=0.59.1,<0.62.0a0 - - numpy >=1.23,<2.1 - - pyarrow>=14.0.0,<20.0.0a0 - - libcudf ={{ version }} - - pylibcudf ={{ version }} - - {{ pin_compatible('rmm', max_pin='x.x') }} - - fsspec >=0.6.0 - {% if cuda_major == "11" %} - - cudatoolkit - - ptxcompiler >=0.7.0 - - cubinlinker # CUDA enhanced compatibility. - - cuda-python >=11.8.5,<12.0a0 - {% else %} - - cuda-cudart - - libcufile # [linux64] - # Needed by Numba for CUDA support - - cuda-nvcc-impl - # TODO: Add nvjitlink here - # xref: https://github.com/rapidsai/cudf/issues/12822 - - cuda-nvrtc - - cuda-python >=12.6.2,<13.0a0 - - pynvjitlink - {% endif %} - - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - - nvtx >=0.2.1 - - packaging - - cachetools - - rich - -test: - requires: - - cuda-version ={{ cuda_version }} - imports: - - cudf - -about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: cuDF GPU DataFrame core library diff --git a/conda/recipes/cudf/recipe.yaml b/conda/recipes/cudf/recipe.yaml new file mode 100644 index 00000000000..2cb330fb76d --- /dev/null +++ b/conda/recipes/cudf/recipe.yaml @@ -0,0 +1,126 @@ +# Copyright (c) 2018-2025, NVIDIA CORPORATION. +schema_version: 1 + +context: + version: ${{ env.get("RAPIDS_PACKAGE_VERSION") }} + minor_version: ${{ (version | split("."))[:2] | join(".") }} + cuda_version: ${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[:2] | join(".") }} + cuda_major: '${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[0] }}' + date_string: '${{ env.get("RAPIDS_DATE_STRING") }}' + py_version: ${{ env.get("RAPIDS_PY_VERSION") }} + py_buildstring: ${{ py_version | version_to_buildstring }} + head_rev: ${{ git.head_rev(".")[:8] }} + +package: + name: cudf + version: ${{ version }} + +source: + path: ../../.. + +build: + string: cuda${{ cuda_major }}_py${{ py_buildstring }}_${{ date_string }}_${{ head_rev }} + script: + content: | + ./build.sh cudf + secrets: + - AWS_ACCESS_KEY_ID + - AWS_SECRET_ACCESS_KEY + - AWS_SESSION_TOKEN + env: + CMAKE_C_COMPILER_LAUNCHER: ${{ env.get("CMAKE_C_COMPILER_LAUNCHER") }} + CMAKE_CUDA_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CUDA_COMPILER_LAUNCHER") }} + CMAKE_CXX_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CXX_COMPILER_LAUNCHER") }} + CMAKE_GENERATOR: ${{ env.get("CMAKE_GENERATOR") }} + SCCACHE_BUCKET: ${{ env.get("SCCACHE_BUCKET") }} + SCCACHE_IDLE_TIMEOUT: ${{ env.get("SCCACHE_IDLE_TIMEOUT") }} + SCCACHE_REGION: ${{ env.get("SCCACHE_REGION") }} + SCCACHE_S3_USE_SSL: ${{ env.get("SCCACHE_S3_USE_SSL") }} + SCCACHE_S3_NO_CREDENTIALS: ${{ env.get("SCCACHE_S3_NO_CREDENTIALS") }} + SCCACHE_S3_KEY_PREFIX: cudf-${{ env.get("RAPIDS_CONDA_ARCH") }} + +requirements: + build: + - cmake ${{ cmake_version }} + - ninja + - ${{ compiler("c") }} + - ${{ compiler("cxx") }} + - ${{ compiler("cuda") }} + - cuda-version =${{ cuda_version }} + - ${{ stdlib("c") }} + host: + - python =${{ py_version }} + - pip + - cython >=3.0.3 + - rapids-build-backend >=0.3.0,<0.4.0.dev0 + - scikit-build-core >=0.10.0 + - dlpack >=0.8,<1.0 + - libcudf =${{ version }} + - pylibcudf =${{ version }} + - rmm =${{ minor_version }} + - if: cuda_major == "11" + then: + - cudatoolkit + else: + - cuda-cudart-dev + - cuda-nvrtc + - if: linux64 + then: + - libcufile-dev + - cuda-version =${{ cuda_version }} + run: + - python + - typing_extensions >=4.0.0 + - pandas >=2.0,<2.2.4dev0 + - cupy >=12.0.0 + - numba-cuda >=0.4.0,<0.5.0a0 + - numba >=0.59.1,<0.62.0a0 + - numpy >=1.23,<2.1 + - pyarrow>=14.0.0,<20.0.0a0 + - libcudf =${{ version }} + - pylibcudf =${{ version }} + - ${{ pin_compatible("rmm", upper_bound="x.x") }} + - fsspec >=0.6.0 + - if: cuda_major == "11" + then: + - cudatoolkit + - ptxcompiler >=0.7.0 + - cubinlinker # CUDA enhanced compatibility. + - cuda-python >=11.8.5,<12.0a0 + else: + - cuda-cudart + # Needed by Numba for CUDA support + - cuda-nvcc-impl + # TODO: Add nvjitlink here + # xref: https://github.com/rapidsai/cudf/issues/12822 + - cuda-nvrtc + - cuda-python >=12.6.2,<13.0a0 + - pynvjitlink + - if: linux64 + then: + - libcufile + - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }} + - nvtx >=0.2.1 + - packaging + - cachetools + - rich + ignore_run_exports: + from_package: + - if: cuda_major != "11" + then: + - cuda-cudart-dev + - if: linux64 + then: libcufile-dev + by_name: + - cuda-version + +tests: + - python: + imports: + - cudf + pip_check: false + +about: + homepage: ${{ load_from_file("python/cudf/pyproject.toml").project.urls.Homepage }} + license: ${{ load_from_file("python/cudf/pyproject.toml").project.license.text }} + summary: ${{ load_from_file("python/cudf/pyproject.toml").project.description }} diff --git a/conda/recipes/cudf_kafka/build.sh b/conda/recipes/cudf_kafka/build.sh deleted file mode 100644 index 9458349d101..00000000000 --- a/conda/recipes/cudf_kafka/build.sh +++ /dev/null @@ -1,3 +0,0 @@ -# Copyright (c) 2020-2023, NVIDIA CORPORATION. - -./build.sh -v cudf_kafka diff --git a/conda/recipes/cudf_kafka/meta.yaml b/conda/recipes/cudf_kafka/meta.yaml deleted file mode 100644 index a070c041d99..00000000000 --- a/conda/recipes/cudf_kafka/meta.yaml +++ /dev/null @@ -1,86 +0,0 @@ -# Copyright (c) 2020-2025, NVIDIA CORPORATION. - -{% set version = environ['RAPIDS_PACKAGE_VERSION'].lstrip('v') %} -{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %} -{% set py_version = environ['CONDA_PY'] %} -{% set cuda_version = '.'.join(environ['RAPIDS_CUDA_VERSION'].split('.')[:2]) %} -{% set cuda_major = cuda_version.split('.')[0] %} -{% set date_string = environ['RAPIDS_DATE_STRING'] %} - -package: - name: cudf_kafka - version: {{ version }} - -source: - path: ../../.. - -build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - script_env: - - AWS_ACCESS_KEY_ID - - AWS_SECRET_ACCESS_KEY - - AWS_SESSION_TOKEN - - CMAKE_C_COMPILER_LAUNCHER - - CMAKE_CUDA_COMPILER_LAUNCHER - - CMAKE_CXX_COMPILER_LAUNCHER - - CMAKE_GENERATOR - - PARALLEL_LEVEL - - SCCACHE_BUCKET - - SCCACHE_IDLE_TIMEOUT - - SCCACHE_REGION - - SCCACHE_S3_KEY_PREFIX=cudf-kafka-aarch64 # [aarch64] - - SCCACHE_S3_KEY_PREFIX=cudf-kafka-linux64 # [linux64] - - SCCACHE_S3_USE_SSL - - SCCACHE_S3_NO_CREDENTIALS - ignore_run_exports_from: - - {{ compiler('cuda') }} - {% if cuda_major != "11" %} - - cuda-cudart-dev - {% endif %} - -requirements: - build: - - cmake {{ cmake_version }} - - ninja - - {{ compiler('c') }} - - {{ compiler('cxx') }} - {% if cuda_major == "11" %} - - {{ compiler('cuda') }} ={{ cuda_version }} - {% else %} - - {{ compiler('cuda') }} - {% endif %} - - cuda-version ={{ cuda_version }} - - {{ stdlib("c") }} - host: - - python - - cython >=3.0.3 - - cuda-version ={{ cuda_version }} - - pylibcudf ={{ version }} - - libcudf_kafka ={{ version }} - - rapids-build-backend >=0.3.0,<0.4.0.dev0 - - scikit-build-core >=0.10.0 - {% if cuda_major != "11" %} - - cuda-cudart-dev - {% endif %} - run: - - python - - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - - libcudf_kafka ={{ version }} - - pylibcudf ={{ version }} - {% if cuda_major != "11" %} - - cuda-cudart - {% endif %} - -test: - requires: - - cuda-version ={{ cuda_version }} - imports: - - cudf_kafka - -about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: libcudf_kafka library diff --git a/conda/recipes/cudf_kafka/recipe.yaml b/conda/recipes/cudf_kafka/recipe.yaml new file mode 100644 index 00000000000..aba9d979e44 --- /dev/null +++ b/conda/recipes/cudf_kafka/recipe.yaml @@ -0,0 +1,85 @@ +# Copyright (c) 2018-2025, NVIDIA CORPORATION. +schema_version: 1 + +context: + version: ${{ env.get("RAPIDS_PACKAGE_VERSION") }} + minor_version: ${{ (version | split("."))[:2] | join(".") }} + cuda_version: ${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[:2] | join(".") }} + cuda_major: '${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[0] }}' + date_string: '${{ env.get("RAPIDS_DATE_STRING") }}' + py_version: ${{ env.get("RAPIDS_PY_VERSION") }} + py_buildstring: ${{ py_version | version_to_buildstring }} + head_rev: ${{ git.head_rev(".")[:8] }} + +package: + name: cudf_kafka + version: ${{ version }} + +source: + path: ../../.. + +build: + string: cuda${{ cuda_major }}_py${{ py_buildstring }}_${{ date_string }}_${{ head_rev }} + script: + content: | + ./build.sh cudf_kafka + secrets: + - AWS_ACCESS_KEY_ID + - AWS_SECRET_ACCESS_KEY + - AWS_SESSION_TOKEN + env: + CMAKE_C_COMPILER_LAUNCHER: ${{ env.get("CMAKE_C_COMPILER_LAUNCHER") }} + CMAKE_CUDA_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CUDA_COMPILER_LAUNCHER") }} + CMAKE_CXX_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CXX_COMPILER_LAUNCHER") }} + CMAKE_GENERATOR: ${{ env.get("CMAKE_GENERATOR") }} + SCCACHE_BUCKET: ${{ env.get("SCCACHE_BUCKET") }} + SCCACHE_IDLE_TIMEOUT: ${{ env.get("SCCACHE_IDLE_TIMEOUT") }} + SCCACHE_REGION: ${{ env.get("SCCACHE_REGION") }} + SCCACHE_S3_USE_SSL: ${{ env.get("SCCACHE_S3_USE_SSL") }} + SCCACHE_S3_NO_CREDENTIALS: ${{ env.get("SCCACHE_S3_NO_CREDENTIALS") }} + SCCACHE_S3_KEY_PREFIX: cudf-kafka-${{ env.get("RAPIDS_CONDA_ARCH") }} + +requirements: + build: + - cmake ${{ cmake_version }} + - ninja + - ${{ compiler("c") }} + - ${{ compiler("cxx") }} + - ${{ compiler("cuda") }} + - cuda-version =${{ cuda_version }} + - ${{ stdlib("c") }} + host: + - python =${{ py_version }} + - pip + - cython >=3.0.3 + - cuda-version =${{ cuda_version }} + - pylibcudf =${{ version }} + - libcudf_kafka =${{ version }} + - rapids-build-backend >=0.3.0,<0.4.0.dev0 + - scikit-build-core >=0.10.0 + - if: cuda_major != "11" + then: cuda-cudart-dev + run: + - python + - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }} + - libcudf_kafka =${{ version }} + - pylibcudf =${{ version }} + - if: cuda_major != "11" + then: cuda-cudart + ignore_run_exports: + from_package: + - if: cuda_major != "11" + then: cuda-cudart-dev + by_name: + - cuda-version + +tests: + - python: + imports: + - cudf_kafka + pip_check: false + +about: + homepage: ${{ load_from_file("python/cudf_kafka/pyproject.toml").project.urls.Homepage }} + license: ${{ load_from_file("python/cudf_kafka/pyproject.toml").project.license.text }} + summary: ${{ load_from_file("python/cudf_kafka/pyproject.toml").project.description }} diff --git a/conda/recipes/custreamz/build.sh b/conda/recipes/custreamz/build.sh deleted file mode 100644 index 88fccf90c69..00000000000 --- a/conda/recipes/custreamz/build.sh +++ /dev/null @@ -1,4 +0,0 @@ -# Copyright (c) 2020-2022, NVIDIA CORPORATION. - -# This assumes the script is executed from the root of the repo directory -./build.sh -v custreamz diff --git a/conda/recipes/custreamz/meta.yaml b/conda/recipes/custreamz/meta.yaml deleted file mode 100644 index a031f05a73a..00000000000 --- a/conda/recipes/custreamz/meta.yaml +++ /dev/null @@ -1,65 +0,0 @@ -# Copyright (c) 2018-2024, NVIDIA CORPORATION. - -{% set version = environ['RAPIDS_PACKAGE_VERSION'].lstrip('v') %} -{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %} -{% set py_version = environ['CONDA_PY'] %} -{% set cuda_version = '.'.join(environ['RAPIDS_CUDA_VERSION'].split('.')[:2]) %} -{% set cuda_major = cuda_version.split('.')[0] %} -{% set date_string = environ['RAPIDS_DATE_STRING'] %} - -package: - name: custreamz - version: {{ version }} - -source: - path: ../../.. - -build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - script_env: - - AWS_ACCESS_KEY_ID - - AWS_SECRET_ACCESS_KEY - - AWS_SESSION_TOKEN - - CMAKE_C_COMPILER_LAUNCHER - - CMAKE_CUDA_COMPILER_LAUNCHER - - CMAKE_CXX_COMPILER_LAUNCHER - - CMAKE_GENERATOR - - PARALLEL_LEVEL - - SCCACHE_BUCKET - - SCCACHE_IDLE_TIMEOUT - - SCCACHE_REGION - - SCCACHE_S3_KEY_PREFIX=custreamz-aarch64 # [aarch64] - - SCCACHE_S3_KEY_PREFIX=custreamz-linux64 # [linux64] - - SCCACHE_S3_USE_SSL - - SCCACHE_S3_NO_CREDENTIALS - -requirements: - host: - - python - - rapids-build-backend >=0.3.0,<0.4.0.dev0 - - setuptools - - python-confluent-kafka >=2.5.0,<2.6.0a0 - - cudf_kafka ={{ version }} - - cuda-version ={{ cuda_version }} - run: - - python - - streamz - - cudf ={{ version }} - - cudf_kafka ={{ version }} - - rapids-dask-dependency ={{ minor_version }} - - python-confluent-kafka >=2.5.0,<2.6.0a0 - - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - -test: - requires: - - cuda-version ={{ cuda_version }} - imports: - - custreamz - -about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: cuStreamz library diff --git a/conda/recipes/custreamz/recipe.yaml b/conda/recipes/custreamz/recipe.yaml new file mode 100644 index 00000000000..4713df9efad --- /dev/null +++ b/conda/recipes/custreamz/recipe.yaml @@ -0,0 +1,54 @@ +# Copyright (c) 2018-2025, NVIDIA CORPORATION. +schema_version: 1 + +context: + version: ${{ env.get("RAPIDS_PACKAGE_VERSION") }} + minor_version: ${{ (version | split("."))[:2] | join(".") }} + cuda_version: ${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[:2] | join(".") }} + cuda_major: '${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[0] }}' + date_string: '${{ env.get("RAPIDS_DATE_STRING") }}' + py_version: ${{ env.get("RAPIDS_PY_VERSION") }} + py_buildstring: ${{ py_version | version_to_buildstring }} + head_rev: ${{ git.head_rev(".")[:8] }} + +package: + name: custreamz + version: ${{ version }} + +source: + path: ../../.. + +build: + string: cuda${{ cuda_major }}_py${{ py_buildstring }}_${{ date_string }}_${{ head_rev }} + script: + content: | + ./build.sh custreamz + +requirements: + host: + - python =${{ py_version }} + - pip + - rapids-build-backend >=0.3.0,<0.4.0.dev0 + - setuptools + - python-confluent-kafka >=2.5.0,<2.6.0a0 + - cudf_kafka =${{ version }} + - cuda-version =${{ cuda_version }} + run: + - python + - streamz + - cudf =${{ version }} + - cudf_kafka =${{ version }} + - rapids-dask-dependency =${{ minor_version }} + - python-confluent-kafka >=2.5.0,<2.6.0a0 + - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }} + +tests: + - python: + imports: + - custreamz + pip_check: false + +about: + homepage: ${{ load_from_file("python/custreamz/pyproject.toml").project.urls.Homepage }} + license: ${{ load_from_file("python/custreamz/pyproject.toml").project.license.text }} + summary: ${{ load_from_file("python/custreamz/pyproject.toml").project.description }} diff --git a/conda/recipes/dask-cudf/build.sh b/conda/recipes/dask-cudf/build.sh deleted file mode 100644 index 473f52c28a0..00000000000 --- a/conda/recipes/dask-cudf/build.sh +++ /dev/null @@ -1,4 +0,0 @@ -# Copyright (c) 2018-2019, NVIDIA CORPORATION. - -# This assumes the script is executed from the root of the repo directory -./build.sh dask_cudf diff --git a/conda/recipes/dask-cudf/meta.yaml b/conda/recipes/dask-cudf/meta.yaml deleted file mode 100644 index a476d5d53df..00000000000 --- a/conda/recipes/dask-cudf/meta.yaml +++ /dev/null @@ -1,62 +0,0 @@ -# Copyright (c) 2018-2024, NVIDIA CORPORATION. - -{% set version = environ['RAPIDS_PACKAGE_VERSION'].lstrip('v') %} -{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %} -{% set py_version = environ['CONDA_PY'] %} -{% set cuda_version = '.'.join(environ['RAPIDS_CUDA_VERSION'].split('.')[:2]) %} -{% set cuda_major = cuda_version.split('.')[0] %} -{% set date_string = environ['RAPIDS_DATE_STRING'] %} - -package: - name: dask-cudf - version: {{ version }} - -source: - path: ../../.. - -build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - script_env: - - AWS_ACCESS_KEY_ID - - AWS_SECRET_ACCESS_KEY - - AWS_SESSION_TOKEN - - CMAKE_C_COMPILER_LAUNCHER - - CMAKE_CUDA_COMPILER_LAUNCHER - - CMAKE_CXX_COMPILER_LAUNCHER - - CMAKE_GENERATOR - - PARALLEL_LEVEL - - SCCACHE_BUCKET - - SCCACHE_IDLE_TIMEOUT - - SCCACHE_REGION - - SCCACHE_S3_KEY_PREFIX=dask-cudf-aarch64 # [aarch64] - - SCCACHE_S3_KEY_PREFIX=dask-cudf-linux64 # [linux64] - - SCCACHE_S3_USE_SSL - - SCCACHE_S3_NO_CREDENTIALS - -requirements: - host: - - python - - rapids-build-backend >=0.3.0,<0.4.0.dev0 - - setuptools - - cuda-version ={{ cuda_version }} - run: - - python - - cudf ={{ version }} - - pynvml >=12.0.0,<13.0.0a0 - - rapids-dask-dependency ={{ minor_version }} - - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - -test: - requires: - - cuda-version ={{ cuda_version }} - imports: - - dask_cudf - - -about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: dask-cudf library diff --git a/conda/recipes/dask-cudf/recipe.yaml b/conda/recipes/dask-cudf/recipe.yaml new file mode 100644 index 00000000000..997150d2832 --- /dev/null +++ b/conda/recipes/dask-cudf/recipe.yaml @@ -0,0 +1,50 @@ +# Copyright (c) 2018-2025, NVIDIA CORPORATION. +schema_version: 1 + +context: + version: ${{ env.get("RAPIDS_PACKAGE_VERSION") }} + minor_version: ${{ (version | split("."))[:2] | join(".") }} + cuda_version: ${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[:2] | join(".") }} + cuda_major: '${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[0] }}' + date_string: '${{ env.get("RAPIDS_DATE_STRING") }}' + py_version: ${{ env.get("RAPIDS_PY_VERSION") }} + py_buildstring: ${{ py_version | version_to_buildstring }} + head_rev: ${{ git.head_rev(".")[:8] }} + +package: + name: dask-cudf + version: ${{ version }} + +source: + path: ../../.. + +build: + string: cuda${{ cuda_major }}_py${{ py_buildstring }}_${{ date_string }}_${{ head_rev }} + script: + content: | + ./build.sh dask_cudf + +requirements: + host: + - python =${{ py_version }} + - pip + - rapids-build-backend >=0.3.0,<0.4.0.dev0 + - setuptools + - cuda-version =${{ cuda_version }} + run: + - python + - cudf =${{ version }} + - pynvml >=12.0.0,<13.0.0a0 + - rapids-dask-dependency =${{ minor_version }} + - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }} + +tests: + - python: + imports: + - dask_cudf + pip_check: false + +about: + homepage: ${{ load_from_file("python/dask_cudf/pyproject.toml").project.urls.Homepage }} + license: ${{ load_from_file("python/dask_cudf/pyproject.toml").project.license.text }} + summary: ${{ load_from_file("python/dask_cudf/pyproject.toml").project.description }} diff --git a/conda/recipes/libcudf/build.sh b/conda/recipes/libcudf/build.sh deleted file mode 100644 index a3a0415575b..00000000000 --- a/conda/recipes/libcudf/build.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/bash -# Copyright (c) 2018-2024, NVIDIA CORPORATION. - -export cudf_ROOT="$(realpath ./cpp/build)" - -./build.sh -n -v \ - libcudf libcudf_kafka benchmarks tests \ - --build_metrics --incl_cache_stats --allgpuarch \ - --cmake-args=\"-DCMAKE_INSTALL_LIBDIR=lib -DCUDF_ENABLE_ARROW_S3=ON\" diff --git a/conda/recipes/libcudf/install_libcudf.sh b/conda/recipes/libcudf/install_libcudf.sh deleted file mode 100644 index 173f8cfa90f..00000000000 --- a/conda/recipes/libcudf/install_libcudf.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/bin/bash -# Copyright (c) 2018-2022, NVIDIA CORPORATION. - -cmake --install cpp/build diff --git a/conda/recipes/libcudf/install_libcudf_example.sh b/conda/recipes/libcudf/install_libcudf_example.sh deleted file mode 100644 index 1a52dec99e3..00000000000 --- a/conda/recipes/libcudf/install_libcudf_example.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/bash -# Copyright (c) 2018-2024, NVIDIA CORPORATION. - -# build and install libcudf examples -./cpp/examples/build.sh --install diff --git a/conda/recipes/libcudf/install_libcudf_kafka.sh b/conda/recipes/libcudf/install_libcudf_kafka.sh deleted file mode 100644 index 9eae2510027..00000000000 --- a/conda/recipes/libcudf/install_libcudf_kafka.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/bin/bash -# Copyright (c) 2018-2022, NVIDIA CORPORATION. - -cmake --install cpp/libcudf_kafka/build diff --git a/conda/recipes/libcudf/install_libcudf_tests.sh b/conda/recipes/libcudf/install_libcudf_tests.sh deleted file mode 100644 index 069462eec9d..00000000000 --- a/conda/recipes/libcudf/install_libcudf_tests.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/bash -# Copyright (c) 2018-2022, NVIDIA CORPORATION. - -cmake --install cpp/build --component testing -cmake --install cpp/libcudf_kafka/build --component testing diff --git a/conda/recipes/libcudf/meta.yaml b/conda/recipes/libcudf/meta.yaml deleted file mode 100644 index f7bd7280f0f..00000000000 --- a/conda/recipes/libcudf/meta.yaml +++ /dev/null @@ -1,220 +0,0 @@ -# Copyright (c) 2018-2025, NVIDIA CORPORATION. - -{% set version = environ['RAPIDS_PACKAGE_VERSION'].lstrip('v') %} -{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %} -{% set cuda_version = '.'.join(environ['RAPIDS_CUDA_VERSION'].split('.')[:2]) %} -{% set cuda_major = cuda_version.split('.')[0] %} -{% set date_string = environ['RAPIDS_DATE_STRING'] %} - -package: - name: libcudf-split - -source: - path: ../../.. - -build: - script_env: - - AWS_ACCESS_KEY_ID - - AWS_SECRET_ACCESS_KEY - - AWS_SESSION_TOKEN - - CMAKE_C_COMPILER_LAUNCHER - - CMAKE_CUDA_COMPILER_LAUNCHER - - CMAKE_CXX_COMPILER_LAUNCHER - - CMAKE_GENERATOR - - PARALLEL_LEVEL - - RAPIDS_ARTIFACTS_DIR - - SCCACHE_BUCKET - - SCCACHE_IDLE_TIMEOUT - - SCCACHE_REGION - - SCCACHE_S3_KEY_PREFIX=libcudf-aarch64 # [aarch64] - - SCCACHE_S3_KEY_PREFIX=libcudf-linux64 # [linux64] - - SCCACHE_S3_USE_SSL - - SCCACHE_S3_NO_CREDENTIALS - -requirements: - build: - - cmake {{ cmake_version }} - - {{ compiler('c') }} - - {{ compiler('cxx') }} - {% if cuda_major == "11" %} - - {{ compiler('cuda') }} ={{ cuda_version }} - {% else %} - - {{ compiler('cuda') }} - {% endif %} - - cuda-version ={{ cuda_version }} - - ninja - - {{ stdlib("c") }} - host: - - librmm ={{ minor_version }} - - libkvikio ={{ minor_version }} - {% if cuda_major == "11" %} - - cudatoolkit - - libcufile {{ cuda11_libcufile_host_version }} # [linux64] - - libcufile-dev {{ cuda11_libcufile_host_version }} # [linux64] - - libcurand {{ cuda11_libcurand_host_version }} - - libcurand-dev {{ cuda11_libcurand_host_version }} - - cuda-nvrtc ={{ cuda_version }} - - cuda-nvrtc-dev ={{ cuda_version }} - - cuda-nvtx ={{ cuda_version }} - {% else %} - - cuda-nvrtc-dev - - cuda-nvtx-dev - - libcufile-dev # [linux64] - - libcurand-dev - {% endif %} - - cuda-version ={{ cuda_version }} - - nvcomp {{ nvcomp_version }} - - dlpack {{ dlpack_version }} - - librdkafka {{ librdkafka_version }} - - flatbuffers {{ flatbuffers_version }} - - rapids-logger =0.1 - - zlib {{ zlib_version }} - -outputs: - - name: libcudf - version: {{ version }} - script: install_libcudf.sh - build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - run_exports: - - {{ pin_subpackage("libcudf", max_pin="x.x") }} - ignore_run_exports_from: - - {{ compiler('cuda') }} - requirements: - build: - - cmake {{ cmake_version }} - host: - - cuda-version ={{ cuda_version }} - run: - - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - {% if cuda_major == "11" %} - - cudatoolkit - - libcufile {{ cuda11_libcufile_run_version }} # [linux64] - {% else %} - - cuda-nvrtc - - libcufile # [linux64] - {% endif %} - - nvcomp {{ nvcomp_version }} - - librmm ={{ minor_version }} - - libkvikio ={{ minor_version }} - - dlpack {{ dlpack_version }} - - rapids-logger =0.1 - test: - commands: - - test -f $PREFIX/lib/libcudf.so - - test -f $PREFIX/include/cudf/column/column.hpp - about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: libcudf library - - name: libcudf_kafka - version: {{ version }} - script: install_libcudf_kafka.sh - build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - ignore_run_exports_from: - - {{ compiler('cuda') }} - requirements: - build: - - cmake {{ cmake_version }} - host: - - librdkafka {{ librdkafka_version }} - - {{ pin_subpackage('libcudf', exact=True) }} - run: - - librdkafka {{ librdkafka_version }} - - {{ pin_subpackage('libcudf', exact=True) }} - test: - commands: - - test -f $PREFIX/lib/libcudf_kafka.so - about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: libcudf_kafka library - - name: libcudf-example - version: {{ version }} - script: install_libcudf_example.sh - build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - ignore_run_exports_from: - - {{ compiler('cuda') }} - {% if cuda_major != "11" %} - - cuda-nvtx-dev - {% endif %} - requirements: - build: - - cmake {{ cmake_version }} - - {{ compiler('c') }} - - {{ compiler('cxx') }} - {% if cuda_major == "11" %} - - {{ compiler('cuda') }} ={{ cuda_version }} - {% else %} - - {{ compiler('cuda') }} - {% endif %} - - cuda-version ={{ cuda_version }} - - ninja - - {{ stdlib("c") }} - host: - - {{ pin_subpackage('libcudf', exact=True) }} - {% if cuda_major == "11" %} - - cuda-nvtx ={{ cuda_version }} - {% else %} - - cuda-nvtx-dev - {% endif %} - - cuda-version ={{ cuda_version }} - run: - - {{ pin_subpackage('libcudf', exact=True) }} - - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - {% if cuda_major != "11" %} - - cuda-nvtx - {% endif %} - about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: libcudf example executables - - name: libcudf-tests - version: {{ version }} - script: install_libcudf_tests.sh - build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - ignore_run_exports_from: - - {{ compiler('cuda') }} - {% if cuda_major != "11" %} - - libcurand-dev - {% endif %} - requirements: - build: - - cmake {{ cmake_version }} - host: - - {{ pin_subpackage('libcudf', exact=True) }} - - {{ pin_subpackage('libcudf_kafka', exact=True) }} - - cuda-version ={{ cuda_version }} - {% if cuda_major == "11" %} - - libcurand {{ cuda11_libcurand_run_version }} - {% else %} - - libcurand-dev - {% endif %} - run: - - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - - {{ pin_subpackage('libcudf', exact=True) }} - - {{ pin_subpackage('libcudf_kafka', exact=True) }} - {% if cuda_major == "11" %} - - libcurand {{ cuda11_libcurand_run_version }} - {% else %} - - libcurand - {% endif %} - about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: libcudf test & benchmark executables diff --git a/conda/recipes/libcudf/recipe.yaml b/conda/recipes/libcudf/recipe.yaml new file mode 100644 index 00000000000..8653dc68a9f --- /dev/null +++ b/conda/recipes/libcudf/recipe.yaml @@ -0,0 +1,323 @@ +# Copyright (c) 2018-2025, NVIDIA CORPORATION. +schema_version: 1 + +context: + version: ${{ env.get("RAPIDS_PACKAGE_VERSION") }} + minor_version: ${{ (version | split("."))[:2] | join(".") }} + cuda_version: ${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[:2] | join(".") }} + cuda_major: '${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[0] }}' + date_string: '${{ env.get("RAPIDS_DATE_STRING") }}' + head_rev: ${{ git.head_rev(".")[:8] }} + +recipe: + name: libcudf-split + +cache: + source: + path: ../../.. + + build: + script: + content: | + + # Remove `-fdebug-prefix-map` line from CFLAGS and CXXFLAGS so the + # incrementing version number in the compile line doesn't break the + # cache + set -x + export CFLAGS=$(echo $CFLAGS | sed -E 's@\-fdebug\-prefix\-map[^ ]*@@g') + export CXXFLAGS=$(echo $CXXFLAGS | sed -E 's@\-fdebug\-prefix\-map[^ ]*@@g') + set +x + + ./build.sh -n -v \ + libcudf libcudf_kafka benchmarks tests \ + --build_metrics --incl_cache_stats --allgpuarch \ + --cmake-args=\"-DCUDF_ENABLE_ARROW_S3=ON\" + secrets: + - AWS_ACCESS_KEY_ID + - AWS_SECRET_ACCESS_KEY + - AWS_SESSION_TOKEN + env: + CMAKE_C_COMPILER_LAUNCHER: ${{ env.get("CMAKE_C_COMPILER_LAUNCHER") }} + CMAKE_CUDA_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CUDA_COMPILER_LAUNCHER") }} + CMAKE_CXX_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CXX_COMPILER_LAUNCHER") }} + CMAKE_GENERATOR: ${{ env.get("CMAKE_GENERATOR") }} + PARALLEL_LEVEL: ${{ env.get("PARALLEL_LEVEL") }} + RAPIDS_ARTIFACTS_DIR: ${{ env.get("RAPIDS_ARTIFACTS_DIR") }} + SCCACHE_BUCKET: ${{ env.get("SCCACHE_BUCKET") }} + SCCACHE_IDLE_TIMEOUT: ${{ env.get("SCCACHE_IDLE_TIMEOUT") }} + SCCACHE_REGION: ${{ env.get("SCCACHE_REGION") }} + SCCACHE_S3_USE_SSL: ${{ env.get("SCCACHE_S3_USE_SSL") }} + SCCACHE_S3_NO_CREDENTIALS: ${{ env.get("SCCACHE_S3_NO_CREDENTIALS") }} + SCCACHE_S3_KEY_PREFIX: libcudf-${{ env.get("RAPIDS_CONDA_ARCH") }} + + requirements: + build: + - ${{ compiler("c") }} + - ${{ compiler("cxx") }} + - ${{ compiler("cuda") }} + - cuda-version =${{ cuda_version }} + - cmake ${{ cmake_version }} + - ninja + - ${{ stdlib("c") }} + host: + - librmm =${{ minor_version }} + - libkvikio =${{ minor_version }} + - if: cuda_major == "11" + then: + - cudatoolkit + - libcurand =${{ cuda11_libcurand_host_version }} + - libcurand-dev =${{ cuda11_libcurand_host_version }} + - cuda-nvrtc =${{ cuda_version }} + - cuda-nvrtc-dev =${{ cuda_version }} + - cuda-nvtx =${{ cuda_version }} + - if: linux64 + then: + - libcufile =${{ cuda11_libcufile_host_version }} + - libcufile-dev =${{ cuda11_libcufile_host_version }} + else: + - cuda-nvrtc-dev + - cuda-nvtx-dev + - libcurand-dev + - if: linux64 + then: + - libcufile-dev + - cuda-version =${{ cuda_version }} + - nvcomp ${{ nvcomp_version }} + - dlpack ${{ dlpack_version }} + - librdkafka ${{ librdkafka_version }} + - flatbuffers =${{ flatbuffers_version }} + - rapids-logger =0.1 + - zlib ${{ zlib_version }} + +outputs: + - package: + name: libcudf + version: ${{ version }} + build: + script: + - cmake --install cpp/build + string: cuda${{ cuda_major }}_${{ date_string }}_${{ head_rev }} + dynamic_linking: + overlinking_behavior: "error" + requirements: + build: + - cmake ${{ cmake_version }} + - ${{ compiler("c") }} + host: + - cuda-version =${{ cuda_version }} + - libkvikio =${{ minor_version }} + - nvcomp ${{ nvcomp_version }} + - rapids-logger =0.1 + - zlib ${{ zlib_version }} + - if: cuda_major == "11" + then: cudatoolkit + else: cuda-cudart-dev + run: + - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }} + - if: cuda_major == "11" + then: + - cudatoolkit + - if: linux64 + then: + - libcufile ${{ cuda11_libcufile_run_version }} + else: + - cuda-nvrtc + - if: linux64 + then: + - libcufile + - nvcomp ${{ nvcomp_version }} + - librmm =${{ minor_version }} + - libkvikio =${{ minor_version }} + - dlpack ${{ dlpack_version }} + - rapids-logger =0.1 + run_exports: + - ${{ pin_subpackage("libcudf", upper_bound="x.x") }} + ignore_run_exports: + by_name: + - cuda-cudart + - cuda-nvrtc + - cuda-nvtx + - cuda-version + - flatbuffers + - libcurand + - libkvikio + - librdkafka + - librmm + - nvcomp + tests: + - script: + - test -f $PREFIX/lib/libcudf.so + - test -f $PREFIX/include/cudf/column/column.hpp + about: + homepage: ${{ load_from_file("python/libcudf/pyproject.toml").project.urls.Homepage }} + license: ${{ load_from_file("python/libcudf/pyproject.toml").project.license.text }} + summary: ${{ load_from_file("python/libcudf/pyproject.toml").project.description }} + + - package: + name: libcudf_kafka + version: ${{ version }} + build: + script: + - cmake --install cpp/libcudf_kafka/build + string: cuda${{ cuda_major }}_${{ date_string }}_${{ head_rev }} + dynamic_linking: + overlinking_behavior: "error" + requirements: + build: + - cmake ${{ cmake_version }} + - ${{ stdlib("c") }} + host: + - librdkafka ${{ librdkafka_version }} + - ${{ pin_subpackage("libcudf", exact=True) }} + run: + - librdkafka ${{ librdkafka_version }} + - ${{ pin_subpackage("libcudf", exact=True) }} + ignore_run_exports: + by_name: + - cuda-cudart + - cuda-nvrtc + - cuda-nvtx + - cuda-version + - flatbuffers + - libcurand + - libkvikio + - librdkafka + - librmm + - nvcomp + tests: + - script: + - test -f $PREFIX/lib/libcudf_kafka.so + about: + homepage: https://rapids.ai/ + license: Apache-2.0 + summary: libcudf_kafka library + + - package: + name: libcudf-example + version: ${{ version }} + build: + script: + content: | + ./cpp/examples/build.sh --install + secrets: + - AWS_ACCESS_KEY_ID + - AWS_SECRET_ACCESS_KEY + - AWS_SESSION_TOKEN + env: + CMAKE_C_COMPILER_LAUNCHER: ${{ env.get("CMAKE_C_COMPILER_LAUNCHER") }} + CMAKE_CUDA_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CUDA_COMPILER_LAUNCHER") }} + CMAKE_CXX_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CXX_COMPILER_LAUNCHER") }} + CMAKE_GENERATOR: ${{ env.get("CMAKE_GENERATOR") }} + PARALLEL_LEVEL: ${{ env.get("PARALLEL_LEVEL") }} + RAPIDS_ARTIFACTS_DIR: ${{ env.get("RAPIDS_ARTIFACTS_DIR") }} + SCCACHE_BUCKET: ${{ env.get("SCCACHE_BUCKET") }} + SCCACHE_IDLE_TIMEOUT: ${{ env.get("SCCACHE_IDLE_TIMEOUT") }} + SCCACHE_REGION: ${{ env.get("SCCACHE_REGION") }} + SCCACHE_S3_USE_SSL: ${{ env.get("SCCACHE_S3_USE_SSL") }} + SCCACHE_S3_NO_CREDENTIALS: ${{ env.get("SCCACHE_S3_NO_CREDENTIALS") }} + SCCACHE_S3_KEY_PREFIX: libcudf-${{ env.get("RAPIDS_CONDA_ARCH") }} + string: cuda${{ cuda_major }}_${{ date_string }}_${{ head_rev }} + dynamic_linking: + overlinking_behavior: "error" + requirements: + build: + - ${{ compiler("c") }} + - ${{ compiler("cuda") }} + - ${{ compiler("cxx") }} + - ${{ stdlib("c") }} + - cmake ${{ cmake_version }} + - cuda-version =${{ cuda_version }} + - ninja + host: + - ${{ pin_subpackage("libcudf", exact=True) }} + - cuda-version =${{ cuda_version }} + - if: cuda_major == "11" + then: + - cuda-nvtx =${{ cuda_version }} + - cudatoolkit + else: + - cuda-nvtx-dev + - cuda-cudart-dev + run: + - ${{ pin_subpackage("libcudf", exact=True) }} + - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }} + - if: cuda_major != "11" + then: + - cuda-nvtx + ignore_run_exports: + from_package: + - if: cuda_major != "11" + then: + - cuda-nvtx-dev + by_name: + - cuda-cudart + - cuda-nvrtc + - cuda-nvtx + - cuda-version + - flatbuffers + - libcurand + - libkvikio + - librdkafka + - librmm + - nvcomp + about: + homepage: ${{ load_from_file("python/libcudf/pyproject.toml").project.urls.Homepage }} + license: ${{ load_from_file("python/libcudf/pyproject.toml").project.license.text }} + summary: libcudf example executables + + - package: + name: libcudf-tests + version: ${{ version }} + build: + script: + - cmake --install cpp/build --component testing + - cmake --install cpp/libcudf_kafka/build --component testing + string: cuda${{ cuda_major }}_${{ date_string }}_${{ head_rev }} + dynamic_linking: + overlinking_behavior: "error" + missing_dso_allowlist: + - "libnvidia-ml.so.1" + requirements: + build: + - cmake ${{ cmake_version }} + - ${{ stdlib("c") }} + host: + - ${{ pin_subpackage("libcudf", exact=True) }} + - ${{ pin_subpackage("libcudf_kafka", exact=True) }} + - cuda-version =${{ cuda_version }} + - if: cuda_major == "11" + then: + - libcurand ${{ cuda11_libcurand_run_version }} + - cudatoolkit + else: + - libcurand-dev + - cuda-cudart-dev + run: + - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }} + - ${{ pin_subpackage("libcudf", exact=True) }} + - ${{ pin_subpackage("libcudf_kafka", exact=True) }} + - if: cuda_major == "11" + then: + - libcurand ${{ cuda11_libcurand_run_version }} + else: + - libcurand + ignore_run_exports: + from_package: + - if: cuda_major != "11" + then: + - libcurand-dev + by_name: + - cuda-cudart + - cuda-nvrtc + - cuda-nvtx + - cuda-version + - flatbuffers + - libcurand + - libkvikio + - librdkafka + - librmm + - nvcomp + about: + homepage: ${{ load_from_file("python/libcudf/pyproject.toml").project.urls.Homepage }} + license: ${{ load_from_file("python/libcudf/pyproject.toml").project.license.text }} + summary: libcudf test & benchmark executables diff --git a/conda/recipes/pylibcudf/build.sh b/conda/recipes/pylibcudf/build.sh deleted file mode 100644 index 483346504db..00000000000 --- a/conda/recipes/pylibcudf/build.sh +++ /dev/null @@ -1,4 +0,0 @@ -# Copyright (c) 2018-2024, NVIDIA CORPORATION. - -# This assumes the script is executed from the root of the repo directory -./build.sh pylibcudf diff --git a/conda/recipes/pylibcudf/meta.yaml b/conda/recipes/pylibcudf/meta.yaml deleted file mode 100644 index ae02cf8d4e5..00000000000 --- a/conda/recipes/pylibcudf/meta.yaml +++ /dev/null @@ -1,100 +0,0 @@ -# Copyright (c) 2018-2025, NVIDIA CORPORATION. - -{% set version = environ['RAPIDS_PACKAGE_VERSION'].lstrip('v') %} -{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %} -{% set py_version = environ['CONDA_PY'] %} -{% set cuda_version = '.'.join(environ['RAPIDS_CUDA_VERSION'].split('.')[:2]) %} -{% set cuda_major = cuda_version.split('.')[0] %} -{% set date_string = environ['RAPIDS_DATE_STRING'] %} - -package: - name: pylibcudf - version: {{ version }} - -source: - path: ../../.. - -build: - number: {{ GIT_DESCRIBE_NUMBER }} - string: cuda{{ cuda_major }}_py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }} - script_env: - - AWS_ACCESS_KEY_ID - - AWS_SECRET_ACCESS_KEY - - AWS_SESSION_TOKEN - - CMAKE_C_COMPILER_LAUNCHER - - CMAKE_CUDA_COMPILER_LAUNCHER - - CMAKE_CXX_COMPILER_LAUNCHER - - CMAKE_GENERATOR - - PARALLEL_LEVEL - - SCCACHE_BUCKET - - SCCACHE_IDLE_TIMEOUT - - SCCACHE_REGION - - SCCACHE_S3_KEY_PREFIX=pylibcudf-aarch64 # [aarch64] - - SCCACHE_S3_KEY_PREFIX=pylibcudf-linux64 # [linux64] - - SCCACHE_S3_USE_SSL - - SCCACHE_S3_NO_CREDENTIALS - ignore_run_exports_from: - - {{ compiler('cuda') }} - {% if cuda_major != "11" %} - - cuda-cudart-dev - - libcufile-dev # [linux64] - {% endif %} - -requirements: - build: - - cmake {{ cmake_version }} - - ninja - - {{ compiler('c') }} - - {{ compiler('cxx') }} - {% if cuda_major == "11" %} - - {{ compiler('cuda') }} ={{ cuda_version }} - {% else %} - - {{ compiler('cuda') }} - {% endif %} - - cuda-version ={{ cuda_version }} - - {{ stdlib("c") }} - host: - - python - - cython >=3.0.3 - - rapids-build-backend >=0.3.0,<0.4.0.dev0 - - scikit-build-core >=0.10.0 - - dlpack >=0.8,<1.0 - - libcudf ={{ version }} - - rmm ={{ minor_version }} - {% if cuda_major == "11" %} - - cudatoolkit - {% else %} - - cuda-cudart-dev - - cuda-nvrtc - - libcufile-dev # [linux64] - {% endif %} - - cuda-version ={{ cuda_version }} - run: - - python - - typing_extensions >=4.0.0 - - pandas >=2.0,<2.2.4dev0 - - numpy >=1.23,<2.1 - - pyarrow>=14.0.0,<20.0.0a0 - - libcudf ={{ version }} - - {{ pin_compatible('rmm', max_pin='x.x') }} - - fsspec >=0.6.0 - {% if cuda_major == "11" %} - - cuda-python >=11.8.5,<12.0a0 - {% else %} - - cuda-python >=12.6.2,<13.0a0 - {% endif %} - - nvtx >=0.2.1 - - packaging - -test: - requires: - - cuda-version ={{ cuda_version }} - imports: - - pylibcudf - -about: - home: https://rapids.ai/ - license: Apache-2.0 - license_family: APACHE - license_file: LICENSE - summary: pylibcudf library diff --git a/conda/recipes/pylibcudf/recipe.yaml b/conda/recipes/pylibcudf/recipe.yaml new file mode 100644 index 00000000000..476f4d83960 --- /dev/null +++ b/conda/recipes/pylibcudf/recipe.yaml @@ -0,0 +1,106 @@ +# Copyright (c) 2018-2025, NVIDIA CORPORATION. +schema_version: 1 + +context: + version: ${{ env.get("RAPIDS_PACKAGE_VERSION") }} + minor_version: ${{ (version | split("."))[:2] | join(".") }} + cuda_version: ${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[:2] | join(".") }} + cuda_major: '${{ (env.get("RAPIDS_CUDA_VERSION") | split("."))[0] }}' + date_string: '${{ env.get("RAPIDS_DATE_STRING") }}' + py_version: ${{ env.get("RAPIDS_PY_VERSION") }} + py_buildstring: ${{ py_version | version_to_buildstring }} + head_rev: ${{ git.head_rev(".")[:8] }} + +package: + name: pylibcudf + version: ${{ version }} + +source: + path: ../../.. + +build: + string: cuda${{ cuda_major }}_py${{ py_buildstring }}_${{ date_string }}_${{ head_rev }} + script: + content: | + ./build.sh pylibcudf + secrets: + - AWS_ACCESS_KEY_ID + - AWS_SECRET_ACCESS_KEY + - AWS_SESSION_TOKEN + env: + CMAKE_C_COMPILER_LAUNCHER: ${{ env.get("CMAKE_C_COMPILER_LAUNCHER") }} + CMAKE_CUDA_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CUDA_COMPILER_LAUNCHER") }} + CMAKE_CXX_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CXX_COMPILER_LAUNCHER") }} + CMAKE_GENERATOR: ${{ env.get("CMAKE_GENERATOR") }} + SCCACHE_BUCKET: ${{ env.get("SCCACHE_BUCKET") }} + SCCACHE_IDLE_TIMEOUT: ${{ env.get("SCCACHE_IDLE_TIMEOUT") }} + SCCACHE_REGION: ${{ env.get("SCCACHE_REGION") }} + SCCACHE_S3_USE_SSL: ${{ env.get("SCCACHE_S3_USE_SSL") }} + SCCACHE_S3_NO_CREDENTIALS: ${{ env.get("SCCACHE_S3_NO_CREDENTIALS") }} + SCCACHE_S3_KEY_PREFIX: pylibcudf-${{ env.get("RAPIDS_CONDA_ARCH") }} + +requirements: + build: + - cmake ${{ cmake_version }} + - ninja + - ${{ compiler("c") }} + - ${{ compiler("cxx") }} + - ${{ compiler("cuda") }} + - cuda-version =${{ cuda_version }} + - ${{ stdlib("c") }} + host: + - python =${{ py_version }} + - pip + - cython >=3.0.3 + - rapids-build-backend >=0.3.0,<0.4.0.dev0 + - scikit-build-core >=0.10.0 + - dlpack >=0.8,<1.0 + - libcudf =${{ version }} + - rmm =${{ minor_version }} + - if: cuda_major == "11" + then: + - cudatoolkit + else: + - cuda-cudart-dev + - cuda-nvrtc + - if: linux64 + then: + - libcufile-dev + - cuda-version =${{ cuda_version }} + run: + - python + - typing_extensions >=4.0.0 + - pandas >=2.0,<2.2.4dev0 + - numpy >=1.23,<2.1 + - pyarrow>=14.0.0,<20.0.0a0 + - libcudf =${{ version }} + - ${{ pin_compatible("rmm", upper_bound="x.x") }} + - fsspec >=0.6.0 + - if: cuda_major == "11" + then: + - cuda-python >=11.8.5,<12.0a0 + else: + - cuda-python >=12.6.2,<13.0a0 + - nvtx >=0.2.1 + - packaging + ignore_run_exports: + from_package: + - if: cuda_major != "11" + then: + - cuda-cudart-dev + - if: linux64 + then: + - libcufile-dev + by_name: + - cuda-version + +tests: + - python: + imports: + - pylibcudf + pip_check: false + +about: + homepage: ${{ load_from_file("python/pylibcudf/pyproject.toml").project.urls.Homepage }} + license: ${{ load_from_file("python/pylibcudf/pyproject.toml").project.license.text }} + summary: ${{ load_from_file("python/pylibcudf/pyproject.toml").project.description }} diff --git a/python/cudf/pyproject.toml b/python/cudf/pyproject.toml index 8b8abe90ac9..2ce5131ea8e 100644 --- a/python/cudf/pyproject.toml +++ b/python/cudf/pyproject.toml @@ -15,7 +15,7 @@ readme = { file = "README.md", content-type = "text/markdown" } authors = [ { name = "NVIDIA Corporation" }, ] -license = { text = "Apache 2.0" } +license = { text = "Apache-2.0" } requires-python = ">=3.10" dependencies = [ "cachetools", diff --git a/python/cudf_kafka/pyproject.toml b/python/cudf_kafka/pyproject.toml index 424010e632c..764c8c64a7e 100644 --- a/python/cudf_kafka/pyproject.toml +++ b/python/cudf_kafka/pyproject.toml @@ -15,7 +15,7 @@ readme = { file = "README.md", content-type = "text/markdown" } authors = [ { name = "NVIDIA Corporation" }, ] -license = { text = "Apache 2.0" } +license = { text = "Apache-2.0" } requires-python = ">=3.10" dependencies = [ "cudf==25.4.*,>=0.0.0a0", diff --git a/python/cudf_polars/pyproject.toml b/python/cudf_polars/pyproject.toml index e9fc054efc2..fb44caaa0c0 100644 --- a/python/cudf_polars/pyproject.toml +++ b/python/cudf_polars/pyproject.toml @@ -16,7 +16,7 @@ readme = { file = "README.md", content-type = "text/markdown" } authors = [ { name = "NVIDIA Corporation" }, ] -license = { text = "Apache 2.0" } +license = { text = "Apache-2.0" } requires-python = ">=3.10" dependencies = [ "polars>=1.20,<1.24", diff --git a/python/custreamz/pyproject.toml b/python/custreamz/pyproject.toml index 665b0a76ecf..b1fbe901189 100644 --- a/python/custreamz/pyproject.toml +++ b/python/custreamz/pyproject.toml @@ -16,7 +16,7 @@ readme = { file = "README.md", content-type = "text/markdown" } authors = [ { name = "NVIDIA Corporation" }, ] -license = { text = "Apache 2.0" } +license = { text = "Apache-2.0" } requires-python = ">=3.10" dependencies = [ "confluent-kafka>=2.5.0,<2.6.0a0", diff --git a/python/dask_cudf/pyproject.toml b/python/dask_cudf/pyproject.toml index 83493d7f2a4..fd2bac3c0d2 100644 --- a/python/dask_cudf/pyproject.toml +++ b/python/dask_cudf/pyproject.toml @@ -16,7 +16,7 @@ readme = { file = "README.md", content-type = "text/markdown" } authors = [ { name = "NVIDIA Corporation" }, ] -license = { text = "Apache 2.0" } +license = { text = "Apache-2.0" } requires-python = ">=3.10" dependencies = [ "cudf==25.4.*,>=0.0.0a0", diff --git a/python/libcudf/pyproject.toml b/python/libcudf/pyproject.toml index 01fe6097936..784a0c49894 100644 --- a/python/libcudf/pyproject.toml +++ b/python/libcudf/pyproject.toml @@ -27,7 +27,7 @@ readme = { file = "README.md", content-type = "text/markdown" } authors = [ { name = "NVIDIA Corporation" }, ] -license = { text = "Apache 2.0" } +license = { text = "Apache-2.0" } requires-python = ">=3.10" classifiers = [ "Intended Audience :: Developers", diff --git a/python/pylibcudf/pyproject.toml b/python/pylibcudf/pyproject.toml index e12d1ffdb39..8ea6f0e94a4 100644 --- a/python/pylibcudf/pyproject.toml +++ b/python/pylibcudf/pyproject.toml @@ -15,7 +15,7 @@ readme = { file = "README.md", content-type = "text/markdown" } authors = [ { name = "NVIDIA Corporation" }, ] -license = { text = "Apache 2.0" } +license = { text = "Apache-2.0" } requires-python = ">=3.10" dependencies = [ "cuda-python>=11.8.5,<12.0a0", From 45d80669367c6bf3b9dc0cd122f0ea36072cb7ea Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Mon, 3 Mar 2025 21:25:11 -0800 Subject: [PATCH 120/129] Remove cudf.Scalar from shift/fillna (#17922) Toward https://github.com/rapidsai/cudf/issues/17843 Authors: - Matthew Roeschke (https://github.com/mroeschke) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/17922 --- python/cudf/cudf/core/column/categorical.py | 9 +++++-- python/cudf/cudf/core/column/column.py | 23 +++++++++++------ python/cudf/cudf/core/column/datetime.py | 14 +++++++++++ python/cudf/cudf/core/column/decimal.py | 28 ++++++++++++++++++--- python/cudf/cudf/core/column/numerical.py | 13 +++++++--- python/cudf/cudf/core/column/timedelta.py | 15 +++++++++++ 6 files changed, 84 insertions(+), 18 deletions(-) diff --git a/python/cudf/cudf/core/column/categorical.py b/python/cudf/cudf/core/column/categorical.py index d41e448254c..c75d285e7de 100644 --- a/python/cudf/cudf/core/column/categorical.py +++ b/python/cudf/cudf/core/column/categorical.py @@ -20,6 +20,7 @@ from cudf.core.scalar import pa_scalar_to_plc_scalar from cudf.utils.dtypes import ( SIZE_TYPE_DTYPE, + cudf_dtype_to_pa_type, find_common_type, is_mixed_with_object_dtype, min_signed_type, @@ -1042,7 +1043,7 @@ def notnull(self) -> ColumnBase: def _validate_fillna_value( self, fill_value: ScalarLike | ColumnLike - ) -> cudf.Scalar | ColumnBase: + ) -> plc.Scalar | ColumnBase: """Align fill_value for .fillna based on column type.""" if cudf.api.types.is_scalar(fill_value): if fill_value != _DEFAULT_CATEGORICAL_VALUE: @@ -1052,7 +1053,11 @@ def _validate_fillna_value( raise ValueError( f"{fill_value=} must be in categories" ) from err - return cudf.Scalar(fill_value, dtype=self.codes.dtype) + return pa_scalar_to_plc_scalar( + pa.scalar( + fill_value, type=cudf_dtype_to_pa_type(self.codes.dtype) + ) + ) else: fill_value = column.as_column(fill_value, nan_as_null=False) if isinstance(fill_value.dtype, CategoricalDtype): diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index 61f4f7d52fb..0d36fd3855b 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -891,12 +891,11 @@ def _fill( @acquire_spill_lock() def shift(self, offset: int, fill_value: ScalarLike) -> Self: - if not isinstance(fill_value, cudf.Scalar): - fill_value = cudf.Scalar(fill_value, dtype=self.dtype) + plc_fill_value = self._scalar_to_plc_scalar(fill_value) plc_col = plc.copying.shift( self.to_pylibcudf(mode="read"), offset, - fill_value.device_value, + plc_fill_value, ) return type(self).from_pylibcudf(plc_col) # type: ignore[return-value] @@ -1188,13 +1187,21 @@ def _check_scatter_key_length( f"{num_keys}" ) + def _scalar_to_plc_scalar(self, scalar: ScalarLike) -> plc.Scalar: + """Return a pylibcudf.Scalar that matches the type of self.dtype""" + if not isinstance(scalar, pa.Scalar): + scalar = pa.scalar(scalar) + return pa_scalar_to_plc_scalar( + scalar.cast(cudf_dtype_to_pa_type(self.dtype)) + ) + def _validate_fillna_value( self, fill_value: ScalarLike | ColumnLike - ) -> cudf.Scalar | ColumnBase: + ) -> plc.Scalar | ColumnBase: """Align fill_value for .fillna based on column type.""" if is_scalar(fill_value): - return cudf.Scalar(fill_value, dtype=self.dtype) - return as_column(fill_value) + return self._scalar_to_plc_scalar(fill_value) + return as_column(fill_value).astype(self.dtype) @acquire_spill_lock() def replace( @@ -1240,8 +1247,8 @@ def fillna( if method == "ffill" else plc.replace.ReplacePolicy.FOLLOWING ) - elif is_scalar(fill_value): - plc_replace = cudf.Scalar(fill_value).device_value + elif isinstance(fill_value, plc.Scalar): + plc_replace = fill_value else: plc_replace = fill_value.to_pylibcudf(mode="read") plc_column = plc.replace.replace_nulls( diff --git a/python/cudf/cudf/core/column/datetime.py b/python/cudf/cudf/core/column/datetime.py index 213e91d7b3f..64ddcae72a7 100644 --- a/python/cudf/cudf/core/column/datetime.py +++ b/python/cudf/cudf/core/column/datetime.py @@ -45,6 +45,7 @@ from cudf._typing import ( ColumnBinaryOperand, + ColumnLike, DatetimeLikeScalar, Dtype, DtypeObj, @@ -269,6 +270,19 @@ def __contains__(self, item: ScalarLike) -> bool: "cudf.core.column.NumericalColumn", self.astype(np.dtype(np.int64)) ) + def _validate_fillna_value( + self, fill_value: ScalarLike | ColumnLike + ) -> plc.Scalar | ColumnBase: + """Align fill_value for .fillna based on column type.""" + if ( + isinstance(fill_value, np.datetime64) + and self.time_unit != np.datetime_data(fill_value)[0] + ): + fill_value = fill_value.astype(self.dtype) + elif isinstance(fill_value, str) and fill_value.lower() == "nat": + fill_value = np.datetime64(fill_value, self.time_unit) + return super()._validate_fillna_value(fill_value) + @functools.cached_property def time_unit(self) -> str: return np.datetime_data(self.dtype)[0] diff --git a/python/cudf/cudf/core/column/decimal.py b/python/cudf/cudf/core/column/decimal.py index 8db6f805bce..848faf6a9ee 100644 --- a/python/cudf/cudf/core/column/decimal.py +++ b/python/cudf/cudf/core/column/decimal.py @@ -24,7 +24,8 @@ DecimalDtype, ) from cudf.core.mixins import BinaryOperand -from cudf.utils.dtypes import CUDF_STRING_DTYPE +from cudf.core.scalar import pa_scalar_to_plc_scalar +from cudf.utils.dtypes import CUDF_STRING_DTYPE, cudf_dtype_to_pa_type from cudf.utils.utils import pa_mask_buffer_to_mask if TYPE_CHECKING: @@ -165,16 +166,35 @@ def _binaryop(self, other: ColumnBinaryOperand, op: str): return result + def _scalar_to_plc_scalar(self, scalar: ScalarLike) -> plc.Scalar: + """Return a pylibcudf.Scalar that matches the type of self.dtype""" + if not isinstance(scalar, pa.Scalar): + # e.g casting int to decimal type isn't allow, but OK in the constructor? + pa_scalar = pa.scalar( + scalar, type=cudf_dtype_to_pa_type(self.dtype) + ) + else: + pa_scalar = scalar.cast(cudf_dtype_to_pa_type(self.dtype)) + plc_scalar = pa_scalar_to_plc_scalar(pa_scalar) + if isinstance(self.dtype, (Decimal32Dtype, Decimal64Dtype)): + # pyarrow.Scalar only supports Decimal128 so conversion + # from pyarrow would only return a pylibcudf.Scalar with Decimal128 + col = ColumnBase.from_pylibcudf( + plc.Column.from_scalar(plc_scalar, 1) + ).astype(self.dtype) + return plc.copying.get_element(col.to_pylibcudf(mode="read"), 0) + return plc_scalar + def _validate_fillna_value( self, fill_value: ScalarLike | ColumnLike - ) -> cudf.Scalar | ColumnBase: + ) -> plc.Scalar | ColumnBase: """Align fill_value for .fillna based on column type.""" if isinstance(fill_value, (int, Decimal)): - return cudf.Scalar(fill_value, dtype=self.dtype) + return super()._validate_fillna_value(fill_value) elif isinstance(fill_value, ColumnBase) and ( isinstance(self.dtype, DecimalDtype) or self.dtype.kind in "iu" ): - return fill_value.astype(self.dtype) + return super()._validate_fillna_value(fill_value) raise TypeError( "Decimal columns only support using fillna with decimal and " "integer values" diff --git a/python/cudf/cudf/core/column/numerical.py b/python/cudf/cudf/core/column/numerical.py index eecb294acee..77c5a6b6caf 100644 --- a/python/cudf/cudf/core/column/numerical.py +++ b/python/cudf/cudf/core/column/numerical.py @@ -559,15 +559,20 @@ def find_and_replace( def _validate_fillna_value( self, fill_value: ScalarLike | ColumnLike - ) -> cudf.Scalar | ColumnBase: + ) -> plc.Scalar | ColumnBase: """Align fill_value for .fillna based on column type.""" if is_scalar(fill_value): - cudf_obj: cudf.Scalar | ColumnBase = cudf.Scalar(fill_value) - if not as_column(cudf_obj).can_cast_safely(self.dtype): + cudf_obj = ColumnBase.from_pylibcudf( + plc.Column.from_scalar( + pa_scalar_to_plc_scalar(pa.scalar(fill_value)), 1 + ) + ) + if not cudf_obj.can_cast_safely(self.dtype): raise TypeError( f"Cannot safely cast non-equivalent " f"{type(fill_value).__name__} to {self.dtype.name}" ) + return super()._validate_fillna_value(fill_value) else: cudf_obj = as_column(fill_value, nan_as_null=False) if not cudf_obj.can_cast_safely(self.dtype): # type: ignore[attr-defined] @@ -576,7 +581,7 @@ def _validate_fillna_value( f"{cudf_obj.dtype.type.__name__} to " f"{self.dtype.type.__name__}" ) - return cudf_obj.astype(self.dtype) + return cudf_obj.astype(self.dtype) def can_cast_safely(self, to_dtype: DtypeObj) -> bool: """ diff --git a/python/cudf/cudf/core/column/timedelta.py b/python/cudf/cudf/core/column/timedelta.py index e4d47f492c2..654d2c2b800 100644 --- a/python/cudf/cudf/core/column/timedelta.py +++ b/python/cudf/cudf/core/column/timedelta.py @@ -30,9 +30,11 @@ from cudf._typing import ( ColumnBinaryOperand, + ColumnLike, DatetimeLikeScalar, Dtype, DtypeObj, + ScalarLike, ) _unit_to_nanoseconds_conversion = { @@ -142,6 +144,19 @@ def __contains__(self, item: DatetimeLikeScalar) -> bool: "cudf.core.column.NumericalColumn", self.astype(np.dtype(np.int64)) ) + def _validate_fillna_value( + self, fill_value: ScalarLike | ColumnLike + ) -> plc.Scalar | ColumnBase: + """Align fill_value for .fillna based on column type.""" + if ( + isinstance(fill_value, np.timedelta64) + and self.time_unit != np.datetime_data(fill_value)[0] + ): + fill_value = fill_value.astype(self.dtype) + elif isinstance(fill_value, str) and fill_value.lower() == "nat": + fill_value = np.timedelta64(fill_value, self.time_unit) + return super()._validate_fillna_value(fill_value) + @property def values(self): """ From 8645992542792870cf2d1a1416c8994db83553b5 Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Mon, 3 Mar 2025 21:25:53 -0800 Subject: [PATCH 121/129] Add pylibcudf.Scalar.from_numpy for bool/int/float/str types (#18020) Towards https://github.com/rapidsai/cudf/issues/17054 Authors: - Matthew Roeschke (https://github.com/mroeschke) - Matthew Murray (https://github.com/Matt711) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18020 --- python/pylibcudf/pylibcudf/scalar.pyx | 148 +++++++++++++++++- .../pylibcudf/pylibcudf/tests/test_scalar.py | 42 +++++ 2 files changed, 189 insertions(+), 1 deletion(-) diff --git a/python/pylibcudf/pylibcudf/scalar.pyx b/python/pylibcudf/pylibcudf/scalar.pyx index 35abab7e838..e252d3072aa 100644 --- a/python/pylibcudf/pylibcudf/scalar.pyx +++ b/python/pylibcudf/pylibcudf/scalar.pyx @@ -2,7 +2,16 @@ from cpython cimport bool as py_bool, datetime from cython cimport no_gc_clear -from libc.stdint cimport int64_t +from libc.stdint cimport ( + int8_t, + int16_t, + int32_t, + int64_t, + uint8_t, + uint16_t, + uint32_t, + uint64_t, +) from libcpp cimport bool as cbool from libcpp.memory cimport unique_ptr from libcpp.utility cimport move @@ -25,6 +34,13 @@ from .types cimport DataType from functools import singledispatch +try: + import numpy as np + np_error = None +except ImportError as err: + np = None + np_error = err + __all__ = ["Scalar"] @@ -111,6 +127,24 @@ cdef class Scalar: """ return _from_py(py_val) + @classmethod + def from_numpy(cls, np_val): + """ + Convert a NumPy scalar to a Scalar. + + Parameters + ---------- + np_val: numpy.generic + Value to convert to a pylibcudf.Scalar + + Returns + ------- + Scalar + New pylibcudf.Scalar + """ + return _from_numpy(np_val) + + cdef Scalar _new_scalar(unique_ptr[scalar] c_obj, DataType dtype): cdef Scalar s = Scalar.__new__(Scalar) s.c_obj.swap(c_obj) @@ -166,3 +200,115 @@ def _(py_val): cdef unique_ptr[scalar] c_obj = make_string_scalar(py_val.encode()) cdef Scalar slr = _new_scalar(move(c_obj), dtype) return slr + + +@singledispatch +def _from_numpy(np_val): + if np_error is not None: + raise np_error + raise TypeError(f"{type(np_val).__name__} cannot be converted to pylibcudf.Scalar") + + +if np is not None: + @_from_numpy.register(np.datetime64) + @_from_numpy.register(np.timedelta64) + def _(np_val): + raise NotImplementedError( + f"{type(np_val).__name__} is currently not supported." + ) + + @_from_numpy.register(np.bool_) + def _(np_val): + cdef DataType dtype = DataType(type_id.BOOL8) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + cdef cbool c_val = np_val + (c_obj.get()).set_value(c_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.str_) + def _(np_val): + cdef DataType dtype = DataType(type_id.STRING) + cdef unique_ptr[scalar] c_obj = make_string_scalar(np_val.item().encode()) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.int8) + def _(np_val): + dtype = DataType(type_id.INT8) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.int16) + def _(np_val): + dtype = DataType(type_id.INT16) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.int32) + def _(np_val): + dtype = DataType(type_id.INT32) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.int64) + def _(np_val): + dtype = DataType(type_id.INT64) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.uint8) + def _(np_val): + dtype = DataType(type_id.UINT8) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.uint16) + def _(np_val): + dtype = DataType(type_id.UINT16) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.uint32) + def _(np_val): + dtype = DataType(type_id.UINT32) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.uint64) + def _(np_val): + dtype = DataType(type_id.UINT64) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.float32) + def _(np_val): + dtype = DataType(type_id.FLOAT32) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr + + @_from_numpy.register(np.float64) + def _(np_val): + dtype = DataType(type_id.FLOAT64) + cdef unique_ptr[scalar] c_obj = make_numeric_scalar(dtype.c_obj) + (c_obj.get()).set_value(np_val) + cdef Scalar slr = _new_scalar(move(c_obj), dtype) + return slr diff --git a/python/pylibcudf/pylibcudf/tests/test_scalar.py b/python/pylibcudf/pylibcudf/tests/test_scalar.py index 45afae91c9a..056fcd5f63c 100644 --- a/python/pylibcudf/pylibcudf/tests/test_scalar.py +++ b/python/pylibcudf/pylibcudf/tests/test_scalar.py @@ -7,6 +7,11 @@ import pylibcudf as plc +@pytest.fixture(scope="module") +def np(): + return pytest.importorskip("numpy") + + @pytest.mark.parametrize( "val", [True, False, -1, 0, 1 - 1.0, 0.0, 1.52, "", "a1!"] ) @@ -28,3 +33,40 @@ def test_from_py_notimplemented(val): def test_from_py_typeerror(val): with pytest.raises(TypeError): plc.Scalar.from_py(val) + + +@pytest.mark.parametrize( + "np_type", + [ + "bool_", + "str_", + "int8", + "int16", + "int32", + "int64", + "uint8", + "uint16", + "uint32", + "uint64", + "float32", + "float64", + ], +) +def test_from_numpy(np, np_type): + np_klass = getattr(np, np_type) + np_val = np_klass("1" if np_type == "str_" else 1) + result = plc.Scalar.from_numpy(np_val) + expected = pa.scalar(np_val) + assert plc.interop.to_arrow(result).equals(expected) + + +@pytest.mark.parametrize("np_type", ["datetime64", "timedelta64"]) +def test_from_numpy_notimplemented(np, np_type): + np_val = getattr(np, np_type)(1, "ns") + with pytest.raises(NotImplementedError): + plc.Scalar.from_numpy(np_val) + + +def test_from_numpy_typeerror(np): + with pytest.raises(TypeError): + plc.Scalar.from_numpy(np.void(5)) From c0c9dfe6ede37ed3d5160891fab747f9a0fab29a Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Mon, 3 Mar 2025 21:25:57 -0800 Subject: [PATCH 122/129] Use more, cheaper dtype checking utilities in cudf Python (#18139) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Avoids using potentially more expensive dtype checking utilities referenced in https://github.com/rapidsai/cudf/issues/12494 `is_string_dtype` -> `== CUDF_STRING_DTYPE` `is_decimal_dtype` -> `isinstance` `is_numeric_dtype` -> (new) `is_dtype_obj_numeric` ```python In [1]: import numpy as np In [2]: from cudf.api.types import is_numeric_dtype In [3]: from cudf.utils.dtypes import is_dtype_obj_numeric In [4]: dtype = np.dtype(np.int64) In [5]: %timeit is_dtype_obj_numeric(dtype) 211 ns ± 2.26 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each) In [6]: %timeit is_numeric_dtype(dtype) 1.14 μs ± 2.61 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each) ``` Also standardizes some imports from `cudf.api.types` Authors: - Matthew Roeschke (https://github.com/mroeschke) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/18139 --- python/cudf/cudf/api/types.py | 13 ------- python/cudf/cudf/core/_internals/where.py | 12 ++++-- python/cudf/cudf/core/column/categorical.py | 11 +++--- python/cudf/cudf/core/column/column.py | 16 ++++---- python/cudf/cudf/core/column/lists.py | 8 ++-- python/cudf/cudf/core/column/numerical.py | 4 +- python/cudf/cudf/core/column/string.py | 4 +- python/cudf/cudf/core/dataframe.py | 40 ++++++++++---------- python/cudf/cudf/core/groupby/groupby.py | 21 +++++----- python/cudf/cudf/core/index.py | 14 ++++--- python/cudf/cudf/core/indexed_frame.py | 28 +++++++------- python/cudf/cudf/core/join/_join_helpers.py | 18 ++++++--- python/cudf/cudf/core/multiindex.py | 22 +++++------ python/cudf/cudf/core/reshape.py | 4 +- python/cudf/cudf/core/scalar.py | 4 +- python/cudf/cudf/core/series.py | 6 ++- python/cudf/cudf/core/single_column_frame.py | 8 ++-- python/cudf/cudf/core/tools/datetimes.py | 2 +- python/cudf/cudf/core/tools/numeric.py | 13 ++++--- python/cudf/cudf/core/window/ewm.py | 4 +- python/cudf/cudf/io/dlpack.py | 7 ++-- python/cudf/cudf/testing/testing.py | 10 ++--- python/cudf/cudf/utils/dtypes.py | 14 +++++++ 23 files changed, 155 insertions(+), 128 deletions(-) diff --git a/python/cudf/cudf/api/types.py b/python/cudf/cudf/api/types.py index 37ef83c8820..8d7d64ab31e 100644 --- a/python/cudf/cudf/api/types.py +++ b/python/cudf/cudf/api/types.py @@ -73,19 +73,6 @@ def is_numeric_dtype(obj): return pd_types.is_numeric_dtype(obj) -# A version of numerical type check that does not include cudf decimals for -# places where we need to distinguish fixed and floating point numbers. -def _is_non_decimal_numeric_dtype(obj): - if isinstance(obj, _BaseDtype) or isinstance( - getattr(obj, "dtype", None), _BaseDtype - ): - return False - try: - return pd_types.is_numeric_dtype(obj) - except TypeError: - return False - - def is_integer(obj): """Return True if given object is integer. diff --git a/python/cudf/cudf/core/_internals/where.py b/python/cudf/cudf/core/_internals/where.py index 73011d6ffe0..cf49dfb2194 100644 --- a/python/cudf/cudf/core/_internals/where.py +++ b/python/cudf/cudf/core/_internals/where.py @@ -7,9 +7,13 @@ import numpy as np import cudf -from cudf.api.types import _is_non_decimal_numeric_dtype, is_scalar +from cudf.api.types import is_scalar from cudf.core.dtypes import CategoricalDtype -from cudf.utils.dtypes import find_common_type, is_mixed_with_object_dtype +from cudf.utils.dtypes import ( + find_common_type, + is_dtype_obj_numeric, + is_mixed_with_object_dtype, +) if TYPE_CHECKING: from cudf._typing import DtypeObj, ScalarLike @@ -18,7 +22,7 @@ def _normalize_categorical(input_col, other): if isinstance(input_col, cudf.core.column.CategoricalColumn): - if cudf.api.types.is_scalar(other): + if is_scalar(other): try: other = input_col._encode(other) except ValueError: @@ -81,7 +85,7 @@ def _check_and_cast_columns_with_other( ) return _normalize_categorical(source_col, other.astype(source_dtype)) - if _is_non_decimal_numeric_dtype(source_dtype) and as_column( + if is_dtype_obj_numeric(source_dtype, include_decimal=False) and as_column( other ).can_cast_safely(source_dtype): common_dtype = source_dtype diff --git a/python/cudf/cudf/core/column/categorical.py b/python/cudf/cudf/core/column/categorical.py index c75d285e7de..ed285934161 100644 --- a/python/cudf/cudf/core/column/categorical.py +++ b/python/cudf/cudf/core/column/categorical.py @@ -14,6 +14,7 @@ import pylibcudf as plc import cudf +from cudf.api.types import is_scalar from cudf.core.column import column from cudf.core.column.methods import ColumnMethods from cudf.core.dtypes import CategoricalDtype, IntervalDtype @@ -623,12 +624,10 @@ def ordered(self) -> bool: return self.dtype.ordered def __setitem__(self, key, value): - if cudf.api.types.is_scalar( - value - ) and cudf.utils.utils._is_null_host_scalar(value): + if is_scalar(value) and cudf.utils.utils._is_null_host_scalar(value): to_add_categories = 0 else: - if cudf.api.types.is_scalar(value): + if is_scalar(value): arr = column.as_column(value, length=1, nan_as_null=False) else: arr = column.as_column(value, nan_as_null=False) @@ -644,7 +643,7 @@ def __setitem__(self, key, value): "category, set the categories first" ) - if cudf.api.types.is_scalar(value): + if is_scalar(value): value = self._encode(value) if value is not None else value else: value = cudf.core.column.as_column(value).astype(self.dtype) @@ -1045,7 +1044,7 @@ def _validate_fillna_value( self, fill_value: ScalarLike | ColumnLike ) -> plc.Scalar | ColumnBase: """Align fill_value for .fillna based on column type.""" - if cudf.api.types.is_scalar(fill_value): + if is_scalar(fill_value): if fill_value != _DEFAULT_CATEGORICAL_VALUE: try: fill_value = self._encode(fill_value) diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index 0d36fd3855b..5a8064dc49d 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -23,13 +23,10 @@ import cudf from cudf.api.types import ( - _is_non_decimal_numeric_dtype, _is_pandas_nullable_extension_dtype, infer_dtype, - is_decimal_dtype, is_dtype_equal, is_scalar, - is_string_dtype, ) from cudf.core._compat import PANDAS_GE_210 from cudf.core._internals import ( @@ -69,6 +66,7 @@ find_common_type, get_time_unit, is_column_like, + is_dtype_obj_numeric, is_mixed_with_object_dtype, min_signed_type, min_unsigned_type, @@ -858,7 +856,7 @@ def _fill( if end <= begin or begin >= self.size: return self if inplace else self.copy() - if not inplace or is_string_dtype(self.dtype): + if not inplace or self.dtype == CUDF_STRING_DTYPE: with acquire_spill_lock(): result = type(self).from_pylibcudf( plc.filling.fill( @@ -868,7 +866,7 @@ def _fill( fill_value, ) ) - if is_string_dtype(self.dtype): + if self.dtype == CUDF_STRING_DTYPE: return self._mimic_inplace(result, inplace=True) return result # type: ignore[return-value] @@ -1599,7 +1597,10 @@ def cast(self, dtype: Dtype) -> ColumnBase: self.to_pylibcudf(mode="read"), dtype_to_pylibcudf_type(dtype) ) ) - if is_decimal_dtype(result.dtype): + if isinstance( + result.dtype, + (cudf.Decimal128Dtype, cudf.Decimal64Dtype, cudf.Decimal32Dtype), + ): result.dtype.precision = dtype.precision # type: ignore[union-attr] return result @@ -2993,7 +2994,8 @@ def concat_columns(objs: "MutableSequence[ColumnBase]") -> ColumnBase: # Notice, we can always cast pure null columns not_null_col_dtypes = [o.dtype for o in objs if o.null_count != len(o)] if len(not_null_col_dtypes) and all( - _is_non_decimal_numeric_dtype(dtype) and dtype.kind == "M" + is_dtype_obj_numeric(dtype, include_decimal=False) + and dtype.kind == "M" for dtype in not_null_col_dtypes ): common_dtype = find_common_type(not_null_col_dtypes) diff --git a/python/cudf/cudf/core/column/lists.py b/python/cudf/cudf/core/column/lists.py index 837763ee30c..ca29f83225b 100644 --- a/python/cudf/cudf/core/column/lists.py +++ b/python/cudf/cudf/core/column/lists.py @@ -14,7 +14,7 @@ import cudf import cudf.core.column.column as column -from cudf.api.types import _is_non_decimal_numeric_dtype, is_scalar +from cudf.api.types import is_scalar from cudf.core.buffer import acquire_spill_lock from cudf.core.column.column import ColumnBase, as_column from cudf.core.column.methods import ColumnMethods, ParentType @@ -22,7 +22,7 @@ from cudf.core.dtypes import ListDtype from cudf.core.missing import NA from cudf.core.scalar import pa_scalar_to_plc_scalar -from cudf.utils.dtypes import SIZE_TYPE_DTYPE +from cudf.utils.dtypes import SIZE_TYPE_DTYPE, is_dtype_obj_numeric if TYPE_CHECKING: from collections.abc import Sequence @@ -718,8 +718,8 @@ def take(self, lists_indices: ColumnLike) -> ParentType: "lists_indices and list column is of different size." ) if ( - not _is_non_decimal_numeric_dtype( - lists_indices_col.children[1].dtype + not is_dtype_obj_numeric( + lists_indices_col.children[1].dtype, include_decimal=False ) or lists_indices_col.children[1].dtype.kind not in "iu" ): diff --git a/python/cudf/cudf/core/column/numerical.py b/python/cudf/cudf/core/column/numerical.py index 77c5a6b6caf..249afe9aba6 100644 --- a/python/cudf/cudf/core/column/numerical.py +++ b/python/cudf/cudf/core/column/numerical.py @@ -14,7 +14,7 @@ import cudf import cudf.core.column.column as column -from cudf.api.types import is_integer, is_scalar +from cudf.api.types import infer_dtype, is_integer, is_scalar from cudf.core._internals import binaryop from cudf.core.buffer import acquire_spill_lock, as_buffer from cudf.core.column.column import ColumnBase, as_column @@ -439,7 +439,7 @@ def _process_values_for_isin( except (MixedTypeError, TypeError) as e: # There is a corner where `values` can be of `object` dtype # but have values of homogeneous type. - inferred_dtype = cudf.api.types.infer_dtype(values) + inferred_dtype = infer_dtype(values) if ( self.dtype.kind in {"i", "u"} and inferred_dtype == "integer" ) or ( diff --git a/python/cudf/cudf/core/column/string.py b/python/cudf/cudf/core/column/string.py index 97ec41f4c39..9f3512369a0 100644 --- a/python/cudf/cudf/core/column/string.py +++ b/python/cudf/cudf/core/column/string.py @@ -18,7 +18,7 @@ import cudf import cudf.core.column.column as column import cudf.core.column.datetime as datetime -from cudf.api.types import is_integer, is_scalar, is_string_dtype +from cudf.api.types import is_integer, is_scalar from cudf.core._internals import binaryop from cudf.core.buffer import Buffer, acquire_spill_lock from cudf.core.column.column import ColumnBase @@ -75,7 +75,7 @@ def __init__(self, parent): if isinstance(parent.dtype, cudf.ListDtype) else parent.dtype ) - if not is_string_dtype(value_type): + if value_type != CUDF_STRING_DTYPE: raise AttributeError( "Can only use .str accessor with string values" ) diff --git a/python/cudf/cudf/core/dataframe.py b/python/cudf/cudf/core/dataframe.py index f909d72687c..eec0bacd5c8 100644 --- a/python/cudf/cudf/core/dataframe.py +++ b/python/cudf/cudf/core/dataframe.py @@ -41,10 +41,7 @@ is_dict_like, is_dtype_equal, is_list_like, - is_numeric_dtype, - is_object_dtype, is_scalar, - is_string_dtype, ) from cudf.core import column, indexing_utils, reshape from cudf.core._compat import PANDAS_LT_300 @@ -90,6 +87,7 @@ cudf_dtype_from_pydata_dtype, find_common_type, is_column_like, + is_dtype_obj_numeric, min_signed_type, ) from cudf.utils.performance_tracking import _performance_tracking @@ -145,7 +143,7 @@ def __setitem__(self, key, value): return self._setitem_tuple_arg(key, value) @_performance_tracking - def _can_downcast_to_series(self, df, arg): + def _can_downcast_to_series(self, df: DataFrame, arg): """ This method encapsulates the logic used to determine whether or not the result of a loc/iloc @@ -170,8 +168,8 @@ def _can_downcast_to_series(self, df, arg): arg[1], slice ): return True - dtypes = df.dtypes.values.tolist() - all_numeric = all(is_numeric_dtype(t) for t in dtypes) + dtypes = [dtype for _, dtype in df._dtypes] + all_numeric = all(is_dtype_obj_numeric(t) for t in dtypes) if all_numeric or ( len(dtypes) and all(t == dtypes[0] for t in dtypes) ): @@ -348,7 +346,7 @@ def _getitem_tuple_arg(self, arg): df.index.name = columns_df.index.name if not isinstance( df.index, MultiIndex - ) and is_numeric_dtype(df.index.dtype): + ) and is_dtype_obj_numeric(df.index.dtype): # Preserve the original index type. df.index = df.index.astype(self._frame.index.dtype) df = df.sort_values(by=[tmp_col_name, cantor_name]) @@ -3158,7 +3156,7 @@ def where(self, cond, other=None, inplace=False, axis=None, level=None): # If other was provided, process that next. if isinstance(other, DataFrame): other_cols = [other._data[col] for col in self._column_names] - elif cudf.api.types.is_scalar(other): + elif is_scalar(other): other_cols = [other] * len(self._column_names) elif isinstance(other, cudf.Series): other_cols = other.to_pandas() @@ -3788,14 +3786,14 @@ def agg(self, aggs, axis=None): * Not supporting: ``axis``, ``*args``, ``**kwargs`` """ - dtypes = [self[col].dtype for col in self._column_names] + dtypes = [dtype for _, dtype in self._dtypes] common_dtype = find_common_type(dtypes) if common_dtype.kind != "b" and any( dtype.kind == "b" for dtype in dtypes ): raise MixedTypeError("Cannot create a column with mixed types") - if any(is_string_dtype(dt) for dt in dtypes): + if any(dt == CUDF_STRING_DTYPE for dt in dtypes): raise NotImplementedError( "DataFrame.agg() is not supported for " "frames containing string columns" @@ -4934,7 +4932,7 @@ def apply_rows( """ for col in incols: current_col_dtype = self._data[col].dtype - if is_string_dtype(current_col_dtype) or isinstance( + if current_col_dtype == CUDF_STRING_DTYPE or isinstance( current_col_dtype, cudf.CategoricalDtype ): raise TypeError( @@ -6294,8 +6292,8 @@ def make_false_column_like_self(): else: # These checks must happen after the conversions above # since numpy can't handle categorical dtypes. - self_is_str = is_string_dtype(self_col.dtype) - other_is_str = is_string_dtype(other_col.dtype) + self_is_str = self_col.dtype == CUDF_STRING_DTYPE + other_is_str = other_col.dtype == CUDF_STRING_DTYPE if self_is_str != other_is_str: # Strings can't compare to anything else. @@ -6352,8 +6350,8 @@ def _prepare_for_rowwise_op(self, method, skipna, numeric_only): common_dtype = find_common_type(filtered.dtypes) if ( not numeric_only - and is_string_dtype(common_dtype) - and any(not is_string_dtype(dt) for dt in filtered.dtypes) + and common_dtype == CUDF_STRING_DTYPE + and any(dtype != CUDF_STRING_DTYPE for dtype in filtered._dtypes) ): raise TypeError( f"Cannot perform row-wise {method} across mixed-dtype columns," @@ -6476,7 +6474,9 @@ def _reduce( if numeric_only: numeric_cols = ( - name for name, dtype in self._dtypes if is_numeric_dtype(dtype) + name + for name, dtype in self._dtypes + if is_dtype_obj_numeric(dtype) ) source = self._get_columns_by_label(numeric_cols) if source.empty: @@ -6507,7 +6507,7 @@ def _reduce( raise NotImplementedError( f"Column {col_label} with type {col.dtype} does not support {op}" ) from err - elif not is_numeric_dtype(col.dtype): + elif not is_dtype_obj_numeric(col.dtype): raise TypeError( "Non numeric columns passed with " "`numeric_only=False`, pass `numeric_only=True` " @@ -6523,9 +6523,9 @@ def _reduce( source_dtypes = [dtype for _, dtype in source._dtypes] common_dtype = find_common_type(source_dtypes) if ( - is_object_dtype(common_dtype) + common_dtype == CUDF_STRING_DTYPE and any( - not is_object_dtype(dtype) for dtype in source_dtypes + dtype != CUDF_STRING_DTYPE for dtype in source_dtypes ) or common_dtype.kind != "b" and any(dtype.kind == "b" for dtype in source_dtypes) @@ -8603,7 +8603,7 @@ def _find_common_dtypes_and_categories( # default to the first non-null dtype dtypes[idx] = cols[0].dtype # If all the non-null dtypes are int/float, find a common dtype - if all(is_numeric_dtype(col.dtype) for col in cols): + if all(is_dtype_obj_numeric(col.dtype) for col in cols): dtypes[idx] = find_common_type([col.dtype for col in cols]) # If all categorical dtypes, combine the categories elif all( diff --git a/python/cudf/cudf/core/groupby/groupby.py b/python/cudf/cudf/core/groupby/groupby.py index 38b519c6d5f..df11ebd4f94 100644 --- a/python/cudf/cudf/core/groupby/groupby.py +++ b/python/cudf/cudf/core/groupby/groupby.py @@ -20,11 +20,7 @@ import cudf from cudf.api.extensions import no_default -from cudf.api.types import ( - is_list_like, - is_numeric_dtype, - is_string_dtype, -) +from cudf.api.types import is_list_like, is_scalar from cudf.core._compat import PANDAS_LT_300 from cudf.core._internals import aggregation, sorting, stream_compaction from cudf.core.abc import Serializable @@ -44,7 +40,12 @@ from cudf.core.multiindex import MultiIndex from cudf.core.scalar import pa_scalar_to_plc_scalar from cudf.core.udf.groupby_utils import _can_be_jitted, jit_groupby_apply -from cudf.utils.dtypes import SIZE_TYPE_DTYPE, cudf_dtype_to_pa_type +from cudf.utils.dtypes import ( + CUDF_STRING_DTYPE, + SIZE_TYPE_DTYPE, + cudf_dtype_to_pa_type, + is_dtype_obj_numeric, +) from cudf.utils.performance_tracking import _performance_tracking from cudf.utils.utils import GetAttrGetItemMixin @@ -91,7 +92,7 @@ @singledispatch def get_valid_aggregation(dtype): - if is_string_dtype(dtype): + if dtype == CUDF_STRING_DTYPE: return _STRING_AGGS return "ALL" @@ -1788,7 +1789,7 @@ def _post_process_chunk_results( ): if not len(chunk_results): return self.obj.head(0) - if isinstance(chunk_results, ColumnBase) or cudf.api.types.is_scalar( + if isinstance(chunk_results, ColumnBase) or is_scalar( chunk_results[0] ): data = ColumnAccessor( @@ -3077,7 +3078,9 @@ def _reduce_numeric_only(self, op: str): columns = list( name for name, dtype in self.obj._dtypes - if (is_numeric_dtype(dtype) and name not in self.grouping.names) + if ( + is_dtype_obj_numeric(dtype) and name not in self.grouping.names + ) ) return self[columns].agg(op) diff --git a/python/cudf/cudf/core/index.py b/python/cudf/cudf/core/index.py index f4e5f6e96ae..05a2a46c051 100644 --- a/python/cudf/cudf/core/index.py +++ b/python/cudf/cudf/core/index.py @@ -20,12 +20,11 @@ import cudf from cudf.api.extensions import no_default from cudf.api.types import ( - _is_non_decimal_numeric_dtype, is_dtype_equal, + is_hashable, is_integer, is_list_like, is_scalar, - is_string_dtype, ) from cudf.core._base_index import BaseIndex, _return_get_indexer_result from cudf.core._compat import PANDAS_LT_300 @@ -57,6 +56,7 @@ cudf_dtype_from_pa_type, cudf_dtype_to_pa_type, find_common_type, + is_dtype_obj_numeric, is_mixed_with_object_dtype, ) from cudf.utils.performance_tracking import _performance_tracking @@ -232,7 +232,7 @@ class RangeIndex(BaseIndex, BinaryOperand): def __init__( self, start, stop=None, step=1, dtype=None, copy=False, name=None ): - if not cudf.api.types.is_hashable(name): + if not is_hashable(name): raise ValueError("Name must be a hashable value.") self._name = name if dtype is not None and cudf.dtype(dtype).kind != "i": @@ -1786,7 +1786,7 @@ def isin(self, values, level=None) -> cupy.ndarray: @property @_performance_tracking def str(self): - if is_string_dtype(self.dtype): + if self.dtype == CUDF_STRING_DTYPE: return StringMethods(parent=self) else: raise AttributeError( @@ -3366,7 +3366,7 @@ def interval_range( "freq, exactly three must be specified" ) - if periods is not None and not cudf.api.types.is_integer(periods): + if periods is not None and not is_integer(periods): warnings.warn( "Non-integer 'periods' in cudf.date_range, and cudf.interval_range" " are deprecated and will raise in a future version.", @@ -3390,7 +3390,9 @@ def interval_range( pa_freq = pa.scalar(freq) if any( - not _is_non_decimal_numeric_dtype(cudf_dtype_from_pa_type(x.type)) + not is_dtype_obj_numeric( + cudf_dtype_from_pa_type(x.type), include_decimal=False + ) for x in (pa_start, pa.scalar(periods), pa_freq, pa_end) ): raise ValueError("start, end, periods, freq must be numeric values.") diff --git a/python/cudf/cudf/core/indexed_frame.py b/python/cudf/cudf/core/indexed_frame.py index 2f4ad360d8b..2f33a860608 100644 --- a/python/cudf/cudf/core/indexed_frame.py +++ b/python/cudf/cudf/core/indexed_frame.py @@ -30,7 +30,6 @@ import cudf.core.common from cudf.api.extensions import no_default from cudf.api.types import ( - _is_non_decimal_numeric_dtype, is_dict_like, is_list_like, is_scalar, @@ -60,7 +59,11 @@ from cudf.utils import docutils, ioutils from cudf.utils._numba import _CUDFNumbaConfig from cudf.utils.docutils import copy_docstring -from cudf.utils.dtypes import SIZE_TYPE_DTYPE +from cudf.utils.dtypes import ( + SIZE_TYPE_DTYPE, + is_column_like, + is_dtype_obj_numeric, +) from cudf.utils.performance_tracking import _performance_tracking from cudf.utils.utils import _warn_no_dask_cudf @@ -71,6 +74,7 @@ ColumnLike, DataFrameOrSeries, Dtype, + DtypeObj, NotImplementedType, ) @@ -6402,9 +6406,9 @@ def rank( dropped_cols = False source = self if numeric_only: - if isinstance( - source, cudf.Series - ) and not _is_non_decimal_numeric_dtype(self.dtype): # type: ignore[attr-defined] + if isinstance(source, cudf.Series) and not is_dtype_obj_numeric( + source.dtype, include_decimal=False + ): # type: ignore[attr-defined] raise TypeError( "Series.rank does not allow numeric_only=True with " "non-numeric dtype." @@ -6412,7 +6416,7 @@ def rank( numeric_cols = ( name for name, dtype in self._dtypes - if _is_non_decimal_numeric_dtype(dtype) + if is_dtype_obj_numeric(dtype, include_decimal=False) ) source = self._get_columns_by_label(numeric_cols) if source.empty: @@ -6554,7 +6558,7 @@ def _check_duplicate_level_names(specified, level_names): @_performance_tracking def _get_replacement_values_for_columns( - to_replace: Any, value: Any, columns_dtype_map: dict[Any, Any] + to_replace: Any, value: Any, columns_dtype_map: dict[Any, DtypeObj] ) -> tuple[dict[Any, bool], dict[Any, Any], dict[Any, Any]]: """ Returns a per column mapping for the values to be replaced, new @@ -6587,24 +6591,22 @@ def _get_replacement_values_for_columns( if is_scalar(to_replace) and is_scalar(value): to_replace_columns = {col: [to_replace] for col in columns_dtype_map} values_columns = {col: [value] for col in columns_dtype_map} - elif cudf.api.types.is_list_like(to_replace) or isinstance( + elif is_list_like(to_replace) or isinstance( to_replace, (ColumnBase, BaseIndex) ): if is_scalar(value): to_replace_columns = {col: to_replace for col in columns_dtype_map} values_columns = { col: [value] - if _is_non_decimal_numeric_dtype(columns_dtype_map[col]) + if is_dtype_obj_numeric(dtype, include_decimal=False) else as_column( value, length=len(to_replace), dtype=cudf.dtype(type(value)), ) - for col in columns_dtype_map + for col, dtype in columns_dtype_map.items() } - elif cudf.api.types.is_list_like( - value - ) or cudf.utils.dtypes.is_column_like(value): + elif is_list_like(value) or is_column_like(value): if len(to_replace) != len(value): raise ValueError( f"Replacement lists must be " diff --git a/python/cudf/cudf/core/join/_join_helpers.py b/python/cudf/cudf/core/join/_join_helpers.py index c329bf11d97..331aa57fca4 100644 --- a/python/cudf/cudf/core/join/_join_helpers.py +++ b/python/cudf/cudf/core/join/_join_helpers.py @@ -9,9 +9,15 @@ import numpy as np import cudf -from cudf.api.types import is_decimal_dtype, is_dtype_equal, is_numeric_dtype +from cudf.api.types import is_dtype_equal from cudf.core.column import CategoricalColumn -from cudf.core.dtypes import CategoricalDtype +from cudf.core.dtypes import ( + CategoricalDtype, + Decimal32Dtype, + Decimal64Dtype, + Decimal128Dtype, +) +from cudf.utils.dtypes import is_dtype_obj_numeric if TYPE_CHECKING: from cudf.core.column import ColumnBase @@ -81,15 +87,17 @@ def _match_join_keys( if is_dtype_equal(ltype, rtype): return lcol, rcol - if is_decimal_dtype(ltype) or is_decimal_dtype(rtype): + if isinstance( + ltype, (Decimal32Dtype, Decimal64Dtype, Decimal128Dtype) + ) or isinstance(rtype, (Decimal32Dtype, Decimal64Dtype, Decimal128Dtype)): raise TypeError( "Decimal columns can only be merged with decimal columns " "of the same precision and scale" ) if ( - is_numeric_dtype(ltype) - and is_numeric_dtype(rtype) + is_dtype_obj_numeric(ltype) + and is_dtype_obj_numeric(rtype) and not (ltype.kind == "m" or rtype.kind == "m") ): common_type = ( diff --git a/python/cudf/cudf/core/multiindex.py b/python/cudf/cudf/core/multiindex.py index 87a8849a260..f681c043186 100644 --- a/python/cudf/cudf/core/multiindex.py +++ b/python/cudf/cudf/core/multiindex.py @@ -17,7 +17,7 @@ import cudf from cudf.api.extensions import no_default -from cudf.api.types import is_integer, is_list_like, is_object_dtype, is_scalar +from cudf.api.types import is_integer, is_list_like, is_scalar from cudf.core import column from cudf.core._base_index import _return_get_indexer_result from cudf.core._internals import sorting @@ -33,7 +33,11 @@ ensure_index, ) from cudf.core.join._join_helpers import _match_join_keys -from cudf.utils.dtypes import SIZE_TYPE_DTYPE, is_column_like +from cudf.utils.dtypes import ( + CUDF_STRING_DTYPE, + SIZE_TYPE_DTYPE, + is_column_like, +) from cudf.utils.performance_tracking import _performance_tracking from cudf.utils.utils import NotIterable, _external_only_api, _is_same_name @@ -42,7 +46,7 @@ from typing_extensions import Self - from cudf._typing import DataFrameOrSeries + from cudf._typing import DataFrameOrSeries, Dtype def _maybe_indices_to_slice(indices: cp.ndarray) -> slice | cp.ndarray: @@ -233,8 +237,8 @@ def to_series(self, index=None, name=None): ) @_performance_tracking - def astype(self, dtype, copy: bool = True) -> Self: - if not is_object_dtype(dtype): + def astype(self, dtype: Dtype, copy: bool = True) -> Self: + if cudf.dtype(dtype) != CUDF_STRING_DTYPE: raise TypeError( "Setting a MultiIndex dtype to anything other than object is " "not supported" @@ -1699,16 +1703,12 @@ def _is_sorted(self, ascending=None, null_position=None) -> bool: Returns True, if sorted as expected by ``ascending`` and ``null_position``, False otherwise. """ - if ascending is not None and not cudf.api.types.is_list_like( - ascending - ): + if ascending is not None and not is_list_like(ascending): raise TypeError( f"Expected a list-like or None for `ascending`, got " f"{type(ascending)}" ) - if null_position is not None and not cudf.api.types.is_list_like( - null_position - ): + if null_position is not None and not is_list_like(null_position): raise TypeError( f"Expected a list-like or None for `null_position`, got " f"{type(null_position)}" diff --git a/python/cudf/cudf/core/reshape.py b/python/cudf/cudf/core/reshape.py index 7d76907916f..b7412f2cc85 100644 --- a/python/cudf/cudf/core/reshape.py +++ b/python/cudf/cudf/core/reshape.py @@ -12,7 +12,7 @@ import cudf from cudf.api.extensions import no_default -from cudf.api.types import is_scalar +from cudf.api.types import is_list_like, is_scalar from cudf.core._compat import PANDAS_LT_300 from cudf.core.column import ( ColumnBase, @@ -1362,7 +1362,7 @@ def _one_hot_encode_column( def _length_check_params(obj, columns, name): - if cudf.api.types.is_list_like(obj): + if is_list_like(obj): if len(obj) != len(columns): raise ValueError( f"Length of '{name}' ({len(obj)}) did not match the " diff --git a/python/cudf/cudf/core/scalar.py b/python/cudf/cudf/core/scalar.py index 29139768a36..8579b7398f0 100644 --- a/python/cudf/cudf/core/scalar.py +++ b/python/cudf/cudf/core/scalar.py @@ -9,7 +9,6 @@ from typing import TYPE_CHECKING, Any import numpy as np -import pandas as pd import pyarrow as pa import pylibcudf as plc @@ -25,6 +24,7 @@ from cudf.core.missing import NA, NaT from cudf.core.mixins import BinaryOperand from cudf.utils.dtypes import ( + CUDF_STRING_DTYPE, cudf_dtype_from_pa_type, get_allowed_combinations_for_operator, to_cudf_compatible_scalar, @@ -191,7 +191,7 @@ def _to_plc_scalar(value: ScalarLike, dtype: Dtype) -> plc.Scalar: if isinstance(dtype, cudf.core.dtypes._BaseDtype): pa_type = dtype.to_arrow() - elif pd.api.types.is_string_dtype(dtype): + elif dtype == CUDF_STRING_DTYPE: # Have to manually convert object types, which we use internally # for strings but pyarrow only supports as unicode 'U' pa_type = pa.string() diff --git a/python/cudf/cudf/core/series.py b/python/cudf/cudf/core/series.py index d25550553b1..42247ce689e 100644 --- a/python/cudf/cudf/core/series.py +++ b/python/cudf/cudf/core/series.py @@ -20,7 +20,6 @@ import cudf from cudf.api.extensions import no_default from cudf.api.types import ( - _is_non_decimal_numeric_dtype, _is_scalar_or_zero_d_array, is_dict_like, is_integer, @@ -64,6 +63,7 @@ from cudf.utils.dtypes import ( can_convert_to_column, find_common_type, + is_dtype_obj_numeric, is_mixed_with_object_dtype, to_cudf_compatible_scalar, ) @@ -357,7 +357,9 @@ def _loc_to_iloc(self, arg): "as labels (consistent with DataFrame behavior). To access " "a value by position, use `ser.iloc[pos]`" ) - if not _is_non_decimal_numeric_dtype(index_dtype) and not ( + if not is_dtype_obj_numeric( + index_dtype, include_decimal=False + ) and not ( isinstance(index_dtype, cudf.CategoricalDtype) and index_dtype.categories.dtype.kind in "iu" ): diff --git a/python/cudf/cudf/core/single_column_frame.py b/python/cudf/cudf/core/single_column_frame.py index f9713ca62d1..aa59d3af640 100644 --- a/python/cudf/cudf/core/single_column_frame.py +++ b/python/cudf/cudf/core/single_column_frame.py @@ -12,12 +12,12 @@ from cudf.api.types import ( _is_scalar_or_zero_d_array, is_integer, - is_numeric_dtype, + is_scalar, ) from cudf.core.column import ColumnBase, as_column from cudf.core.column_accessor import ColumnAccessor from cudf.core.frame import Frame -from cudf.utils.dtypes import SIZE_TYPE_DTYPE +from cudf.utils.dtypes import SIZE_TYPE_DTYPE, is_dtype_obj_numeric from cudf.utils.performance_tracking import _performance_tracking from cudf.utils.utils import NotIterable @@ -54,7 +54,7 @@ def _reduce( if axis not in (None, 0, no_default): raise NotImplementedError("axis parameter is not implemented yet") - if numeric_only and not is_numeric_dtype(self.dtype): + if numeric_only and not is_dtype_obj_numeric(self.dtype): raise TypeError( f"Series.{op} does not allow numeric_only={numeric_only} " "with non-numeric dtypes." @@ -374,7 +374,7 @@ def where(self, cond, other=None, inplace=False): """Array conditional must be same shape as self""" ) - if not cudf.api.types.is_scalar(other): + if not is_scalar(other): other = cudf.core.column.as_column(other) input_col, other = _check_and_cast_columns_with_other( diff --git a/python/cudf/cudf/core/tools/datetimes.py b/python/cudf/cudf/core/tools/datetimes.py index 4478be2fd04..89abc120de9 100644 --- a/python/cudf/cudf/core/tools/datetimes.py +++ b/python/cudf/cudf/core/tools/datetimes.py @@ -882,7 +882,7 @@ def date_range( "three must be specified" ) - if periods is not None and not cudf.api.types.is_integer(periods): + if periods is not None and not is_integer(periods): warnings.warn( "Non-integer 'periods' in cudf.date_range, and cudf.interval_range" " are deprecated and will raise in a future version.", diff --git a/python/cudf/cudf/core/tools/numeric.py b/python/cudf/cudf/core/tools/numeric.py index 9746234cfb1..18e96ee4a68 100644 --- a/python/cudf/cudf/core/tools/numeric.py +++ b/python/cudf/cudf/core/tools/numeric.py @@ -8,11 +8,14 @@ import pandas as pd import cudf -from cudf.api.types import _is_non_decimal_numeric_dtype, is_string_dtype from cudf.core.column import as_column from cudf.core.dtypes import CategoricalDtype from cudf.core.index import ensure_index -from cudf.utils.dtypes import can_convert_to_column +from cudf.utils.dtypes import ( + CUDF_STRING_DTYPE, + can_convert_to_column, + is_dtype_obj_numeric, +) if TYPE_CHECKING: from cudf.core.column.numerical import NumericalColumn @@ -142,7 +145,7 @@ def to_numeric( return arg else: raise e - elif is_string_dtype(dtype): + elif dtype == CUDF_STRING_DTYPE: try: col = _convert_str_col(col, errors, downcast) # type: ignore[arg-type] except ValueError as e: @@ -152,7 +155,7 @@ def to_numeric( raise e elif isinstance(dtype, (cudf.ListDtype, cudf.StructDtype)): raise ValueError("Input does not support nested datatypes") - elif _is_non_decimal_numeric_dtype(dtype): + elif is_dtype_obj_numeric(dtype, include_decimal=False): pass else: raise ValueError("Unrecognized datatype") @@ -218,7 +221,7 @@ def _convert_str_col( ------- Converted numeric column """ - if not is_string_dtype(col): + if col.dtype != CUDF_STRING_DTYPE: raise TypeError("col must be string dtype.") if col.is_integer().all(): diff --git a/python/cudf/cudf/core/window/ewm.py b/python/cudf/cudf/core/window/ewm.py index 3e8a6ab400c..4b94e3e52b1 100644 --- a/python/cudf/cudf/core/window/ewm.py +++ b/python/cudf/cudf/core/window/ewm.py @@ -6,8 +6,8 @@ import numpy as np -from cudf.api.types import is_numeric_dtype from cudf.core.window.rolling import _RollingBase +from cudf.utils.dtypes import is_dtype_obj_numeric if TYPE_CHECKING: from cudf.core.column.column import ColumnBase @@ -184,7 +184,7 @@ def cov( def _apply_agg_column( self, source_column: ColumnBase, agg_name: str ) -> ColumnBase: - if not is_numeric_dtype(source_column.dtype): + if not is_dtype_obj_numeric(source_column.dtype): raise TypeError("No numeric types to aggregate") # libcudf ewm has special casing for nulls only diff --git a/python/cudf/cudf/io/dlpack.py b/python/cudf/cudf/io/dlpack.py index 3b3fd5f7c56..e7b224a40e7 100644 --- a/python/cudf/cudf/io/dlpack.py +++ b/python/cudf/cudf/io/dlpack.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019-2024, NVIDIA CORPORATION. +# Copyright (c) 2019-2025, NVIDIA CORPORATION. from __future__ import annotations import pylibcudf as plc @@ -6,6 +6,7 @@ import cudf from cudf.core.column import ColumnBase from cudf.utils import ioutils +from cudf.utils.dtypes import find_common_type, is_dtype_obj_numeric def from_dlpack(pycapsule_obj) -> cudf.Series | cudf.DataFrame: @@ -83,12 +84,12 @@ def to_dlpack(cudf_obj: cudf.Series | cudf.DataFrame | cudf.BaseIndex): ) if any( - not cudf.api.types._is_non_decimal_numeric_dtype(dtype) + not is_dtype_obj_numeric(dtype, include_decimal=False) for _, dtype in gdf._dtypes # type: ignore[union-attr] ): raise TypeError("non-numeric data not yet supported") - dtype = cudf.utils.dtypes.find_common_type( + dtype = find_common_type( [dtype for _, dtype in gdf._dtypes] # type: ignore[union-attr] ) gdf = gdf.astype(dtype) diff --git a/python/cudf/cudf/testing/testing.py b/python/cudf/cudf/testing/testing.py index 9c20a42d215..e1b0c17eb00 100644 --- a/python/cudf/cudf/testing/testing.py +++ b/python/cudf/cudf/testing/testing.py @@ -10,15 +10,15 @@ from pandas import testing as tm import cudf -from cudf.api.types import is_numeric_dtype, is_string_dtype from cudf.core.missing import NA, NaT +from cudf.utils.dtypes import CUDF_STRING_DTYPE, is_dtype_obj_numeric def dtype_can_compare_equal_to_other(dtype): # return True if values of this dtype can compare # as equal to equal values of a different dtype return not ( - is_string_dtype(dtype) + dtype == CUDF_STRING_DTYPE or isinstance( dtype, ( @@ -218,10 +218,10 @@ def assert_column_equal( elif not ( ( not dtype_can_compare_equal_to_other(left.dtype) - and is_numeric_dtype(right.dtype) + and is_dtype_obj_numeric(right.dtype) ) or ( - is_numeric_dtype(left.dtype) + is_dtype_obj_numeric(left.dtype) and not dtype_can_compare_equal_to_other(right.dtype) ) ): @@ -234,7 +234,7 @@ def assert_column_equal( if ( columns_equal and not check_exact - and is_numeric_dtype(left.dtype) + and is_dtype_obj_numeric(left.dtype) ): # non-null values must be the same columns_equal = cp.allclose( diff --git a/python/cudf/cudf/utils/dtypes.py b/python/cudf/cudf/utils/dtypes.py index 489b804583a..adee17e7bfb 100644 --- a/python/cudf/cudf/utils/dtypes.py +++ b/python/cudf/cudf/utils/dtypes.py @@ -612,6 +612,20 @@ def _get_base_dtype(dtype: pd.DatetimeTZDtype) -> np.dtype: return dtype.base +def is_dtype_obj_numeric( + dtype: DtypeObj, include_decimal: bool = True +) -> bool: + """Like is_numeric_dtype but does not introspect argument.""" + is_non_decimal = dtype.kind in set("iufb") + if include_decimal: + return is_non_decimal or isinstance( + dtype, + (cudf.Decimal32Dtype, cudf.Decimal64Dtype, cudf.Decimal128Dtype), + ) + else: + return is_non_decimal + + def dtype_to_pylibcudf_type(dtype) -> plc.DataType: if isinstance(dtype, cudf.ListDtype): return plc.DataType(plc.TypeId.LIST) From 54fc0c708f0d9252a695b57b3cc109aba961a431 Mon Sep 17 00:00:00 2001 From: David Wendt <45795991+davidwendt@users.noreply.github.com> Date: Tue, 4 Mar 2025 00:32:34 -0500 Subject: [PATCH 123/129] Minor typo fix in filling.pxd (#18120) Found this misspelled word while working on other things. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) - Yunsong Wang (https://github.com/PointKernel) - Muhammad Haseeb (https://github.com/mhaseeb123) URL: https://github.com/rapidsai/cudf/pull/18120 --- cpp/examples/interop/interop.cpp | 4 ++-- python/pylibcudf/pylibcudf/libcudf/filling.pxd | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/cpp/examples/interop/interop.cpp b/cpp/examples/interop/interop.cpp index 133a4e3a514..b01b04489a6 100644 --- a/cpp/examples/interop/interop.cpp +++ b/cpp/examples/interop/interop.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. + * Copyright (c) 2024-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -24,7 +24,7 @@ #include #include -// Helper functuons to create StringViews +// Helper functions to create StringViews inline arrow::StringViewType::c_type to_inline_string_view(const void* data, int32_t const& size) { arrow::StringViewType::c_type out; diff --git a/python/pylibcudf/pylibcudf/libcudf/filling.pxd b/python/pylibcudf/pylibcudf/libcudf/filling.pxd index f0bfe8ca80b..d9ae573d23b 100644 --- a/python/pylibcudf/pylibcudf/libcudf/filling.pxd +++ b/python/pylibcudf/pylibcudf/libcudf/filling.pxd @@ -1,4 +1,4 @@ -# Copyright (c) 2020-2024, NVIDIA CORPORATION. +# Copyright (c) 2020-2025, NVIDIA CORPORATION. from libcpp cimport bool from libcpp.memory cimport unique_ptr from pylibcudf.exception_handler cimport libcudf_exception_handler @@ -23,7 +23,7 @@ cdef extern from "cudf/filling.hpp" namespace "cudf" nogil: cdef void fill_in_place( const mutable_column_view & destination, - size_type beign, + size_type begin, size_type end, const scalar & value ) except +libcudf_exception_handler From 1420ef2c792cf56d3c91d7240560c3d0d2cb7629 Mon Sep 17 00:00:00 2001 From: Vukasin Milovanovic Date: Mon, 3 Mar 2025 22:20:34 -0800 Subject: [PATCH 124/129] Add `host_read_async` interfaces to `datasource` (#18018) kvikIO supports asynchronous host reads, but we don't utilize them to optimize host reads such as metadata access. This PR adds the async versions of the `host_read` APIs to allow efficient use of the kvikIO pool for host reads. The `datasource`s that are not backed by kvikIO implement these as deferred calls to the synchronous versions. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Tianyu Liu (https://github.com/kingcrimsontianyu) - Yunsong Wang (https://github.com/PointKernel) URL: https://github.com/rapidsai/cudf/pull/18018 --- cpp/include/cudf/io/datasource.hpp | 39 +++++- cpp/src/io/orc/reader_impl_chunking.cu | 36 +++--- cpp/src/io/utilities/datasource.cpp | 163 +++++++++++++------------ 3 files changed, 137 insertions(+), 101 deletions(-) diff --git a/cpp/include/cudf/io/datasource.hpp b/cpp/include/cudf/io/datasource.hpp index 7bec40893fd..92859ec0895 100644 --- a/cpp/include/cudf/io/datasource.hpp +++ b/cpp/include/cudf/io/datasource.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2024, NVIDIA CORPORATION. + * Copyright (c) 2020-2025, NVIDIA CORPORATION. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -67,7 +67,7 @@ class datasource { /** * @brief Base class destructor */ - virtual ~buffer() {} + virtual ~buffer() = default; /** * @brief Factory to construct a datasource buffer object from a container. @@ -156,7 +156,7 @@ class datasource { /** * @brief Base class destructor */ - virtual ~datasource(){}; + virtual ~datasource() = default; /** * @brief Returns a buffer with a subset of data from the source. @@ -168,6 +168,21 @@ class datasource { */ virtual std::unique_ptr host_read(size_t offset, size_t size) = 0; + /** + * @brief Asynchronously reads a specified portion of data from the datasource. + * + * This function initiates an asynchronous read operation that reads `size` bytes of data + * starting from the given `offset` in the datasource. Depending on the concrete datasource + * implementation, the read operation may be deferred until the returned future is waited upon. + * + * @param offset The starting position in the datasource from which to read. + * @param size The number of bytes to read from the datasource. + * @return A std::future that will hold a unique pointer to a datasource::buffer containing + * the read data once the operation completes. + */ + virtual std::future> host_read_async(size_t offset, + size_t size); + /** * @brief Reads a selected range into a preallocated buffer. * @@ -179,6 +194,22 @@ class datasource { */ virtual size_t host_read(size_t offset, size_t size, uint8_t* dst) = 0; + /** + * @brief Asynchronously reads data from the source into the provided host memory buffer. + * + * This function initiates an asynchronous read operation from the data source starting at the + * specified offset and reads the specified number of bytes into the destination buffer. Depending + * on the concrete datasource implementation, the read operation may be deferred and will be + * executed when the returned future is waited upon. + * + * @param offset The starting position in the data source from which to read. + * @param size The number of bytes to read from the data source. + * @param dst Pointer to the destination buffer where the read data will be stored. + * @return A std::future object that will hold the number of bytes read once the operation + * completes. + */ + virtual std::future host_read_async(size_t offset, size_t size, uint8_t* dst); + /** * @brief Whether or not this source supports reading directly into device memory. * @@ -296,7 +327,7 @@ class datasource { */ class non_owning_buffer : public buffer { public: - non_owning_buffer() {} + non_owning_buffer() = default; /** * @brief Construct a new non owning buffer object diff --git a/cpp/src/io/orc/reader_impl_chunking.cu b/cpp/src/io/orc/reader_impl_chunking.cu index 5c663950b00..5b0c7ae11a9 100644 --- a/cpp/src/io/orc/reader_impl_chunking.cu +++ b/cpp/src/io/orc/reader_impl_chunking.cu @@ -486,13 +486,11 @@ void reader_impl::load_next_stripe_data(read_mode mode) // Load stripe data into memory: // - // If we load data from sources into host buffers, we need to transfer (async) data to device - // memory. Such host buffers need to be kept alive until we sync the transfers. - std::vector> host_read_buffers; - - // If we load data directly from sources into device memory, the loads are also async. - // Thus, we need to make sure to sync all them at the end. + // Storing the future and the expected size of the read data std::vector, std::size_t>> device_read_tasks; + // Storing the future, the expected size of the read data and the device destination pointer + std::vector>, std::size_t, uint8_t*>> + host_read_tasks; // Range of the read info (offset, length) to read for the current being loaded stripes. auto const [read_begin, read_end] = @@ -518,24 +516,22 @@ void reader_impl::load_next_stripe_data(read_mode mode) source_ptr->device_read_async( read_info.offset, read_info.length, dst_base + read_info.dst_pos, _stream), read_info.length); - } else { - auto buffer = source_ptr->host_read(read_info.offset, read_info.length); - CUDF_EXPECTS(buffer->size() == read_info.length, "Unexpected discrepancy in bytes read."); - CUDF_CUDA_TRY(cudaMemcpyAsync(dst_base + read_info.dst_pos, - buffer->data(), - read_info.length, - cudaMemcpyDefault, - _stream.value())); - host_read_buffers.emplace_back(std::move(buffer)); + host_read_tasks.emplace_back(source_ptr->host_read_async(read_info.offset, read_info.length), + read_info.length, + dst_base + read_info.dst_pos); } } - - if (host_read_buffers.size() > 0) { // if there was host read - _stream.synchronize(); - host_read_buffers.clear(); // its data was copied to device memory after stream sync + std::vector> host_read_buffers; + for (auto& [fut, expected_size, dev_dst] : host_read_tasks) { // if there were host reads + host_read_buffers.emplace_back(fut.get()); + auto* host_buffer = host_read_buffers.back().get(); + CUDF_EXPECTS(host_buffer->size() == expected_size, "Unexpected discrepancy in bytes read."); + CUDF_CUDA_TRY(cudaMemcpyAsync( + dev_dst, host_buffer->data(), host_buffer->size(), cudaMemcpyDefault, _stream.value())); } - for (auto& task : device_read_tasks) { // if there was device read + + for (auto& task : device_read_tasks) { // if there were device reads CUDF_EXPECTS(task.first.get() == task.second, "Unexpected discrepancy in bytes read."); } diff --git a/cpp/src/io/utilities/datasource.cpp b/cpp/src/io/utilities/datasource.cpp index 2cb2b303cb3..2f181188fb2 100644 --- a/cpp/src/io/utilities/datasource.cpp +++ b/cpp/src/io/utilities/datasource.cpp @@ -44,37 +44,56 @@ namespace io { namespace { /** - * @brief Base class for file input. Only implements direct device reads. + * @brief Base class for kvikIO-based data sources. */ -class file_source : public datasource { - public: - explicit file_source(char const* filepath) - { - kvikio_integration::set_up_kvikio(); - _kvikio_file = kvikio::FileHandle(filepath, "r"); - CUDF_EXPECTS(!_kvikio_file.closed(), "KvikIO did not open the file successfully."); - CUDF_LOG_INFO("Reading a file using kvikIO, with compatibility mode %s.", - _kvikio_file.get_compat_mode_manager().is_compat_mode_preferred() ? "on" : "off"); - } +template +class kvikio_source : public datasource { + class kvikio_initializer { + public: + kvikio_initializer() { kvikio_integration::set_up_kvikio(); } + }; - std::unique_ptr host_read(size_t offset, size_t size) override + std::pair, std::future> clamped_read_to_vector(size_t offset, + size_t size) { // Clamp length to available data auto const read_size = std::min(size, this->size() - offset); std::vector v(read_size); - CUDF_EXPECTS(_kvikio_file.pread(v.data(), read_size, offset).get() == read_size, "read failed"); + return {std::move(v), _kvikio_handle.pread(v.data(), read_size, offset)}; + } + + public: + kvikio_source(HandleT&& h) : _kvikio_handle(std::move(h)) {} + std::unique_ptr host_read(size_t offset, size_t size) override + { + auto [v, fut] = clamped_read_to_vector(offset, size); + fut.get(); return buffer::create(std::move(v)); } + std::future> host_read_async(size_t offset, + size_t size) override + { + auto clamped_read = clamped_read_to_vector(offset, size); + return std::async(std::launch::deferred, [cr = std::move(clamped_read)]() mutable { + cr.second.get(); + return buffer::create(std::move(cr.first)); + }); + } + size_t host_read(size_t offset, size_t size, uint8_t* dst) override + { + return host_read_async(offset, size, dst).get(); + } + + std::future host_read_async(size_t offset, size_t size, uint8_t* dst) override { // Clamp length to available data auto const read_size = std::min(size, this->size() - offset); - CUDF_EXPECTS(_kvikio_file.pread(dst, read_size, offset).get() == read_size, "read failed"); - return read_size; + return _kvikio_handle.pread(dst, read_size, offset); } - ~file_source() override = default; + ~kvikio_source() override = default; [[nodiscard]] bool supports_device_read() const override { return true; } @@ -91,7 +110,7 @@ class file_source : public datasource { CUDF_EXPECTS(supports_device_read(), "Device reads are not supported for this file."); auto const read_size = std::min(size, this->size() - offset); - return _kvikio_file.pread(dst, read_size, offset); + return _kvikio_handle.pread(dst, read_size, offset); } size_t device_read(size_t offset, @@ -113,10 +132,29 @@ class file_source : public datasource { return datasource::buffer::create(std::move(out_data)); } - [[nodiscard]] size_t size() const override { return _kvikio_file.nbytes(); } + [[nodiscard]] size_t size() const override { return _kvikio_handle.nbytes(); } + + kvikio_initializer _; protected: - kvikio::FileHandle _kvikio_file; + HandleT _kvikio_handle; +}; + +/** + * @brief A class representing a file source using kvikIO. + * + * This class is derived from `kvikio_source` and is used to handle file operations + * using kvikIO library. + */ +class file_source : public kvikio_source { + public: + explicit file_source(char const* filepath) : kvikio_source{kvikio::FileHandle(filepath, "r")} + { + CUDF_EXPECTS(!_kvikio_handle.closed(), "KvikIO did not open the file successfully."); + CUDF_LOG_INFO( + "Reading a file using kvikIO, with compatibility mode %s.", + _kvikio_handle.get_compat_mode_manager().is_compat_mode_preferred() ? "on" : "off"); + } }; /** @@ -132,7 +170,7 @@ class memory_mapped_source : public file_source { { if (this->size() != 0) { // Memory mapping is not exclusive, so we can include the whole region we expect to read - map(_kvikio_file.fd(), offset, max_size_estimate); + map(_kvikio_handle.fd(), offset, max_size_estimate); } } @@ -331,6 +369,17 @@ class user_datasource_wrapper : public datasource { return source->host_read(offset, size); } + std::future host_read_async(size_t offset, size_t size, uint8_t* dst) override + { + return source->host_read_async(offset, size, dst); + } + + std::future> host_read_async(size_t offset, + size_t size) override + { + return source->host_read_async(offset, size); + } + [[nodiscard]] bool supports_device_read() const override { return source->supports_device_read(); @@ -376,68 +425,18 @@ class user_datasource_wrapper : public datasource { /** * @brief Remote file source backed by KvikIO, which handles S3 filepaths seamlessly. */ -class remote_file_source : public datasource { - static std::unique_ptr create_s3_endpoint(char const* filepath) +class remote_file_source : public kvikio_source { + static auto create_s3_handle(char const* filepath) { auto [bucket_name, bucket_object] = kvikio::S3Endpoint::parse_s3_url(filepath); - return std::make_unique(bucket_name, bucket_object); + return kvikio::RemoteHandle{std::make_unique(bucket_name, bucket_object)}; } public: - explicit remote_file_source(char const* filepath) : _kvikio_file{create_s3_endpoint(filepath)} {} + explicit remote_file_source(char const* filepath) : kvikio_source{create_s3_handle(filepath)} {} ~remote_file_source() override = default; - [[nodiscard]] bool supports_device_read() const override { return true; } - - [[nodiscard]] bool is_device_read_preferred(size_t size) const override { return true; } - - [[nodiscard]] size_t size() const override { return _kvikio_file.nbytes(); } - - std::future device_read_async(size_t offset, - size_t size, - uint8_t* dst, - rmm::cuda_stream_view stream) override - { - CUDF_EXPECTS(supports_device_read(), "Device reads are not supported for this file."); - - auto const read_size = std::min(size, this->size() - offset); - return _kvikio_file.pread(dst, read_size, offset); - } - - size_t device_read(size_t offset, - size_t size, - uint8_t* dst, - rmm::cuda_stream_view stream) override - { - return device_read_async(offset, size, dst, stream).get(); - } - - std::unique_ptr device_read(size_t offset, - size_t size, - rmm::cuda_stream_view stream) override - { - rmm::device_buffer out_data(size, stream); - size_t const read = - device_read(offset, size, reinterpret_cast(out_data.data()), stream); - out_data.resize(read, stream); - return datasource::buffer::create(std::move(out_data)); - } - - size_t host_read(size_t offset, size_t size, uint8_t* dst) override - { - auto const read_size = std::min(size, this->size() - offset); - return _kvikio_file.pread(dst, read_size, offset).get(); - } - - std::unique_ptr host_read(size_t offset, size_t size) override - { - auto const count = std::min(size, this->size() - offset); - std::vector h_data(count); - this->host_read(offset, count, h_data.data()); - return datasource::buffer::create(std::move(h_data)); - } - /** * @brief Is `url` referring to a remote file supported by KvikIO? * @@ -449,9 +448,6 @@ class remote_file_source : public datasource { static std::regex const pattern{R"(^s3://)", std::regex_constants::icase}; return std::regex_search(url, pattern); } - - private: - kvikio::RemoteHandle _kvikio_file; }; #else /** @@ -509,5 +505,18 @@ std::unique_ptr datasource::create(datasource* source) return std::make_unique(source); } +std::future> datasource::host_read_async(size_t offset, + size_t size) +{ + return std::async(std::launch::deferred, + [this, offset, size] { return host_read(offset, size); }); +} + +std::future datasource::host_read_async(size_t offset, size_t size, uint8_t* dst) +{ + return std::async(std::launch::deferred, + [this, offset, size, dst] { return host_read(offset, size, dst); }); +} + } // namespace io } // namespace cudf From d9e64b2361083f30785d61e5ad03bbd9bc353220 Mon Sep 17 00:00:00 2001 From: Peter Andreas Entschev Date: Tue, 4 Mar 2025 11:12:11 +0100 Subject: [PATCH 125/129] Add `pylibcudf.gpumemoryview` support for `len()`/`nbytes` (#18133) Add support for `len()` and `nbytes` in `pylibcudf.gpumemoryview`. Having those methods is helpful to ensure proper serialization in Dask/Distributed, as utility methods that serialize objects, in this case used by cudf-polars, may use the appropriate method or property to determine the size of the object being transferred. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Matthew Murray (https://github.com/Matt711) - Richard (Rick) Zamora (https://github.com/rjzamora) URL: https://github.com/rapidsai/cudf/pull/18133 --- python/pylibcudf/pylibcudf/gpumemoryview.pyi | 3 + python/pylibcudf/pylibcudf/gpumemoryview.pyx | 20 ++++++- .../pylibcudf/tests/test_gpumemoryview.py | 58 +++++++++++++++++++ 3 files changed, 80 insertions(+), 1 deletion(-) create mode 100644 python/pylibcudf/pylibcudf/tests/test_gpumemoryview.py diff --git a/python/pylibcudf/pylibcudf/gpumemoryview.pyi b/python/pylibcudf/pylibcudf/gpumemoryview.pyi index 50f1f39a515..236ff6e56a6 100644 --- a/python/pylibcudf/pylibcudf/gpumemoryview.pyi +++ b/python/pylibcudf/pylibcudf/gpumemoryview.pyi @@ -7,3 +7,6 @@ class gpumemoryview: def __init__(self, data: Any): ... @property def __cuda_array_interface__(self) -> Mapping[str, Any]: ... + def __len__(self) -> int: ... + @property + def nbytes(self) -> int: ... diff --git a/python/pylibcudf/pylibcudf/gpumemoryview.pyx b/python/pylibcudf/pylibcudf/gpumemoryview.pyx index 41316eddb60..954d35a6ce3 100644 --- a/python/pylibcudf/pylibcudf/gpumemoryview.pyx +++ b/python/pylibcudf/pylibcudf/gpumemoryview.pyx @@ -1,4 +1,7 @@ -# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# Copyright (c) 2023-2025, NVIDIA CORPORATION. + +import functools +import operator __all__ = ["gpumemoryview"] @@ -27,4 +30,19 @@ cdef class gpumemoryview: def __cuda_array_interface__(self): return self.obj.__cuda_array_interface__ + def __len__(self): + return self.obj.__cuda_array_interface__["shape"][0] + + @property + def nbytes(self): + cai = self.obj.__cuda_array_interface__ + shape, typestr = cai["shape"], cai["typestr"] + + # Get element size from typestr, format is two character specifying + # the type and the latter part is the number of bytes. E.g., ' Date: Tue, 4 Mar 2025 12:16:08 +0000 Subject: [PATCH 126/129] Added polynomials benchmark (#17695) This merge request implements benchmarks for comparing the AST, UDF Transform, and BINARY_OP methods by computing a polynomial. Closes https://github.com/rapidsai/cudf/issues/17561 Authors: - Basit Ayantunde (https://github.com/lamarrr) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Muhammad Haseeb (https://github.com/mhaseeb123) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/17695 --- cpp/benchmarks/CMakeLists.txt | 11 ++- cpp/benchmarks/ast/polynomials.cpp | 94 +++++++++++++++++++ cpp/benchmarks/binaryop/polynomials.cpp | 101 +++++++++++++++++++++ cpp/benchmarks/transform/polynomials.cpp | 109 +++++++++++++++++++++++ 4 files changed, 313 insertions(+), 2 deletions(-) create mode 100644 cpp/benchmarks/ast/polynomials.cpp create mode 100644 cpp/benchmarks/binaryop/polynomials.cpp create mode 100644 cpp/benchmarks/transform/polynomials.cpp diff --git a/cpp/benchmarks/CMakeLists.txt b/cpp/benchmarks/CMakeLists.txt index 03f11cc957b..549cb8e5d5d 100644 --- a/cpp/benchmarks/CMakeLists.txt +++ b/cpp/benchmarks/CMakeLists.txt @@ -344,11 +344,18 @@ ConfigureNVBench(CSV_WRITER_NVBENCH io/csv/csv_writer.cpp) # ################################################################################################## # * ast benchmark --------------------------------------------------------------------------------- -ConfigureNVBench(AST_NVBENCH ast/transform.cpp) +ConfigureNVBench(AST_NVBENCH ast/polynomials.cpp ast/transform.cpp) # ################################################################################################## # * binaryop benchmark ---------------------------------------------------------------------------- -ConfigureNVBench(BINARYOP_NVBENCH binaryop/binaryop.cpp binaryop/compiled_binaryop.cpp) +ConfigureNVBench( + BINARYOP_NVBENCH binaryop/binaryop.cpp binaryop/compiled_binaryop.cpp binaryop/polynomials.cpp +) + +# ################################################################################################## +# * transform benchmark +# --------------------------------------------------------------------------------- +ConfigureNVBench(TRANSFORM_NVBENCH transform/polynomials.cpp) # ################################################################################################## # * nvtext benchmark ------------------------------------------------------------------- diff --git a/cpp/benchmarks/ast/polynomials.cpp b/cpp/benchmarks/ast/polynomials.cpp new file mode 100644 index 00000000000..b8e4ca46b72 --- /dev/null +++ b/cpp/benchmarks/ast/polynomials.cpp @@ -0,0 +1,94 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include +#include +#include +#include +#include +#include + +#include + +#include +#include + +#include + +template +static void BM_ast_polynomials(nvbench::state& state) +{ + auto const num_rows = static_cast(state.get_int64("num_rows")); + auto const order = static_cast(state.get_int64("order")); + + CUDF_EXPECTS(order > 0, "Polynomial order must be greater than 0"); + + data_profile profile; + profile.set_distribution_params(cudf::type_to_id(), + distribution_id::NORMAL, + static_cast(0), + static_cast(1)); + auto table = create_random_table({cudf::type_to_id()}, row_count{num_rows}, profile); + auto column_view = table->get_column(0); + + std::vector> constants; + { + std::random_device random_device; + std::mt19937 generator; + std::uniform_real_distribution distribution{0, 1}; + + std::transform(thrust::make_counting_iterator(0), + thrust::make_counting_iterator(order + 1), + std::back_inserter(constants), + [&](int) { return distribution(generator); }); + } + + cudf::ast::tree tree{}; + + auto& column_ref = tree.push(cudf::ast::column_reference{0}); + + // computes polynomials: (((ax + b)x + c)x + d)x + e... = ax**4 + bx**3 + cx**2 + dx + e.... + tree.push(cudf::ast::literal{constants[0]}); + + for (cudf::size_type i = 0; i < order; i++) { + auto& product = + tree.push(cudf::ast::operation{cudf::ast::ast_operator::MUL, tree.back(), column_ref}); + auto& constant = tree.push(cudf::ast::literal{constants[i + 1]}); + tree.push(cudf::ast::operation{cudf::ast::ast_operator::ADD, product, constant}); + } + + // Use the number of bytes read from global memory + state.add_global_memory_reads(num_rows); + state.add_global_memory_writes(num_rows); + + state.exec(nvbench::exec_tag::sync, [&](nvbench::launch& launch) { + cudf::scoped_range range{"benchmark_iteration"}; + cudf::compute_column(*table, tree.back(), launch.get_stream().get_stream()); + }); +} + +#define AST_POLYNOMIAL_BENCHMARK_DEFINE(name, key_type) \ + static void name(::nvbench::state& st) { ::BM_ast_polynomials(st); } \ + NVBENCH_BENCH(name) \ + .set_name(#name) \ + .add_int64_axis("num_rows", {100'000, 1'000'000, 10'000'000, 100'000'000}) \ + .add_int64_axis("order", {1, 2, 4, 8, 16, 32}) + +AST_POLYNOMIAL_BENCHMARK_DEFINE(ast_polynomials_float32, float); + +AST_POLYNOMIAL_BENCHMARK_DEFINE(ast_polynomials_float64, double); diff --git a/cpp/benchmarks/binaryop/polynomials.cpp b/cpp/benchmarks/binaryop/polynomials.cpp new file mode 100644 index 00000000000..782ae1db927 --- /dev/null +++ b/cpp/benchmarks/binaryop/polynomials.cpp @@ -0,0 +1,101 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include +#include +#include +#include +#include +#include + +#include + +#include + +#include +#include + +template +static void BM_binaryop_polynomials(nvbench::state& state) +{ + auto const num_rows{static_cast(state.get_int64("num_rows"))}; + auto const order{static_cast(state.get_int64("order"))}; + + CUDF_EXPECTS(order > 0, "Polynomial order must be greater than 0"); + + data_profile profile; + profile.set_distribution_params(cudf::type_to_id(), + distribution_id::NORMAL, + static_cast(0), + static_cast(1)); + auto table = create_random_table({cudf::type_to_id()}, row_count{num_rows}, profile); + auto column_view = table->get_column(0); + + std::vector> constants; + { + std::random_device random_device; + std::mt19937 generator; + std::uniform_real_distribution distribution{0, 1}; + + std::transform(thrust::make_counting_iterator(0), + thrust::make_counting_iterator(order + 1), + std::back_inserter(constants), + [&](int) { return cudf::numeric_scalar(distribution(generator)); }); + } + + // Use the number of bytes read from global memory + state.add_global_memory_reads(num_rows); + state.add_global_memory_writes(num_rows); + + state.exec(nvbench::exec_tag::sync, [&](nvbench::launch& launch) { + // computes polynomials: (((ax + b)x + c)x + d)x + e... = ax**4 + bx**3 + cx**2 + dx + e.... + cudf::scoped_range range{"benchmark_iteration"}; + rmm::cuda_stream_view stream{launch.get_stream().get_stream()}; + std::vector> intermediates; + + auto result = cudf::make_column_from_scalar(constants[0], num_rows, stream); + + for (cudf::size_type i = 0; i < order; i++) { + auto product = cudf::binary_operation(result->view(), + column_view, + cudf::binary_operator::MUL, + cudf::data_type{cudf::type_to_id()}, + stream); + auto sum = cudf::binary_operation(product->view(), + constants[i + 1], + cudf::binary_operator::ADD, + cudf::data_type{cudf::type_to_id()}, + stream); + intermediates.push_back(std::move(product)); + intermediates.push_back(std::move(result)); + result = std::move(sum); + } + }); +} + +#define BINARYOP_POLYNOMIALS_BENCHMARK_DEFINE(name, key_type) \ + \ + static void name(::nvbench::state& st) { ::BM_binaryop_polynomials(st); } \ + NVBENCH_BENCH(name) \ + .set_name(#name) \ + .add_int64_axis("num_rows", {100'000, 1'000'000, 10'000'000, 100'000'000}) \ + .add_int64_axis("order", {1, 2, 4, 8, 16, 32}) + +BINARYOP_POLYNOMIALS_BENCHMARK_DEFINE(binaryop_polynomials_float32, float); + +BINARYOP_POLYNOMIALS_BENCHMARK_DEFINE(binaryop_polynomials_float64, double); diff --git a/cpp/benchmarks/transform/polynomials.cpp b/cpp/benchmarks/transform/polynomials.cpp new file mode 100644 index 00000000000..07f8a47c771 --- /dev/null +++ b/cpp/benchmarks/transform/polynomials.cpp @@ -0,0 +1,109 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include +#include +#include +#include +#include + +#include + +#include + +#include +#include + +template +static void BM_transform_polynomials(nvbench::state& state) +{ + auto const num_rows{static_cast(state.get_int64("num_rows"))}; + auto const order{static_cast(state.get_int64("order"))}; + + CUDF_EXPECTS(order > 0, "Polynomial order must be greater than 0"); + + data_profile profile; + profile.set_distribution_params(cudf::type_to_id(), + distribution_id::NORMAL, + static_cast(0), + static_cast(1)); + auto column = create_random_column(cudf::type_to_id(), row_count{num_rows}, profile); + + std::vector> constants; + + std::transform( + thrust::make_counting_iterator(0), + thrust::make_counting_iterator(order + 1), + std::back_inserter(constants), + [&](int) { return create_random_column(cudf::type_to_id(), row_count{1}, profile); }); + + // Use the number of bytes read from global memory + state.add_global_memory_reads(num_rows); + state.add_global_memory_writes(num_rows); + + std::vector inputs{*column}; + std::transform(constants.begin(), + constants.end(), + std::back_inserter(inputs), + [](auto& col) -> cudf::column_view { return *col; }); + + state.exec(nvbench::exec_tag::sync, [&](nvbench::launch& launch) { + // computes polynomials: (((ax + b)x + c)x + d)x + e... = ax**4 + bx**3 + cx**2 + dx + e.... + + cudf::scoped_range range{"benchmark_iteration"}; + + std::string type = cudf::type_to_name(cudf::data_type{cudf::type_to_id()}); + + std::string params_decl = type + " c0"; + std::string expr = "c0"; + + for (cudf::size_type i = 1; i < order + 1; i++) { + expr = "( " + expr + " ) * x + c" + std::to_string(i); + params_decl += ", " + type + " c" + std::to_string(i); + } + + static_assert(std::is_same_v || std::is_same_v); + + // clang-format off + std::string udf = + "__device__ inline void compute_polynomial(" + type + "* out, " + type + " x, " + params_decl + ")" + +"{ " +" *out = " + expr + ";" +"}"; + + // clang-format on + + cudf::transform(inputs, + udf, + cudf::data_type{cudf::type_to_id()}, + false, + launch.get_stream().get_stream()); + }); +} + +#define TRANSFORM_POLYNOMIALS_BENCHMARK_DEFINE(name, key_type) \ + \ + static void name(::nvbench::state& st) { ::BM_transform_polynomials(st); } \ + NVBENCH_BENCH(name) \ + .set_name(#name) \ + .add_int64_axis("num_rows", {100'000, 1'000'000, 10'000'000, 100'000'000}) \ + .add_int64_axis("order", {1, 2, 4, 8, 16, 32}) + +TRANSFORM_POLYNOMIALS_BENCHMARK_DEFINE(transform_polynomials_float32, float); + +TRANSFORM_POLYNOMIALS_BENCHMARK_DEFINE(transform_polynomials_float64, double); From 5b0a85b5397b69155fe0c740185945a9fe0848ac Mon Sep 17 00:00:00 2001 From: Basit Ayantunde Date: Tue, 4 Mar 2025 13:05:21 +0000 Subject: [PATCH 127/129] Added Imbalanced Tree Benchmarks for Transforms (#18032) This merge request follows up on https://github.com/rapidsai/cudf/issues/18023 and adds a benchmark for comparing imbalanced trees for transforms against AST and binaryops. Authors: - Basit Ayantunde (https://github.com/lamarrr) Approvers: - Paul Mattione (https://github.com/pmattione-nvidia) - Nghia Truong (https://github.com/ttnghia) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/18032 --- cpp/benchmarks/CMakeLists.txt | 4 + cpp/benchmarks/transform/transform.cpp | 122 +++++++++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 cpp/benchmarks/transform/transform.cpp diff --git a/cpp/benchmarks/CMakeLists.txt b/cpp/benchmarks/CMakeLists.txt index 549cb8e5d5d..e82c7517145 100644 --- a/cpp/benchmarks/CMakeLists.txt +++ b/cpp/benchmarks/CMakeLists.txt @@ -357,6 +357,10 @@ ConfigureNVBench( # --------------------------------------------------------------------------------- ConfigureNVBench(TRANSFORM_NVBENCH transform/polynomials.cpp) +# ################################################################################################## +# * transform benchmark ---------------------------------------------------------------------------- +ConfigureNVBench(TRANSFORM_NVBENCH transform/transform.cpp) + # ################################################################################################## # * nvtext benchmark ------------------------------------------------------------------- ConfigureNVBench( diff --git a/cpp/benchmarks/transform/transform.cpp b/cpp/benchmarks/transform/transform.cpp new file mode 100644 index 00000000000..1b371fa3c1d --- /dev/null +++ b/cpp/benchmarks/transform/transform.cpp @@ -0,0 +1,122 @@ +/* + * Copyright (c) 2025, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include + +#include +#include + +#include +#include +#include +#include +#include + +enum class TreeType { + IMBALANCED_LEFT // All operator expressions have a left child operator expression and a right + // child column reference +}; + +template +static void BM_transform(nvbench::state& state) +{ + auto const num_rows = static_cast(state.get_int64("num_rows")); + auto const tree_levels = static_cast(state.get_int64("tree_levels")); + + // Create table data + auto const num_columns = reuse_columns ? 1 : tree_levels + 1; + auto const source_table = + create_sequence_table(cycle_dtypes({cudf::type_to_id()}, num_columns), + row_count{num_rows}, + Nullable ? std::optional{0.5} : std::nullopt); + auto table = source_table->view(); + + // Construct expression that chains additions like (((a + b) + c) + d) + std::string const op = "+"; + std::string expression; + if constexpr (reuse_columns) { + expression = "c0 " + op + " c0"; + std::for_each(thrust::make_counting_iterator(1), + thrust::make_counting_iterator(num_columns), + [&](int) { expression = "( " + expression + " ) " + op + " c0 "; }); + } else { + expression = "c0 " + op + " c1"; + std::for_each( + thrust::make_counting_iterator(2), thrust::make_counting_iterator(num_columns), [&](int col) { + expression = "( " + expression + " ) " + op + " c" + std::to_string(col); + }); + } + + std::string type_name = cudf::type_to_name(cudf::data_type{cudf::type_to_id()}); + std::string params = type_name + " c0"; + + std::for_each(thrust::make_counting_iterator(1), + thrust::make_counting_iterator(num_columns), + [&](int param) { params += ", " + type_name + " c" + std::to_string(param); }); + + std::string code = + "void transform(" + type_name + "* out, " + params + " ) { *out = " + expression + "; }"; + + std::vector inputs; + + std::transform(thrust::make_counting_iterator(0), + thrust::make_counting_iterator(source_table->num_columns()), + std::back_inserter(inputs), + [&source_table](int col) { return source_table->get_column(col).view(); }); + + // Use the number of bytes read from global memory + state.add_global_memory_reads(static_cast(num_rows) * (tree_levels + 1)); + state.add_global_memory_writes(num_rows); + + state.exec(nvbench::exec_tag::sync, [&](nvbench::launch& launch) { + cudf::transform(inputs, + code, + cudf::data_type{cudf::type_to_id()}, + false, + launch.get_stream().get_stream()); + }); +} + +#define AST_TRANSFORM_BENCHMARK_DEFINE(name, key_type, tree_type, reuse_columns, nullable) \ + static void name(::nvbench::state& st) \ + { \ + ::BM_transform(st); \ + } \ + NVBENCH_BENCH(name) \ + .set_name(#name) \ + .add_int64_axis("tree_levels", {1, 5, 10}) \ + .add_int64_axis("num_rows", {100'000, 1'000'000, 10'000'000, 100'000'000}) + +AST_TRANSFORM_BENCHMARK_DEFINE( + transform_int32_imbalanced_unique, int32_t, TreeType::IMBALANCED_LEFT, false, false); +AST_TRANSFORM_BENCHMARK_DEFINE( + transform_int32_imbalanced_reuse, int32_t, TreeType::IMBALANCED_LEFT, true, false); +AST_TRANSFORM_BENCHMARK_DEFINE( + transform_double_imbalanced_unique, double, TreeType::IMBALANCED_LEFT, false, false); From 8ff194de7563b0a1378fd542a64c07875ca00c1d Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Tue, 4 Mar 2025 05:55:04 -0800 Subject: [PATCH 128/129] Fix Series construction from numpy array with non-native byte order (#18151) closes https://github.com/rapidsai/cudf/issues/18149 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/18151 --- python/cudf/cudf/core/column/column.py | 3 +++ python/cudf/cudf/tests/test_series.py | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/python/cudf/cudf/core/column/column.py b/python/cudf/cudf/core/column/column.py index 5a8064dc49d..5c72bb74d6a 100644 --- a/python/cudf/cudf/core/column/column.py +++ b/python/cudf/cudf/core/column/column.py @@ -2727,6 +2727,9 @@ def as_column( return as_column(arbitrary, dtype=dtype, nan_as_null=nan_as_null) elif arbitrary.dtype.kind in "biuf": from_pandas = nan_as_null is None or nan_as_null + if not arbitrary.dtype.isnative: + # Not supported by pyarrow + arbitrary = arbitrary.astype(arbitrary.dtype.newbyteorder("=")) return as_column( pa.array(arbitrary, from_pandas=from_pandas), dtype=dtype, diff --git a/python/cudf/cudf/tests/test_series.py b/python/cudf/cudf/tests/test_series.py index 45910c17f95..2e53b2b146f 100644 --- a/python/cudf/cudf/tests/test_series.py +++ b/python/cudf/cudf/tests/test_series.py @@ -3040,3 +3040,13 @@ def test_series_dataframe_count_float(): gs.to_frame().count(), gs.to_frame().to_pandas(nullable=True).count(), ) + + +def test_construct_nonnative_np_array(): + data = [1, 2, 3.5, 4] + dtype = np.dtype("f4") + np_array = np.array(data, dtype=dtype) + np_nonnative = np.array(data, dtype=dtype.newbyteorder()) + result = cudf.Series(np_nonnative) + expected = cudf.Series(np_array) + assert_eq(result, expected) From 29b81eb42aa60c5e41785b3b2eb9e80b964c90ad Mon Sep 17 00:00:00 2001 From: Gil Forsyth Date: Tue, 4 Mar 2025 12:13:36 -0500 Subject: [PATCH 129/129] Combine separate ConfigureNVBench calls to fix cpp conda builds (#18155) Two PRs merged in quick succession each added a separate transform benchmark with the same name, leading CMake to get angry: https://github.com/rapidsai/cudf/commit/5b0a85b5397b69155fe0c740185945a9fe0848ac https://github.com/rapidsai/cudf/commit/8ca4bc43d4650ae364d1f9ee412a5597f310b4f7 Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Bradley Dice (https://github.com/bdice) - Basit Ayantunde (https://github.com/lamarrr) URL: https://github.com/rapidsai/cudf/pull/18155 --- cpp/benchmarks/CMakeLists.txt | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/cpp/benchmarks/CMakeLists.txt b/cpp/benchmarks/CMakeLists.txt index e82c7517145..b2559208c3c 100644 --- a/cpp/benchmarks/CMakeLists.txt +++ b/cpp/benchmarks/CMakeLists.txt @@ -355,11 +355,7 @@ ConfigureNVBench( # ################################################################################################## # * transform benchmark # --------------------------------------------------------------------------------- -ConfigureNVBench(TRANSFORM_NVBENCH transform/polynomials.cpp) - -# ################################################################################################## -# * transform benchmark ---------------------------------------------------------------------------- -ConfigureNVBench(TRANSFORM_NVBENCH transform/transform.cpp) +ConfigureNVBench(TRANSFORM_NVBENCH transform/polynomials.cpp transform/transform.cpp) # ################################################################################################## # * nvtext benchmark -------------------------------------------------------------------