Make a few memory management objects public + Miscellaneous doc updates

[leofang]

Description

closes #596.

• Buffer and MemoryResource are already public (ex: returned by Device.memory_resource.allocate()). We just expose them under the cuda.core namespace for documentation purposes (Doc: Document cuda.core public (but non-entry-point) objects #601)
  • I noticed that Buffer's destructor would use the default stream if stream is not passed explicitly. This isn't right because it takes away the control from the memory resource authors. We now always let the memory resource make the call.
  • I noticed Buffer has an __init__(). This isn't right (since we want buffers to be returned by MR.allocate()) if we want to make it public. I moved this support to from_handle().
• The new exposed objects DeviceMemoryResource and LegacyPinnedMemoryResource are named after their respective cccl-rt/cudax counterparts.
• The __cuda_stream__ protocol has a protocol type now (IsStreamT)
• Internal refactoring to consolidate support for objects that have __cuda_stream__
• Documentation is clarified and expanded to reflect the new public exposure of these classes, in particular many type hints are updated/fixed
• Update the docs to mention all other merged PRs targeting this release (v0.3.0)

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[leofang]

/ok to test a6387a0

[leofang]

/ok to test cc0f6ce

[leofang]

/ok to test 58323ac

[kkraus14]

It's a bit inconsistent where only this API supports __cuda_stream__ protocol, but many other APIs only work with explicit Stream type objects. We should probably be consistent.

Maybe push this to a follow up PR?

[leofang]

We already have public examples for PyTorch and CuPy showcasing the use of this protocol. With this PR we made it slightly faster in favor of our own Stream (through try-except). Perhaps we should instead encourage users to use our native objects, and state that using the protocol could add a slight overhead?

[kkraus14]

I think we should discuss this. This will introduce non-negligible overhead, similar to DLPack, where we might be better off having a user explicitly wrap their stream objects in a cuda.core.Stream and reuse that Stream object as needed.

[leofang]

Yes right now only launch() and Device.create_stream() support the protocol, due to the exact concern of adding overheads. Overhead aside, I think essentially we are assessing which interoperability story we want to tell:

• We give people a convenient way for them to convert their types to ours, and only accept our types in our APIs
• We accept everyone's types in our APIs

Both stories need the protocol so it's not a waste. Just how to use the protocol is different.

[kkraus14]

Yes, we need the protocol regardless. My point was that we document here that an object that supports __cuda_stream__ protocol can be used anywhere cuda.core expects a stream-like object, which isn't currently true. We should do one of following things here:

1. Update the docs to make it clear that a user is required to wrap their __cuda_stream__ protocol supporting object in a cuda.core.Stream before passing it to other cuda.core APIs.
2. Update the cuda.core implementation to support IsStreamT objects everywhere Streams are an API argument.
3. Update the docs to make it clear which APIs explicitly allow IsStreamT objects vs required explicit Stream objects.
   • I'm -1 on this because it is confusing and non-intuitive to users

Based on the current state of implementation, I think we're the closest to 1 (minus the launch() API) where we should probably make the docs reflect that for the time being?

[leofang]

I'll work on 1 now.

[leofang]

@kkraus14 could you check out commit bbc1c65? I try to rephrase to hint that we want people to wrap their streams, instead of accepting them as-is. We probably have to defer the change of launch() to a later time. Vlad's graph support (nicely) took advantage of launch() supporting __cuda_stream__. We can look into it during graph phase 2.

[leofang]

/ok to test bbc1c65

[leofang]

CI is green now.

[github-actions]

Doc Preview CI
Preview removed because the pull request was closed or merged.