Expose a path for converting bytes::Bytes into arrow_buffer::Buffer without copy

[chebbyChefNEQ]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

It would be nice if we can call let buffer: Buffer = bytes.into() without incurring a copy, since object_store returns bytes::Bytes.

Describe the solution you'd like

I see that arrow_buffer::bytes::Bytes already implements From<bytes::Bytes> here

I guess we just need to add a implementation of From<bytes::Bytes> for Buffer

Describe alternatives you've considered

Additional context

[tustvold]

As you've identified this should be a relatively straightforward addition. I will, however, temper your expectations that this will only be possible when the buffer is sufficiently aligned for the array - you will need to be careful here to avoid panics

[chebbyChefNEQ]

Yeah, I noticed that the in memory object store implementation doesn't guarantee 8-byte aligned buffers, which is sufficient for most types.

I was debating if I should open an issue and make object store return aligned buffers. Since there is no mmap in the object store I think this is doable?

[tustvold]

Since there is no mmap in the object store I think this is doable?

The challenge is that hyper, etc... use Bytes to avoid copying data when they slice up HTTP data frames. I'm therefore not sure this is tractable without introducing potential performance regressions. It also seems like something object_store should not be concerned with

[wjones127]

I was debating if I should open an issue and make object store return aligned buffers. Since there is no mmap in the object store I think this is doable?

I think there might be a way to get aligned buffers more optimally within the existing APIs: Right now, the body is read as a stream of Bytes and then concatenated here:

arrow-rs/object_store/src/lib.rs

Line 1021 in 6d4b8bb

GetResultPayload::Stream(s) => collect_bytes(s, Some(len)).await,

The concatenation does a mem copy already, so we could mem copy them into an buffer of our preferred alignment. The GetResult struct seems to have a pretty open API, so I think users could achieve this themselves if they want:

arrow-rs/object_store/src/lib.rs

Lines 969 to 976 in 6d4b8bb

	pub struct GetResult {
	/// The [`GetResultPayload`]
	pub payload: GetResultPayload,
	/// The [`ObjectMeta`] for this object
	pub meta: ObjectMeta,
	/// The range of bytes returned by this request
	pub range: Range<usize>,
	}

[tustvold]

The concatenation does a mem copy already

Not always, if there is only a single Bytes object it won't. But as you allude to, we're really just playing with where the copy takes place, the current API is flexible enough that users can make this judgement call themselves

[tustvold]

Closed by #4260