Support Apache ORC File Format, and Use Sparse Index

[hrh007]

1. Support a new file format: apache orc
2. Use sparse index(page level, row group level) in orc to speed up.

[hrh007]

Is this feature planned? @alamb @andygrove

[alamb]

Hi @hrh007 -- I do not know of any plans to support ORC at this time.

I think we would welcome a contribution if you would like to do so.

[hrh007]

Hi @hrh007 -- I do not know of any plans to support ORC at this time.

I think we would welcome a contribution if you would like to do so.

I will contribute to this issue, but it may take a lot time; because there is no official ORC implementation of the Rust language

[alamb]

I will contribute to this issue, but it may take a lot time; because there is no official ORC implementation of the Rust language

That does sound like an important dependency to implement first 🤔

[andygrove]

There is https://github.com/DataEngineeringLabs/orc-format but it builds on arrow2 rather than arrow-rs

[Jefffrey]

Hi @hrh007 just wondering if there's been progress on this? If not then I'm interested in picking this up

[hrh007]

Hi @hrh007 just wondering if there's been progress on this? If not then I'm interested in picking this up

I have not made any progress yet; Glad you can participate, Thanks for your contribution!

[Jefffrey]

See apache/arrow-rs#4980

[hrh007]

See apache/arrow-rs#4980

Great job， thanks for your efforts ！！！

[alamb]

BTW I think we need some more help to get ORC implemented: apache/arrow-rs#4980 (comment)

[Jefffrey]

Following discussion in apache/arrow-rs#4980

We will focus on implementing ORC file format support for Arrow in https://github.com/datafusion-contrib/datafusion-orc first

Which when ready could be used by DataFusion to query

Eventually we hope https://github.com/datafusion-contrib/datafusion-orc could be merged into arrow-rs which DataFusion could then use directly

[waynexia]

Drafted a short-term roadmap for datafusion-orc