March 2025 ASF Board Report (March 12)

[alamb]

Related Items:

• Coordination Google Doc: https://docs.google.com/document/d/11b2GEmPh5gblWWegeZi3G38e97vRqHSRElkLTwZHrjY/edit
• Mailing Lis: https://lists.apache.org/thread/7g8b66wdhpdj9tn77ptzy2790bj3l47d

Is your feature request related to a problem or challenge?

Per https://whimsy.apache.org/roster/committee/datafusion the DataFusion ASF board report schedule is

March, June, September, December

Describe the solution you'd like

I would like to draft a board report for the ASF board meeting, ideally with community help.

The meetings are typically in the second or third week of the month

Describe alternatives you've considered

I plan to do this in the same style that worked well in Arrow (see an example from @andygrove
here https://lists.apache.org/thread/7w4mgy98qomc6drvj2fo81gvhq6p0boc) --

• make a google doc (in addition to this issue) that people can add relevant content to
• Solicit community feedback for the various sub projects
• The chair (me for the time being) submits it to the board
• Create a ticket for the next report

Here is a past example:

• December 2024 ASF Board Report #10157

Additional context

No response

[alamb]

Here is a google doc to coordinate board reporting: https://docs.google.com/document/d/11b2GEmPh5gblWWegeZi3G38e97vRqHSRElkLTwZHrjY/edit?tab=t.0

Please feel free to post comments to the doc or this ticket.

Current draft is below.

Description:

The mission of Apache DataFusion is the creation and maintenance of software
related to an extensible query engine

Project Status:

Current project status: New + Ongoing (high activity)
Issues for the board: None

Membership Data:

Apache DataFusion was founded 2024-04-16 (10 months ago)
There are currently 43 committers and 15 PMC members in this project.
The Committer-to-PMC ratio is roughly 3:1.

Community changes, past quarter:

• Jonah Gao was added to the PMC on 2024-12-16
• Piotr Findeisen was added as committer on 2024-12-03
• Ruiqiu Cao was added as committer on 2024-12-10
• Yongting You was added as committer on 2025-01-18

Note that almost all communication for DataFusion and its subprojects happens on github and so our dev mailing list traffic is fairly light.

Project Activity:

Overall

DataFusion core

45.0.0 was released on 2025-02-07.
44.0.0 was released on 2024-12-31.

https://github.com/apache/datafusion

Releases continue monthly and the project has been very active with many commits a day. It seems many new projects have been using DataFusion for query processing which brings more contributors but also means we are spending more time fielding questions and figuring out how many more features to accept.

Bruce Ritchie recently authored a blog about some of the features and the outlook for the next 6 months.

We have been focusing more recently on pre-release testing and making it easier for downstream consumers to use DataFusion, which is still a challenge given how fast the project is moving.

Sub project: DataFusion Python

https://github.com/apache/datafusion-python

PYTHON-45.2.0 was released on 2025-02-23.
PYTHON-44.0.0 was released on 2025-02-07.
PYTHON-43.1.0 was released on 2024-12-12.

Sub project: DataFusion Comet

https://github.com/apache/datafusion-comet

COMET-0.6.0 was released on 2025-02-17.
COMET-0.5.0 was released on 2025-01-17.

Sub project: DataFusion Ballista

https://github.com/apache/datafusion-ballista

(New!) Sub project: DataFusion Ray

https://github.com/apache/datafusion-ray

This is a new project aims to make it easier to run DataFusion in a distributed environment using the https://www.ray.io/ compute engine

Sub project: Sqlparser

We have made two releases since sqlparser became part of DataFusion.

• SQLPARSER-0.54.0 was released on 2025-01-23.
• SQLPARSER-0.53.0 was released on 2024-12-18.

Ifeanyi / iffyio is doing a great job reviewing PRs to keep the code consistent and flowing.

Community Health:

While we as always struggle to get enough code review capacity, we have many
active committers, and the community in general helps each other out with
reviews. We continue to actively grow our committer and PMC ranks.

We had several in person meetups in Chicago, Boston, and Amsterdam, though we don’t have any more

[alamb]

FYI @iffyio and @robtandy in case you wanted to suggest any additions for datafusion ray / sqlparser

[alamb]

Mailing list announcement: https://lists.apache.org/thread/7g8b66wdhpdj9tn77ptzy2790bj3l47d

[kevinjqliu]

I feel like there are a few items from the blog post that would be great to include in the report.

Specifically,

In the core DataFusion repo alone we reviewed and accepted almost 1600 PRs from 206 different committers, created over 1100 issues and closed 751 of them 🚀. All changes are listed in the detailed changelogs.

and

DataFusion has put in an application to be part of Google Summer of Code with a number of ideas for projects with mentors already selected. Additionally, some ideas on how to make DataFusion an ideal selection for university database projects such as the CMU database classes have been put forward.

For DataFusion Python, I think support for FFI TableProvider (#12920) would be a great callout, along with the new user documentation on FFI (apache/datafusion-python#1031)

[robtandy]

Working so hard at the moment to get DataFusionRay 0.1.0 out! Hopefully we can do that before the announcement and then there should be plenty to add.

[alamb]

Working so hard at the moment to get DataFusionRay 0.1.0 out! Hopefully we can do that before the announcement and then there should be plenty to add.

I don't think the board report needs lots of detail -- just highlights are fine (that show progress). Your update is alread good in my opinion @robtandy -- no need to do more

[robtandy]

ha! ok good. I consider myself off the hook for a more detailed update!

[alamb]

ha! ok good. I consider myself off the hook for a more detailed update!

Indeed -- ship the code! Not reports!

[alamb]

I have incorporated @robtandy and @kevinjqliu 's comments. Here is the current draft

## Description:
The mission of Apache DataFusion is the creation and maintenance of software 
related to an extensible query engine

## Project Status:
Current project status: New + Ongoing (high activity)
Issues for the board: None

## Membership Data:
Apache DataFusion was founded 2024-04-16 (10 months ago)
There are currently 43 committers and 15 PMC members in this project.
The Committer-to-PMC ratio is roughly 3:1.

Community changes, past quarter:
- Jonah Gao was added to the PMC on 2024-12-16
- Piotr Findeisen was added as committer on 2024-12-03
- Ruiqiu Cao was added as committer on 2024-12-10
- Yongting You was added as committer on 2025-01-18

Note that almost all communication for DataFusion and its subprojects happens on github and so our dev mailing list traffic is fairly light.

## Project Activity:

### Overall

DataFusion is participating in Google Summer of Code with a number of ideas for projects with mentors already selected[1][2][3]. Additionally, some ideas on how to make DataFusion an ideal selection for university database projects such as the CMU database classes have been put forward.

[1]: https://github.com/apache/datafusion/issues/14577 
[2]: https://summerofcode.withgoogle.com/programs/2025/organizations/apache-datafusion

[3]: https://datafusion.apache.org/contributor-guide/gsoc_application_guidelines.html


### DataFusion core

https://github.com/apache/datafusion

- 46.0.0 was released on 2025-03-07.
- 45.0.0 was released on 2025-02-07.
- 44.0.0 was released on 2024-12-31.

Releases continue monthly and the project has been very active with many commits a day. It seems more new projects have been using DataFusion for query processing, which brings more contributors but also means we are spending more time fielding questions and figuring out how many more features to accept.

Bruce Ritchie recently authored a [blog] about some of the features and the outlook for the next 6 months. A relevant quote:

> In the core DataFusion repo alone we reviewed and accepted almost 1600 PRs from 206 different committers, created over 1100 issues and closed 751 of them 🚀. 

We have been focusing more recently on pre-release testing and making it easier for downstream consumers to use DataFusion, which is still a challenge given how fast the project is moving. 

[blog]: https://datafusion.apache.org/blog/2025/02/20/datafusion-45.0.0/

### Sub project: DataFusion Python

https://github.com/apache/datafusion-python

- PYTHON-45.2.0 was released on 2025-02-23.
- PYTHON-44.0.0 was released on 2025-02-07.
- PYTHON-43.1.0 was released on 2024-12-12.

We have been working on making it easier to interoperate with other systems, including support for FFI TableProvider ([#12920]) and new user documentation on FFI [#1031]

[#12920]: https://github.com/apache/datafusion/pull/12920 
[#1031]: https://github.com/apache/datafusion-python/pull/1031 

### Sub project: DataFusion Comet

https://github.com/apache/datafusion-comet

- COMET-0.6.0 was released on 2025-02-17.
- COMET-0.5.0 was released on 2025-01-17.

You can read about the recent happenings in Comet in the [0.6.0 blog]


[0.6.0 blog]: https://datafusion.apache.org/blog/2025/02/17/datafusion-comet-0.6.0/


### Sub project: DataFusion Ballista

https://github.com/apache/datafusion-ballista

- BALLISTA-44.0.0 was released on 2025-03-05.

There has been some renewed interest in this project as the foundation for distributed query engines, and we made a new release recently.

### (New!) Sub project: DataFusion Ray

https://github.com/apache/datafusion-ray 

This is a new project aims to make it easier to run DataFusion in a distributed environment using the https://www.ray.io/ compute engine

Contributors are working hard at the moment to get DataFusionRay 0.1.0 out! Hopefully we can do that before the announcement and then there should be plenty to add.

### Sub project: sqlparser-rs

https://github.com/apache/datafusion-sqlparser-rs 

We have made two releases since sqlparser became part of DataFusion. 

- SQLPARSER-0.55.0 was released on 2025-03-05.
- SQLPARSER-0.54.0 was released on 2025-01-23.
- SQLPARSER-0.53.0 was released on 2024-12-18.

Ifeanyi Ubah (iffyio) is doing a great job reviewing PRs to keep the code consistent and flowing. 

## Community Health:

While we as always struggle with code review capacity, we have many
active committers, and the community in general helps each other out with
reviews. We continue to actively grow our committer and PMC ranks.

We had several in person meetups in Chicago, Boston, and Amsterdam, and are working on organizing one in London in April 2025[1].

[1]: https://github.com/apache/datafusion/discussions/14647

[alamb]

Submitted!

Final Draft

## Description:
The mission of Apache DataFusion is the creation and maintenance of software 
related to an extensible query engine

## Project Status:
Current project status: New + Ongoing (high activity)
Issues for the board: None


## Membership Data:
Apache DataFusion was founded 2024-04-16 (a year ago)
There are currently 44 committers and 15 PMC members in this project.
The Committer-to-PMC ratio is roughly 3:1.

Community changes, past quarter:
- Jonah Gao was added to the PMC on 2024-12-16
- Parth Chandra was added as committer on 2025-03-10
- Yongting You was added as committer on 2025-01-18


## Project Activity:

### Overall

DataFusion is participating in Google Summer of Code with a number of ideas
for projects with mentors already selected[1][2][3]. Additionally, some ideas
on how to make DataFusion an ideal selection for university database projects
such as the CMU database classes have been put forward.

[1]: https://github.com/apache/datafusion/issues/14577
[2]: https://summerofcode.withgoogle.com/programs/2025/
 organizations/apache-datafusion
[3]: https://datafusion.apache.org/
contributor-guide/gsoc_application_guidelines.html


### DataFusion core

https://github.com/apache/datafusion

- 46.0.0 was released on 2025-03-07.
- 45.0.0 was released on 2025-02-07.
- 44.0.0 was released on 2024-12-31.

Releases continue monthly and the project has been very active with many
commits a day. It seems more new projects have been using DataFusion for query
processing, which brings more contributors but also means we are spending more
time fielding questions and figuring out how many more features to accept.

Bruce Ritchie recently authored a [blog] about some of the features and the
outlook for the next 6 months. A relevant quote:

> In the core DataFusion repo alone we reviewed and accepted almost 1600 PRs
  from 206 different committers, created over 1100 issues and closed 751 of
  them 🚀.

We have been focusing more recently on pre-release testing and making it
easier for downstream consumers to use DataFusion, which is still a challenge
given how fast the project is moving.

[blog]: https://datafusion.apache.org/blog/ 
2025/02/20/datafusion-45.0.0/

### Sub project: DataFusion Python

https://github.com/apache/datafusion-python

- PYTHON-45.2.0 was released on 2025-02-23.
- PYTHON-44.0.0 was released on 2025-02-07.
- PYTHON-43.1.0 was released on 2024-12-12.

We have been working on making it easier to interoperate with other systems,
including support for FFI TableProvider ([#12920]) and new user documentation
on FFI [#1031]

[#12920]: https://github.com/apache/datafusion/pull/12920
[#1031]: https://github.com/apache/datafusion-python/pull/1031

### Sub project: DataFusion Comet

https://github.com/apache/datafusion-comet

- COMET-0.6.0 was released on 2025-02-17.
- COMET-0.5.0 was released on 2025-01-17.

You can read about the recent happenings in Comet in the [0.6.0 blog]


[0.6.0 blog]: https://datafusion.apache.org/blog/
 2025/02/17/datafusion-comet-0.6.0/


### Sub project: DataFusion Ballista

https://github.com/apache/datafusion-ballista

- BALLISTA-44.0.0 was released on 2025-03-05.

There has been some renewed interest in this project as the foundation for
distributed query engines, and we made a new release recently.

### (New!) Sub project: DataFusion Ray

https://github.com/apache/datafusion-ray

This is a new project aims to make it easier to run DataFusion in a
distributed environment using the https://www.ray.io/ compute engine

Contributors are working hard at the moment to get DataFusionRay 0.1.0 out!
Hopefully we can do that before the announcement and then there should be
plenty to add.

### Sub project: sqlparser-rs

https://github.com/apache/datafusion-sqlparser-rs

We have made two releases since sqlparser became part of DataFusion.

- SQLPARSER-0.55.0 was released on 2025-03-05.
- SQLPARSER-0.54.0 was released on 2025-01-23.
- SQLPARSER-0.53.0 was released on 2024-12-18.

Ifeanyi Ubah (iffyio) is doing a great job reviewing PRs to keep the code
consistent and flowing.

## Community Health:

While we as always struggle with code review capacity, we have many active
committers, and the community in general helps each other out with reviews. We
continue to actively grow our committer and PMC ranks.

We had several in person meetups in Chicago, Boston, and Amsterdam, and are
working on organizing one in London in April 2025[1].

[1]: https://github.com/apache/datafusion/discussions/14647

I will file a follow on ticket for the next report and then close this ticket

[alamb]

Next report tracked in

• June 2025 ASF Board Report #15182