Add foreign table providers (#921)

* testing FFI for table provider

* Was able to get round trip schema from datafusion -> delta table -> datafusion

* Expand file structure

* WIP on execution plan

* Working through execution plan FFI

* Using datafusion-proto for execution plan properties

* Adding plan properties parsing from ffi

* Standardize naming for FFI structs

* Intermediate testing and troubleshooting

* Adding record batch stream ffi representation

* Mimimum viable product demonstrating foreign table provider

* Move ffi module to datafusion core

* Modifications need to compile against latest DF

* Set DF to 42.0.0

* Rebasing and pulling in a few changes for DF43.0

* Add wrapper for register table provider

* Suppress deprecation warning

* Add example for FFI table provider

* Add pytest for FFI module to CI

* Add license text

* Change the name of the FFI table provider test so it doesn't try to run during the first pass of pytest when the module hasn't been built

* Build example in build stage to be used during test stage

* Combine pytests into one stage

* Fix path for unit test

* Installing maturin for ffi test in test script

* Need to install the wheel for unit test

* Add online documentation about using custom table providers

* Raise an error if method is not implemented when it is expected

.github/workflows/build.yml

@@ -156,7 +156,9 @@ jobs:
           repo-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build Python package
-        run: maturin build --release --strip --features substrait
+        run: |
+          maturin build --release --strip --features substrait
+
       - name: List Mac wheels
         run: find target/wheels/
 

.github/workflows/test.yaml

@@ -110,6 +110,15 @@ jobs:
           pip install -e . -vv
           pytest -v .
 
+      - name: FFI unit tests
+        run: |
+          source venv/bin/activate
+          pip install -e . -vv
+          pip install maturin==1.5.1
+          cd examples/ffi-table-provider
+          maturin develop --release --strip
+          pytest python/tests/_test_table_provider.py
+
       - name: Cache the generated dataset
         id: cache-tpch-dataset
         uses: actions/cache@v4