leverage ibis expression for getting readablerelations (#2046)

* add ibis dataset in own class for now

* make error clearer

* fix some linting and fix broken test

* make most destinations work with selecting the right db and catalog, transpiling sql via postgres in some cases and selecting the right dialect in others

* add missing motherduck and sqlalchemy mappings

* casefold identifiers for ibis wrapper calss

* re-organize existing dataset code to prepare ibis relation integration

* integrate ibis relation into existing code

* re-order tests

* fall back to default dataset if table not in schema

* make dataset type selectable

* add dataset type selection test and fix bug in tests

* update docs for ibis expressions use

* ensure a bunch of ibis operations continue working

* add some more tests and typings

* fix typing (with brute force get_attr typing..)

* move ibis to dependency group

* move ibis stuff to helpers

* post devel merge, put in change from dataset, update lockfile

* add ibis to sqlalchemy tests

* improve docs a bit

* fix ibis dep group

* fix dataset snippets

* fix ibis version

* add support for column schema in certion query cases

---------

Co-authored-by: Marcin Rudolf <[email protected]>

.github/workflows/test_destinations.yml

@@ -77,13 +77,8 @@ jobs:
       #     key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}-redshift
 
       - name: Install dependencies
-        run: poetry install --no-interaction -E redshift -E postgis -E postgres -E gs -E s3 -E az -E parquet -E duckdb -E cli -E filesystem --with sentry-sdk --with pipeline,ibis -E deltalake -E pyiceberg
-
-      - name: enable certificates for azure and duckdb
-        run: sudo mkdir -p /etc/pki/tls/certs && sudo ln -s /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt
-
-      - name: Upgrade sqlalchemy
-        run: poetry run pip install sqlalchemy==2.0.18  # minimum version required by `pyiceberg`
+        # if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: poetry install --no-interaction -E redshift -E postgis -E postgres -E gs -E s3 -E az -E parquet -E duckdb -E cli -E filesystem --with sentry-sdk --with pipeline,ibis -E deltalake
 
       - name: create secrets.toml
         run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

.github/workflows/test_local_destinations.yml

@@ -95,10 +95,7 @@ jobs:
           key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}-local-destinations
 
       - name: Install dependencies
-        run: poetry install --no-interaction -E postgres -E postgis -E duckdb -E parquet -E filesystem -E cli -E weaviate -E qdrant -E sftp --with sentry-sdk --with pipeline,ibis -E deltalake -E pyiceberg
-
-      - name: Upgrade sqlalchemy
-        run: poetry run pip install sqlalchemy==2.0.18  # minimum version required by `pyiceberg`
+        run: poetry install --no-interaction -E postgres -E postgis -E duckdb -E parquet -E filesystem -E cli -E weaviate -E qdrant -E sftp --with sentry-sdk --with pipeline,ibis -E deltalake
 
       - name: Start SFTP server
         run: docker compose -f "tests/load/filesystem_sftp/docker-compose.yml" up -d