Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Local object cache for cached_httpfs extension #88

Merged
merged 3 commits into from
Oct 10, 2024
Merged

Local object cache for cached_httpfs extension #88

merged 3 commits into from
Oct 10, 2024

Commits on Oct 10, 2024

  1. Local object cache for cached_httpfs extension 

    * To cache remote object, user need to explicilty use
  `duckdb.cache(path, type)` function. Path is remote HTTPFS/S3/GCS/R2
  object path and type is either `parquet` or `csv` indicating remote
  object type. Caching will not be triggered on normal `SELECT` queries.

* When file is currently being cached and if we run in parallel another
  query targeting same remote object - second query will fail because it
  will not be able to take read lock.

* PG_DATADIR/duckdb_cache will be used as cache directory
    @mkaruza
    mkaruza committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    f61a002 View commit details
    Browse the repository at this point in the history

  2. Reuse CachedFile if MetadataCache is found. 

    * When MetadataCache for file is found we need to reuse Cached File
  information for reading in thread.
    @mkaruza
    mkaruza committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    aafb662 View commit details
    Browse the repository at this point in the history

  3. Changes after rebase on top of HEAD

    @mkaruza
    mkaruza committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    75e0018 View commit details
    Browse the repository at this point in the history