Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NSETM-2279] Use shared memory to write partial DataFrames of features #33

Merged
merged 1 commit into from
Apr 24, 2024
Merged

[NSETM-2279] Use shared memory to write partial DataFrames of features #33

merged 1 commit into from
Apr 24, 2024

Commits on Apr 24, 2024

  1. Use shared memory to write partial DataFrames 

    - Improve performance of report extraction and features calculation by writing partial DataFrames to the shared memory (or temp directory, if shared memory is not available).
- Use zstd compression instead of snappy when writing parquet files.
- When repo is pickled, extract the DataFrames only if they aren't already stored in the cache.
- Remove fastparquet extra dependency.
    @GianlucaFicarelli
    GianlucaFicarelli committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    7edbf63 View commit details
    Browse the repository at this point in the history