Check for Parquet file changes #1
Labels
feature
New feature or request
nice-to-have
Stuff that would be nice to have
optimization
Optimizing some exisiting functionality
Is your feature request related to a problem? Please describe.
Currently the parquet performance data is downloaded and processed on a regular schedule regardless of changes to the parquet file. This unnecessarily downloads and process parquet data that has already been processed.
Describe the solution you'd like
Check whether the parquet performance data has changed by verifying the HTTP Response Headers for either the Etag or Last Modified date/time. If it hasn't changed then skip the downloading and processing step. If the file has been updated since the last time the parquet file was processed, download the new parquet file and process it.
The text was updated successfully, but these errors were encountered: