Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache for google-bucket stored files #52

Open
dvayanu opened this issue Jan 14, 2025 · 0 comments
Open

Cache for google-bucket stored files #52

dvayanu opened this issue Jan 14, 2025 · 0 comments

Comments

@dvayanu
Copy link
Member

dvayanu commented Jan 14, 2025

When we use google buckets to store files the retrieval is pretty time intensive (and probably costly).
Instead of retrieving every time from the bucket, lets add a local cache (configured in the same file as the rest of bucket).

Case 1 - No file stored in cache.

  • Retrieve file from google bucket and store it locally.
  • Retrieve meta-information from the bucket and store it locally in the cache.
  • Return the file

Case 2 - File stored in cache.

  • Read and return file to user.
  • Afterwards, in another executor, asynchronously, retrieve meta information about the file from bucket. If the file hasn't changed stop.
  • If the file has changed retrieve and store both the file and the meta-information (same as 1).

Therefor the file will always be outdated once, but update by next request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant