Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add CSV votes cleanup strategy #80

Open
wa0x6e opened this issue Jun 16, 2023 · 6 comments
Open

feat: add CSV votes cleanup strategy #80

wa0x6e opened this issue Jun 16, 2023 · 6 comments

Comments

@wa0x6e
Copy link
Contributor

wa0x6e commented Jun 16, 2023

Votes CSV reports will grow in count and size over time, and will increase the AWS storage cost.

We should have a prune strategy, which remove old and files that were not accessed for some time. In case someone request them again, they can trigger the cache creation and download them again.

A very basic strategy:

  • Get all files older than 6 months.
  • If the file was not accessed within the last 3 months, remove it from cache, freeing space
@ChaituVR
Copy link
Member

I think it is ok to keep this cache 🙈 @bonustrack what do you think? would it be costly?

@ChaituVR
Copy link
Member

I didn't try before, but maybe something like the expiry date will help this case (if we need this)

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putBucketLifecycle-property

@wa0x6e
Copy link
Contributor Author

wa0x6e commented Jun 21, 2023

I think there is no api currently returning the bucket size. Only way is to consult the admin panel.

We do not have analytics, but I believe that old proposals are forgotten forever, there is no need to keep it’s cache. In the worst case, user can trigger the cache again.

Setting expiry is a possibility, will look into it

@ChaituVR
Copy link
Member

If expiry doesn't work, another idea is to have subdirectories with auto incremental numbers for every six months

@wa0x6e
Copy link
Contributor Author

wa0x6e commented Jun 30, 2023

I didn't try before, but maybe something like the expiry date will help this case (if we need this)

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putBucketLifecycle-property

This one is expiring the whole bucket, there seems to be no way to expire files individually.

Subdirectory method works only if we expire by creation time, but this is not the case, we expire also by last access date, in addition of creation time.

@ChaituVR
Copy link
Member

ChaituVR commented Jul 2, 2023

Yea we can just delete files once in a while if it is adding lot of cost,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants