In case user is still active -- I think it would be useful to report to the long running users, after reaching some threshold on any of those folders (e.g. 50MB) asking to clean them up.

Hi @asmacdo, should we add a separate point here about monitoring and reporting the quotas of cache directories for active users?

meanwhile it might be worth creating a simple data record schema to store those records as well so they could be reused by the tools to assemble higher level stats etc.

Since S3 buckets can now mount to EC2 instances (reference: August 2023 blog post) and S3 costs are ~10X cheaper than EFS, as part of this data retention work perhaps we should also look into what it would take to move to S3 storage (and discuss any features that would not be available with this migration)?

We were previously considering providing a /scratch directory for each user that is automatically cleaned up after 30 days. In addition to the policy for the /home/jovyan directory, do we also want to implement a /scratch directory with a 30 day clean up policy?

I suggested a quota of 1 TB for home directories as many datasets are getting to be quite large. This would provide temporary, high-capacity storage, but hopefully users won't get anywhere near this threshold. This would cost $300/user/month for standard EFS, and $23/user/month if we move to Standard S3.

If we implement a scratch directory, then perhaps the home directory can have a much smaller quota.