(aspirational hack project) Auto-generate EFS usage report & Log storage

[asmacdo]

We will run a github action at 1 month intervals (or whatever) that generates the efs-usage report from kubernetes, commits it to a private datalad repo.

Github action:

• report generation and retrieve log: kubectl apply -f inspect-pvc-pod.yaml && kubectl logs -f $(kubectl get pods -o jsonpath="{.items[?(@.metadata.name=='inspect-pvc-pod')].metadata.name}") (ish)
• execution via datalad run (@yarikoptic this is the usecase for datalad publishing some containers)
• push updated datalad repo to some repo

(Side Bonus: GH Action with access to EKS builds towards GitOps goal)

[yarikoptic]

if you do this way (storing into EFS itself), better to IMHO

Suggested change

	du -sh $EFS_HOME/* | awk '{print $1, $2}' > $EFS_HOME/size_report.txt
	cd "$EFS_HOME" && d8 -sh * | awk '{print $1, $2}' > size_report.txt

then you would avoid having some irrelevant path prefix there.

[yarikoptic]

needs cleanup

Suggested change

	# Keep the container running
	sleep 3600

[yarikoptic]

ATM we are transferring 10TB but good amount of space is occupied by a few users, e.g. the largest (500G) is by one user having 1 single NWB file. We could/should quickly inspect and reach out asking if could be removed.

But ideally we need a simple script which may be prints N largest files per each user and sorts users by the total size or smth like that....

# cd /mnt/efs-persist/home; for u in *; do biggies=$(find $u -size +1000000 -print0 | xargs -0 -r ls -Sl | head -n 5 | tr '\n' '>'); [ -z "$biggies" ] || { du -sm $u; echo  "${biggies}" | tr '>' '\n'; }; done

but ran on the original efs

here is the pod definition we used for pod to mount target EFS

apiVersion: v1
kind: Pod
metadata:
  name: sleep-tender
spec:
  containers:
  - name: inspect-pvc-container
    image: busybox
    command:
        - /bin/sh
        - -c
        - ls -la /mnt/efs-persist; echo sleeping; sleep 3600; echo exiting
    volumeMounts:
    - name: efs-storage
      mountPath: /mnt/efs-persist
      readOnly: true
  volumes:
  - name: efs-storage
    persistentVolumeClaim:
      claimName: efs-persist