prune_spot832: Permission Denied (Globus)

[davramov]

The prune_spot832 flow scheduled on flow-prd has been failing, while other pruning flows are completing successfully. Globus is generating a "permission denied" error.

07:16:48 AM
prefect.flow_runs
Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 877, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/tmp/tmpaz513h73prefect/orchestration/flows/bl832/prune.py", line 54, in prune_spot832
    prune_files(
  File "/tmp/tmpaz513h73prefect/orchestration/flows/bl832/prune.py", line 36, in prune_files
    prune_one_safe(
  File "/tmp/tmpaz513h73prefect/orchestration/globus/transfer.py", line 294, in prune_one_safe
    delete_id = prune_files(
  File "/tmp/tmpaz513h73prefect/orchestration/globus/transfer.py", line 202, in prune_files
    task_wait(
  File "/tmp/tmpaz513h73prefect/orchestration/globus/transfer.py", line 250, in task_wait
    raise TransferError(f"Received PERMISSION_DENIED, cancelling Globus task {task_id}")
orchestration.globus.transfer.TransferError: Received PERMISSION_DENIED, cancelling Globus task 0799ba5e-b89c-11ef-995f-7d33a6eb4aa1```

[davramov]

Dylan and I have been investigating why this occurs on spot832. Interestingly, it does not affect all projects and files, although it affects a majority of them.

Some notes to help us tackle this in the new year (from Dylan):

• newer directories are getting created without g+w
• This would tend to indicate the umask setting is 0022
• How is umask set in our case? From docker?
• /etc/profile is setting umask 0022
• Let's wait and see if this is still a problem with new folders

Example file permissions (notice the missing group write permission in the bottom set of folders):