7.3.3 Flows

[davramov]

Now that things have settled with infrastructure for 8.3.2, it's time to start extending this work to other beamlines.

Goals:

• Look into how file watching is currently implemented at 7.3.3 (meet with Wiebke, Dylan, and David)
• Create a new Prefect server
  • Maybe use Prefect v3 instead of v2 since we are starting fresh
• Once the Globus endpoint is running (should be soon), we can implement automated transfer code and other desired flows, similar to the 8.3.2
  • Move/store data long-term on NERSC
• Implement better authentication
  • OAuth2Proxy
  • KeyCloak -- Look into whether it makes more sense to have a different realm per beamline with user assignments or one realm with adjustable user permissions?
• See if there is any analysis code that BL scientists want to run after scans are completed that we can deploy as Prefect Flows, and call them automatically