Experimental GCP MC workspace creation support #2981

aherbst-broad · 2024-08-07T18:32:52Z

Context

This is a proof-of-concept demonstrating the ability for Rawls to handle the creation and fetching of GCP MC workspaces (i.e., GCP workspaces managed by WSM).

This PR

Adds code in the MultiCloudWorkspaceService to create GCP workspaces in WSM. This behavior is flagged and requires that a) the parent billing project has a billing profile and b) the system:mc_gcp flag is passed in the attributes map in the creation request
- Deletion seems to "just work", cloning does not.
- I've removed a check that the entity service was making to allow storage of tabular data in Rawls'.
- I make no claims about workflows
- There is no affordance in the UI to make these requests so this is an API only PoC at the moment
Cleans up and adds documentation in related areas

…es in Rawls; formatting, comments and breadcrumbs

* add handler for SQL exceptions * temp logic to trigger exception in a bee * remove code used for bee testing * Update exception message Co-authored-by: Marc Talbott <[email protected]> --------- Co-authored-by: Marc Talbott <[email protected]>

* Preliminary refactor into `PreparedSubmission` * Address `Stream` => `LazyList` deprecation * Unify root path calculation * `scalafmtAll`

…2974)

…2-SNAPSHOT (#2987)

…5-SNAPSHOT (#2986)

#2985)

…st (#2992)

...cala/org/broadinstitute/dsde/rawls/dataaccess/workspacemanager/HttpWorkspaceManagerDAO.scala

rtitle · 2024-08-14T17:54:26Z

core/src/main/scala/org/broadinstitute/dsde/rawls/workspace/MultiCloudWorkspaceService.scala

+      workspaceOpt <-
+        if (isMcRequest(workspaceRequest, billingProfileOpt)) {
+          billingProfileOpt
+            .traverse { profileModel =>


I haven't compiled but -- does this give you an Option[Option[..]]? This may be a use case for .flatTraverse

Check out 5a10420

rtitle · 2024-08-14T17:55:04Z

core/src/main/scala/org/broadinstitute/dsde/rawls/entities/EntityManager.scala

-    if (!WorkspaceType.RawlsWorkspace.equals(requestArguments.workspace.workspaceType)) {
+    // Azure workspaces use a WDS application in the data plane, Rawls entity service requests are not supported.
+    if (
+      WorkspaceType.McWorkspace.eq(


Thinking out loud, wondering what might be a good way to represent the "flavor" of workspace in the Rawls model. I guess we have 3 cases that we're supporting, right?

Rawls workspace - GCP

MC workspace - GCP

MC workspace - Azure

It's seems like there's a 4th one, the Rawls w/"Rawls Stage" MC Workspace.

*nod *

I wonder if it's worth explicitly enumerating all the cases, instead of relying on combinations of certain fields to distinguish them.

Yeah, it's admittedly awkward. I'll see about evolving the data model.

rtitle · 2024-08-14T17:56:13Z

core/src/main/scala/org/broadinstitute/dsde/rawls/workspace/MultiCloudWorkspaceService.scala

@@ -331,6 +359,20 @@ class MultiCloudWorkspaceService(override val ctx: RawlsRequestContext,
      } yield profileModel
    }

+  /**


Thanks for adding scaladoc comments, I want to do the same in many places in Leo

core/src/main/scala/org/broadinstitute/dsde/rawls/workspace/MultiCloudWorkspaceService.scala

…ute/rawls into arh-exp-wor-1104-mk2

Testing

cd04a77

aherbst-broad force-pushed the arh-exp-wor-1104-mk2 branch 3 times, most recently from 3463083 to ce9feee Compare August 7, 2024 18:44

[WOR-1104] Add very rough, gated support for creating MC GCP workspac…

9543b5f

…es in Rawls; formatting, comments and breadcrumbs

aherbst-broad force-pushed the arh-exp-wor-1104-mk2 branch from ce9feee to 9543b5f Compare August 7, 2024 18:51

aherbst-broad changed the title ~~arh exp wor 1104 mk2~~ Experimental GCP MC workspace creation support Aug 7, 2024

aherbst-broad force-pushed the arh-exp-wor-1104-mk2 branch from be3825e to 8a4b3b2 Compare August 14, 2024 15:20

Set google project ID and bucket name on creation

463d02c

aherbst-broad force-pushed the arh-exp-wor-1104-mk2 branch from 8a4b3b2 to 67b89f8 Compare August 14, 2024 15:32

blakery and others added 12 commits August 14, 2024 11:41

add handler for SQL exceptions (#2980)

1458383

* add handler for SQL exceptions * temp logic to trigger exception in a bee * remove code used for bee testing * Update exception message Co-authored-by: Marc Talbott <[email protected]> --------- Co-authored-by: Marc Talbott <[email protected]>

WOR-1797 background processes using rawls SA (#2956)

b6da7b7

WX-1711 Refactor submission settings into PreparedSubmission (#2982)

7379008

* Preliminary refactor into `PreparedSubmission` * Address `Stream` => `LazyList` deprecation * Unify root path calculation * `scalafmtAll`

[WOR-1706] Allow owners to remove themselves from billing projects (#…

de25d08

…2974)

Update sam-client from v0.0.264 to v0.0.267 (#2990)

d74c551

Update sentry-logback from 7.12.1 to 7.13.0 (#2989)

4aaba47

Update google-cloud-nio from 0.127.20 to 0.127.21 (#2988)

8ab213a

Update workspace-manager-client from 0.254.1139-SNAPSHOT to 0.254.114…

430e233

…2-SNAPSHOT (#2987)

Update billing-profile-manager-client from 0.1.562-SNAPSHOT to 0.1.56…

3016de0

…5-SNAPSHOT (#2986)

AJ-1027: submission monitor uses same methods as APIs to save entities (

891445c

#2985)

[WX-1800] Fix Rawls not returning 404 when method version doesn't exi…

756ec89

…st (#2992)

Update

7de8cb6

aherbst-broad force-pushed the arh-exp-wor-1104-mk2 branch from 67b89f8 to 7de8cb6 Compare August 14, 2024 15:43

aherbst-broad and others added 2 commits August 14, 2024 11:43

Merge branch 'develop' into arh-exp-wor-1104-mk2

520f024

Merge branch 'develop' into arh-exp-wor-1104-mk2

189349a

aherbst-broad requested review from rtitle, a team, cahrens and blakery and removed request for a team August 14, 2024 15:58

rtitle reviewed Aug 14, 2024

View reviewed changes

...cala/org/broadinstitute/dsde/rawls/dataaccess/workspacemanager/HttpWorkspaceManagerDAO.scala Show resolved Hide resolved

cahrens marked this pull request as ready for review August 14, 2024 17:39

rtitle reviewed Aug 14, 2024

View reviewed changes

core/src/main/scala/org/broadinstitute/dsde/rawls/workspace/MultiCloudWorkspaceService.scala Show resolved Hide resolved

rtitle reviewed Aug 14, 2024

View reviewed changes

core/src/main/scala/org/broadinstitute/dsde/rawls/workspace/MultiCloudWorkspaceService.scala Outdated Show resolved Hide resolved

aherbst-broad added 2 commits August 15, 2024 08:54

Fix futures

a52f21d

Merge branch 'arh-exp-wor-1104-mk2' of https://github.com/broadinstit…

fe78dd7

…ute/rawls into arh-exp-wor-1104-mk2

cahrens approved these changes Aug 15, 2024

View reviewed changes

aherbst-broad and others added 3 commits August 15, 2024 09:36

Fix tests

40addea

Clean up

5a10420

Merge develop

1473ab9

aherbst-broad closed this Aug 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experimental GCP MC workspace creation support #2981

Experimental GCP MC workspace creation support #2981

aherbst-broad commented Aug 7, 2024 •

edited

Loading

rtitle Aug 14, 2024

aherbst-broad Aug 19, 2024

rtitle Aug 14, 2024 •

edited

Loading

cahrens Aug 14, 2024

rtitle Aug 14, 2024

aherbst-broad Aug 15, 2024

rtitle Aug 14, 2024

Experimental GCP MC workspace creation support #2981

Experimental GCP MC workspace creation support #2981

Conversation

aherbst-broad commented Aug 7, 2024 • edited Loading

Context

This PR

rtitle Aug 14, 2024

Choose a reason for hiding this comment

aherbst-broad Aug 19, 2024

Choose a reason for hiding this comment

rtitle Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

cahrens Aug 14, 2024

Choose a reason for hiding this comment

rtitle Aug 14, 2024

Choose a reason for hiding this comment

aherbst-broad Aug 15, 2024

Choose a reason for hiding this comment

rtitle Aug 14, 2024

Choose a reason for hiding this comment

aherbst-broad commented Aug 7, 2024 •

edited

Loading

rtitle Aug 14, 2024 •

edited

Loading