WX-1385 Reject blob URLs with external SAS tokens as unparsable #7347
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aednichols
commented
Dec 15, 2023
| @@ -499,7 +499,7 @@ object Dependencies { | |||
| List("scalatest", "mysql", "mariadb", "postgresql") | |||
| .map(name => "com.dimafeng" %% s"testcontainers-scala-$name" % testContainersScalaV % Test) | |||
|
|
|||
| val blobFileSystemDependencies: List[ModuleID] = azureDependencies ++ wsmDependencies | |||
| val blobFileSystemDependencies: List[ModuleID] = azureDependencies ++ wsmDependencies ++ akkaHttpDependencies | |||
There was a problem hiding this comment.
I added this dependency because akka.http.scaladsl.model.Uri has much better query parsing compared to java.net.URI.
| val SignatureKey = "sig" | ||
|
|
||
| val query = uri.query().toMap | ||
| query.isDefinedAt(SignedVersionKey) && query.isDefinedAt(SignatureKey) |
There was a problem hiding this comment.
👏 Nice, I like this definition.
THWiseman
approved these changes
Dec 20, 2023
jgainerdewar
approved these changes
Jan 2, 2024
There was a problem hiding this comment.
This LGTM as-is but makes me think - should we also be redacting SAS tokens when printing the HTTP paths? When preparing the TES task we print all inputs and outputs, both cloud paths and localized paths. It feels like full external SAS token support should include updating that logging (and maybe other logging?) to redact the tokens.
|
@jgainerdewar I looked around a bit and couldn't immediately identify where we do the printing – if you can give me a pointer I will gladly submit a follow-on PR. |
henriqueribeiro
added a commit
to henriqueribeiro/cromwell
that referenced
this pull request
Nov 6, 2024
* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838) * Draft support for optional FileHash * Draft getMd5 for BlobPath * Resolve non-parallel IO to fix tests * Checksum validation for BlobPath * Nicer error message * Test for missing Blob hash * Break attr acquisition into separate method * Cleanup, comments * In-progress tests of blob hash command * Remove test * Remove unused import * BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831) * BT-711 Refresh SAS token for filesystem on expiry * Rough cut of token refresh using exceptions * Ignore tests, and minor cleanup * Remove stray line * Draft of manager class for handling expiring file systems * Style fixes * Refactor of blobfilesystemManager and tests covering its functionality * Refined tests to validate close filesystem as separate unit * Ignore connected tests * Clean up of some things * Refactor BlobFileSystemManager to separate file, and some other cleanup * Some additional scala-ifying * Small cleanup * Correcting imports * trigger tests * trigger tests * Batch 1 of scala steward updates (broadinstitute#6903) * Batch 1 of scala steward updates * Rollback snakeYAML * Attempt 3, with only the passing dependancies * Revert google API and Big Query udpates * Winding back other google deps * rollback remaining google updates * trigger tests * trigger tests * [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907) * BT-745 Batch 2 of scala steward updates (broadinstitute#6906) * Update SBT to 2.0.0 * Fix sbt-git import * Update mouse to 1.0.11 * Update rhino 1.7.14 * SUP-692 Retry with more memory after RC 137 (broadinstitute#6912) * Reorder execution result checks so 137 can retry with more memory * Test for memory retry after 137 RC * Fix test expectations * Make memory retry checks consistent * Revert changes to existing test * Rename retryWithMoreMemory to outOfMemoryDetected * Scala steward updates batch 3 (broadinstitute#6913) * Scala steward updates batch 3 * WX-745 Batch 4 scala steward updates (broadinstitute#6916) * WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-755 Build all images instead of just Cromwell (broadinstitute#6919) * WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923) * WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924) * WX-755 Fix environment variable syntax (broadinstitute#6926) * WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921) * Give blob SAS tokens write permission * Case class wrapper for subscription id * Resolve duplicate container name in absolute BlobPath * Ignored test demonstrating correct absolute path generation * Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala Co-authored-by: Brian Reilly <[email protected]> * PR feedback Co-authored-by: Brian Reilly <[email protected]> * [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927) * update snakeyaml to 1.33 * Don't use deprecated no-arg Constructor constructor Co-authored-by: Janet Gainer-Dewar <[email protected]> * WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925) * Upgrade Postgres to 42.4.1 (broadinstitute#6932) * WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930) * WX-772 Update Scala to 2.13.9 (broadinstitute#6928) * Update Scala to 2.13.9 * Try updating sbt-scoverage * Does this version exist anywhere we can see? * This version actually exists * Update library version to remove conflict * Codegen version * Fix fun new 2.13.9 compiler errors * Resolve warnings * Newest Scala? * I guess not * Does this please Travis? * force ci * Back out changes to generated code Co-authored-by: Adam Nichols <[email protected]> * WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935) Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1. - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * WX-808 Host allowlist for HTTP imports (broadinstitute#6938) * `hostAllowlist` that allows everything * Refactor * Stick allow list in HttpResolver * Better default config * Allow list tests * Make it build Co-authored-by: Janet Gainer-Dewar <[email protected]> * Update commons text to 1.10.0 (broadinstitute#6937) * WX-751 Token refresh signal for monitoring (broadinstitute#6939) * Log messages * `DEBUG` -> `INFO` * WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941) * Modify blob paths for TES * Make blob transformation configurable * Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala Co-authored-by: Adam Nichols <[email protected]> * Apply PR feedback in second place Co-authored-by: Adam Nichols <[email protected]> * Update changelog for wdl http allow list (broadinstitute#6944) * WM-1491 Fixing Cromwell-client (broadinstitute#6943) * More updated client for use in cbas * Removing excess code * Fix client build script (broadinstitute#6945) * WX-837: Remove CWL references from documentation (broadinstitute#6949) * wx-837 removed cwl references in markdown doc files * wx-837 removed cwlParsingOverview.md, updated mkdocs.yml * wx-837 updated cromwell.yaml, generated new RESTAPI file * WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948) * Dependencies * Compiles but no tests * Formatting * Moar exclusions * Update to latest WSM * Add additional dependency * We need some UUID here to make the request * Formatting * Clarify what is fake * Formatting * Use our own version of Jersey and Jackson stuff * Port-in Khalid's changes (thank you!) Co-authored-by: Khalid Shakir <[email protected]> * Test longevity Don't break the test if someone decides to add a cert to `ws.org` * Cleanup * Cleanup * Cleanup * Adjust TES config file for CI Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Khalid Shakir <[email protected]> * CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950) * crom-6554 removed references to PAPI v1 from doc * crom-6554 pr feedback, reworded doc to use example conf as a starting point * WX-833 Real Azure DRS Credentials (broadinstitute#6952) * Remove B2C reference from name * Get token for current user rather than getting from KeyVault * Remove KeyVault config for engine * Remove KeyVault config for DRSLocalizer * Remove KeyVault dependency * Remove KeyVault support from localizer repo template * Cleaned up and working Azure token acquisition for engine * Collapse localizer's AccessTokenStrategy into DrsCredentials * Cleanup * WX-853 Remove most CWL (broadinstitute#6955) * WX-696 Enable getting SAS token from WSM (broadinstitute#6954) * WX-696 Enable getting SAS token from WSM * Wire container resource id from config * Move resource-container-id config path * First pass at config for WSM * Remove unused singleton config * Tests for new config * Fix config parsing * Modified b2c token to be provided each time * Remove singletonConfig arg from factory * Restore types to factory configs * Clean up comments and empty token default * Default to config b2c before searching environment * Fix token default on api client * Fix test * Refactor error handling for when there is no token * Remove token constructor arg for clientProvider * Move configs to global singleton config * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala * default -> override * Add override token to test * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala Co-authored-by: Adam Nichols <[email protected]> * Parentheses * Reduce token timeout * Move AzureCredentials to separate file * Make AzureCredentials an object * WSM token cleanup * Config refactor (broadinstitute#6960) Co-authored-by: Janet Gainer-Dewar <[email protected]> * Initial blob token documentation * Refine language in BlobSasTokenGenerator * Update comment and formatting Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961) * WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962) * WX-842 Add Pact Dependency for Cromwell * Remove incomplete test spec * Initial Pact Test * Fix pact so it compiles * Add breadcrumb comment and clean up * ID-125 Add support for drshub, rename all the things (broadinstitute#6959) * Add support for drshub, rename all the things * fallback to martha if resolver is not in config * WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970) * Translate crc32c hashes to b64 for getm * Update tests * Remove obsolete b64 handling for md5, centralize hex validation * Restore old test, fix other test * WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965) * WX-859 Accept workflow execution identity in config (broadinstitute#6967) * WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981) * Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975) * WX-863 Turn off Azure NIO logging (broadinstitute#6982) * Turn off Azure NIO logging * Poke Travis * WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985) * WX-878 Single shared BlobFileSystemManager (broadinstitute#6986) * Make BlobFileSystemManager shared across all BlobPathBuilders * Update TES conf file to reflect new singleton config * Shell escape reference image files [VS-796] [WX-910] (broadinstitute#6989) * WX-769 `disks` compatibility for TES backend (broadinstitute#6991) * Update FiveMinuteIntro.md (broadinstitute#6994) * WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992) * WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003) * WX-926 Support falling back to OCI Manifest Forma * Only mount reference disks if requested [WX-925] (broadinstitute#7001) * [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status * Address feedback * Address feedback (broadinstitute#6997) * Address additional feedback (broadinstitute#7000) * Fix copy/paste error (broadinstitute#7005) * Address additional feedback * Fix copy/paste error * Trigger CI --------- Co-authored-by: Blair Murri <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996) * WX-903 Pre-GHA test suite disablement * WX-877 Update CHANGELOG for release 85 (broadinstitute#7011) * Update cromwell version from 85 to 86 * WX-905 (broadinstitute#7012) Co-authored-by: Thomas Wiseman <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-719 modernize `cromwell-publish` image (broadinstitute#7013) * WX-930 Add `CODEOWNERS` file (broadinstitute#7015) * Update FiveMinuteIntro: Java 11 requirement (broadinstitute#6830) Co-authored-by: Adam Nichols <[email protected]> * WX-952 Fix Cromwell version update in Helm chart (broadinstitute#7014) * Reference Disk Manifest Builder App Test (broadinstitute#7017) * [WM-1696] Update Womtool (broadinstitute#7019) Co-authored-by: Tom Wiseman <[email protected]> * WX-958 write_map() should write its last entry with a newline (broadinstitute#7022) * Centaur slurm (broadinstitute#7083) * Remove Deploy Key (broadinstitute#7084) * WX-950 Set user agent in Java client (broadinstitute#7087) * WX-950 Upgrade Azure libs to probably fix extraneous log (broadinstitute#7088) * WX-984 Revert "WX-950 Upgrade Azure libs to probably fix extraneous log" (broadinstitute#7090) * Add cron run integrations each day of the week (broadinstitute#7089) * 85 release (#28) * Update cromwell version from 83 to 84 * BW-1255 Implement POST /runs endpoint (broadinstitute#6779) * Adding route * Fixing HTTP method error * All formFields made optional * A compliling state * Saving * Saving * All three endpoints functioning as expected; updated RESTAPI.md * Updated response for submission from 200 to 201 to pass tests * Test submission response * Moved updated submission response to askSubmit * test * updating RESTAPI.md * saving * Adding utility file for submitRequest * cleanup * Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814) * BW-1305 Swagger Update (broadinstitute#6818) * Properly documenting metadataArchiveStatus in WorkflowQueryResult model * Update docs * BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817) BT-710 Add configs for BlobPathBuilderFactory * BW-1305 Make "name" optional in workflow query response (broadinstitute#6821) * BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816) Modify blobPathBuilder to fallback to creating a filesystem if one is not found * Logging updates: (broadinstitute#6813) * [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824) * first pass on BlobTokenGenerator with E2E test * update BlobPathBuilder constructor args in test * account -> container level client * [BT-687] specify correct types (broadinstitute#6829) * specify correct types * fix test with new type * remove type declarations in function call * remove unnecessary sas-token config * BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833) * Add tests, getting frid of WesRunRoutes.scala * wesWorkflowId fix, ec implicits errors gone * Refactoring path for GET /runs * Indentation fix * Commit to rollback * Revert "Indentation fix" This reverts commit 63fc484. * PR trigger * Optimize imports * Missed import * BW-1354 - Porting CBAS preliminary step (broadinstitute#6837) * Getting rid of shared utility file; Adding/Updating WES version of submit. * Edit spec file * Adding Wes-like error * BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826) * Update cromwell version from 84 to 85 * BW-1393 Release doc updates (broadinstitute#6839) * BT-732 Checksum validation for blobs read by engine (broadinstitute#6838) * Draft support for optional FileHash * Draft getMd5 for BlobPath * Resolve non-parallel IO to fix tests * Checksum validation for BlobPath * Nicer error message * Test for missing Blob hash * Break attr acquisition into separate method * Cleanup, comments * In-progress tests of blob hash command * Remove test * Remove unused import * BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831) * BT-711 Refresh SAS token for filesystem on expiry * Rough cut of token refresh using exceptions * Ignore tests, and minor cleanup * Remove stray line * Draft of manager class for handling expiring file systems * Style fixes * Refactor of blobfilesystemManager and tests covering its functionality * Refined tests to validate close filesystem as separate unit * Ignore connected tests * Clean up of some things * Refactor BlobFileSystemManager to separate file, and some other cleanup * Some additional scala-ifying * Small cleanup * Correcting imports * trigger tests * trigger tests * Batch 1 of scala steward updates (broadinstitute#6903) * Batch 1 of scala steward updates * Rollback snakeYAML * Attempt 3, with only the passing dependancies * Revert google API and Big Query udpates * Winding back other google deps * rollback remaining google updates * trigger tests * trigger tests * [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907) * BT-745 Batch 2 of scala steward updates (broadinstitute#6906) * Update SBT to 2.0.0 * Fix sbt-git import * Update mouse to 1.0.11 * Update rhino 1.7.14 * SUP-692 Retry with more memory after RC 137 (broadinstitute#6912) * Reorder execution result checks so 137 can retry with more memory * Test for memory retry after 137 RC * Fix test expectations * Make memory retry checks consistent * Revert changes to existing test * Rename retryWithMoreMemory to outOfMemoryDetected * Scala steward updates batch 3 (broadinstitute#6913) * Scala steward updates batch 3 * WX-745 Batch 4 scala steward updates (broadinstitute#6916) * WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-755 Build all images instead of just Cromwell (broadinstitute#6919) * WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923) * WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924) * WX-755 Fix environment variable syntax (broadinstitute#6926) * WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921) * Give blob SAS tokens write permission * Case class wrapper for subscription id * Resolve duplicate container name in absolute BlobPath * Ignored test demonstrating correct absolute path generation * Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala Co-authored-by: Brian Reilly <[email protected]> * PR feedback Co-authored-by: Brian Reilly <[email protected]> * [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927) * update snakeyaml to 1.33 * Don't use deprecated no-arg Constructor constructor Co-authored-by: Janet Gainer-Dewar <[email protected]> * WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925) * Upgrade Postgres to 42.4.1 (broadinstitute#6932) * WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930) * WX-772 Update Scala to 2.13.9 (broadinstitute#6928) * Update Scala to 2.13.9 * Try updating sbt-scoverage * Does this version exist anywhere we can see? * This version actually exists * Update library version to remove conflict * Codegen version * Fix fun new 2.13.9 compiler errors * Resolve warnings * Newest Scala? * I guess not * Does this please Travis? * force ci * Back out changes to generated code Co-authored-by: Adam Nichols <[email protected]> * WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935) Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1. - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * WX-808 Host allowlist for HTTP imports (broadinstitute#6938) * `hostAllowlist` that allows everything * Refactor * Stick allow list in HttpResolver * Better default config * Allow list tests * Make it build Co-authored-by: Janet Gainer-Dewar <[email protected]> * Update commons text to 1.10.0 (broadinstitute#6937) * WX-751 Token refresh signal for monitoring (broadinstitute#6939) * Log messages * `DEBUG` -> `INFO` * WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941) * Modify blob paths for TES * Make blob transformation configurable * Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala Co-authored-by: Adam Nichols <[email protected]> * Apply PR feedback in second place Co-authored-by: Adam Nichols <[email protected]> * Update changelog for wdl http allow list (broadinstitute#6944) * WM-1491 Fixing Cromwell-client (broadinstitute#6943) * More updated client for use in cbas * Removing excess code * Fix client build script (broadinstitute#6945) * WX-837: Remove CWL references from documentation (broadinstitute#6949) * wx-837 removed cwl references in markdown doc files * wx-837 removed cwlParsingOverview.md, updated mkdocs.yml * wx-837 updated cromwell.yaml, generated new RESTAPI file * WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948) * Dependencies * Compiles but no tests * Formatting * Moar exclusions * Update to latest WSM * Add additional dependency * We need some UUID here to make the request * Formatting * Clarify what is fake * Formatting * Use our own version of Jersey and Jackson stuff * Port-in Khalid's changes (thank you!) Co-authored-by: Khalid Shakir <[email protected]> * Test longevity Don't break the test if someone decides to add a cert to `ws.org` * Cleanup * Cleanup * Cleanup * Adjust TES config file for CI Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Khalid Shakir <[email protected]> * CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950) * crom-6554 removed references to PAPI v1 from doc * crom-6554 pr feedback, reworded doc to use example conf as a starting point * WX-833 Real Azure DRS Credentials (broadinstitute#6952) * Remove B2C reference from name * Get token for current user rather than getting from KeyVault * Remove KeyVault config for engine * Remove KeyVault config for DRSLocalizer * Remove KeyVault dependency * Remove KeyVault support from localizer repo template * Cleaned up and working Azure token acquisition for engine * Collapse localizer's AccessTokenStrategy into DrsCredentials * Cleanup * WX-853 Remove most CWL (broadinstitute#6955) * WX-696 Enable getting SAS token from WSM (broadinstitute#6954) * WX-696 Enable getting SAS token from WSM * Wire container resource id from config * Move resource-container-id config path * First pass at config for WSM * Remove unused singleton config * Tests for new config * Fix config parsing * Modified b2c token to be provided each time * Remove singletonConfig arg from factory * Restore types to factory configs * Clean up comments and empty token default * Default to config b2c before searching environment * Fix token default on api client * Fix test * Refactor error handling for when there is no token * Remove token constructor arg for clientProvider * Move configs to global singleton config * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala * default -> override * Add override token to test * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala Co-authored-by: Adam Nichols <[email protected]> * Parentheses * Reduce token timeout * Move AzureCredentials to separate file * Make AzureCredentials an object * WSM token cleanup * Config refactor (broadinstitute#6960) Co-authored-by: Janet Gainer-Dewar <[email protected]> * Initial blob token documentation * Refine language in BlobSasTokenGenerator * Update comment and formatting Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961) * WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962) * WX-842 Add Pact Dependency for Cromwell * Remove incomplete test spec * Initial Pact Test * Fix pact so it compiles * Add breadcrumb comment and clean up * ID-125 Add support for drshub, rename all the things (broadinstitute#6959) * Add support for drshub, rename all the things * fallback to martha if resolver is not in config * WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970) * Translate crc32c hashes to b64 for getm * Update tests * Remove obsolete b64 handling for md5, centralize hex validation * Restore old test, fix other test * WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965) * WX-859 Accept workflow execution identity in config (broadinstitute#6967) * WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981) * Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975) * WX-863 Turn off Azure NIO logging (broadinstitute#6982) * Turn off Azure NIO logging * Poke Travis * WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985) * WX-878 Single shared BlobFileSystemManager (broadinstitute#6986) * Make BlobFileSystemManager shared across all BlobPathBuilders * Update TES conf file to reflect new singleton config * Shell escape reference image files [VS-796] [WX-910] (broadinstitute#6989) * WX-769 `disks` compatibility for TES backend (broadinstitute#6991) * Update FiveMinuteIntro.md (broadinstitute#6994) * WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992) * WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003) * WX-926 Support falling back to OCI Manifest Forma * Only mount reference disks if requested [WX-925] (broadinstitute#7001) * [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status * Address feedback * Address feedback (broadinstitute#6997) * Address additional feedback (broadinstitute#7000) * Fix copy/paste error (broadinstitute#7005) * Address additional feedback * Fix copy/paste error * Trigger CI --------- Co-authored-by: Blair Murri <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996) * WX-903 Pre-GHA test suite disablement * WX-877 Update CHANGELOG for release 85 (broadinstitute#7011) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Katrina P <[email protected]> Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Saloni Shah <[email protected]> Co-authored-by: kshakir <[email protected]> Co-authored-by: mspector <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Brian Reilly <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Variath Thomas <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Trevyn Langsford <[email protected]> Co-authored-by: Miguel Covarrubias <[email protected]> Co-authored-by: ekiernan <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Blair Murri <[email protected]> * Remove Yaml (broadinstitute#7098) * Azure blob read/write integration test (broadinstitute#7024) Co-authored-by: Tom Wiseman <[email protected]> * Goodbye Travis (broadinstitute#7100) * Single workflow runner (broadinstitute#7106) * WX-975 Revert special TES path munging (broadinstitute#7103) * Centaur blob filesystem (broadinstitute#7104) * WX-1010 Update Github Actions set output syntax (broadinstitute#7111) * WX-1001 Upgrade JSON Small and Fast Parser (broadinstitute#7116) * WX-1015 Override glob behavior for TES to use absolute path (broadinstitute#7120) * WX-990 Make TES request backoff behavior configurable (broadinstitute#7122) * WX-1103 Revert CI to old TES polling behavior (broadinstitute#7126) * Develop aws (#29) * stuck on globbing * efs works, no callcaching * update readme * extended EFS support * fix for globbing in nested scatters * updated config for globbing, to prevent issues with empty folders * WX-1103 Accelerate TES CI (broadinstitute#7130) * Fix Horicromtal Deadlock Test (broadinstitute#7131) * WM-1963: Validate PRs begin with Jira tags (broadinstitute#7127) * WX-1106 Add logging for failed docker manifest pulls (broadinstitute#7135) Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * [WX-1108] Disable drs tests (broadinstitute#7145) * ID-377 Setup pact for consumer contract testing. (broadinstitute#7123) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1092 Support `size` engine function for public HTTP files (broadinstitute#7128) * WX-1105 Fix interpretation of full http blob paths (broadinstitute#7138) Co-authored-by: Adam Nichols <[email protected]> * Revert "ID-377 Setup pact for consumer contract testing." (broadinstitute#7146) * WX-966 Add Cascades, remove Directory from Biscayne (broadinstitute#7105) * WX-966 Post-merge branch cleanup + test fix (broadinstitute#7149) * WX-1122 Enable Azure ApplicationInsights (broadinstitute#7143) * [WX-1108] Add tests back (broadinstitute#7148) * [WX-1136] Docker Build Test (broadinstitute#7151) * [WX-1136] Self Hosted (broadinstitute#7152) * WX-1145 Fix minor regression introduced in WDL 1.1 foundation (broadinstitute#7153) * ID-377 Setup pact for consumer contract testing. Retry (broadinstitute#7147) Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Ivan <[email protected]> * WX-1133 Shorten TES localized blob paths (broadinstitute#7150) * WX-1154 Fix Slack Messaging (broadinstitute#7159) * efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs (#32) * WX-1160 Fix bad MariaDB version assumptions (broadinstitute#7160) * WX-1126 Upgrade to modern Python, 3.8 not available in package repo anymore (broadinstitute#7164) * WX-1122 Use legacy AppInsights to get better control over logging (broadinstitute#7157) * WX-1137 Replace 4-byte unicode chars in PAPI event descriptions (broadinstitute#7166) * WX-1101 Update library versions to support batch (broadinstitute#7155) Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1195 Update Azure identity libs (broadinstitute#7173) * WX-1114 Initial inclusion of Azure NIO 'fork' (broadinstitute#7168) * WX-1110[risk=low] Added endpoint to fetch failed tasks by root workflow id (broadinstitute#7165) * WX-1210 Added JIRA ID for Cromwhelm auto commit message (broadinstitute#7180) * WX-1179 DRS image build updates, remove self-hosted runners (broadinstitute#7179) * WX-1210-action-fix Use PR title to find JIRA ID for cromwhelm commit message (broadinstitute#7184) * WX-1110-query-fix Corrected Query to pull in attributes outside of executionStatus and backendStatus (broadinstitute#7185) * WX-1179 GCP Batch new base (broadinstitute#7177) Co-authored-by: dspeck <[email protected]> Co-authored-by: jarroyo28 <[email protected]> Co-authored-by: Jacob Jennings <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Tristan Garwood <[email protected]> * WM-2100: Update terra-helmfile on publish (broadinstitute#7187) * WM-2100: Correct commit and PR description of terra-helmfile auto update (broadinstitute#7188) * WX-1230 Ignore `invalidate_bad_caches_use_good_local.test` (broadinstitute#7197) * WX-1078 ACR support (broadinstitute#7192) * WX-1179 Enable GCP Batch Integration Tests (broadinstitute#7199) Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * WX-1179 GCP Batch Docs Update (broadinstitute#7196) Co-authored-by: Jacob Jennings <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * ID-734 Increase Timeout for DRSHub Communication (broadinstitute#7198) * [WX-1156] internal_path_prefix for TES 4.4 (broadinstitute#7190) * [WM-2184] Remove brackets from Jira ID (broadinstitute#7206) * WX-1153 Azure blob read md5 from metadata for large files (broadinstitute#7204) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1156 Fix internal_path_prefix (broadinstitute#7208) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1256 Temporarily turn off engine hashing for blob files (broadinstitute#7209) Co-authored-by: Adam Nichols <[email protected]> * WX-1173 Reopen filesystem for blob storage outside workspace (broadinstitute#7178) Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * WX-1174 Adjust NIO Copy functionality (broadinstitute#7207) Co-authored-by: Adam Nichols <[email protected]> * [WX-1168] TES Log Paths (broadinstitute#7210) * Develop aws (#34) * efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs * add deployment manual, fix issue with empty disks * update documentation * update documentation * update documentation * WX-1264 Don't expire an unexpirable filesystem (broadinstitute#7216) * [WX-495] DRS Parallel Downloads (broadinstitute#7214) * WX-1225 Print TES error messages to job logger (broadinstitute#7220) * WX-1217 Workflow completion callback (broadinstitute#7213) Co-authored-by: Chris Llanwarne <[email protected]> * [WM-2199] Changeset for modifying ownership of tables for WORKFLOWS app (broadinstitute#7218) Co-authored-by: Chris Llanwarne <[email protected]> * [WX-1234] Update CHANGELOG.md (broadinstitute#7227) Merging past CI since this is doc only. * WM-2252: Configurable metadata write role (broadinstitute#7225) Co-authored-by: dvoet <[email protected]> * Update cromwell version from 86 to 87 * WX-1282 Update failedJobs Query to use `lo_get` instead of INNER JOIN against pg_largeobject (broadinstitute#7228) * [WX-1234] Update Release Process with Docker Instructions (broadinstitute#7231) Co-authored-by: Adam Nichols <[email protected]> * WM-2294: Allow role-setter action to run on change (broadinstitute#7233) * WM-2296: Callback should supply fully qualified output names (broadinstitute#7234) * [WX-499] DRS Parallel Downloads Follow-up (broadinstitute#7229) * WX-1318 gcp batch: Add GPU driver install (broadinstitute#7235) Co-authored-by: Adam Nichols <[email protected]> * WX-1232 Include useful workflow ids in TES tags (broadinstitute#7221) * Options to publish status only (#36) * add options to publish status only * updated readme.md --------- Co-authored-by: quekx <[email protected]> * WX-1307 Azure E2E test (broadinstitute#7239) Co-authored-by: Janet Gainer-Dewar <[email protected]> * Fix aws unit tests (#39) * checkpoint * fix ecr and batch tests * fix AwsBatchJobSpec.scala --------- Co-authored-by: quekx <[email protected]> * return bucket directly instead of listing and checking it (#38) Co-authored-by: quekx <[email protected]> * WX-1340 GCP Batch: Mount with extra colon issue and multiple zones support (broadinstitute#7240) Co-authored-by: Adam Nichols <[email protected]> * WX-1339 Make `throwExceptionOnExecuteError` false for PAPI aborts (broadinstitute#7245) * WX-1338 Fix cron invocation of E2E test (broadinstitute#7244) * WX-1341 Better logging when a runner stops picking up new workflows (broadinstitute#7246) * Add evaluteOnExit for aws batch retry (#40) Co-authored-by: quekx <[email protected]> * [WX-1260] Acquire sas token from task runner (broadinstitute#7241) Co-authored-by: Adam Nichols <[email protected]> * [WX-1183] Ice Lake (broadinstitute#7252) * WX-1333 Improve logging visibility for load management (broadinstitute#7253) * [WX-1391] Fix Bash Bug (broadinstitute#7326) * [WX-1393] Add Content Length to Curl request (broadinstitute#7328) * [WX-1346] Scalafmt (broadinstitute#7257) * WX-1351 Speed up `Centaur Horicromtal PapiV2 Beta` (broadinstitute#7329) * WX-1351 CI CWL cleanup (broadinstitute#7327) * WX-1351 Remove slow/misbehaving localization test (broadinstitute#7330) * WX-1351 Split out restart tests (broadinstitute#7333) * [WX-1345] Automatic Token Acquisition for TES Config (broadinstitute#7256) * WX-1387 Remove unused case classes (broadinstitute#7334) * [WX-1394] Upgrade Logback Core to address CVE-2023-6378 (broadinstitute#7332) * WX-1351 Another round of CI cleanup (broadinstitute#7336) * [WX-1395] Scalafmt GitHub Action (broadinstitute#7337) * WX-1351 SBT compile speedup (broadinstitute#7339) * WX-1407 Fix contract tests (broadinstitute#7340) Co-authored-by: Ivan <[email protected]> * WX-1409 Java 17 (broadinstitute#7342) * Revert "WX-1409 Java 17 (broadinstitute#7342)" (broadinstitute#7343) * WX-1409 Restore Java 17 (broadinstitute#7342) (broadinstitute#7344) * Revert "WX-1409 Java 17" (broadinstitute#7346) * [WX-1395] Scala formatter Github Action (broadinstitute#7341) Co-authored-by: ScalaFmt Fixer <[email protected]> Co-authored-by: Broad Bot <[email protected]> * Improved tagging support (#37) * efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs * add deployment manual, fix issue with empty disks * update documentation * update documentation * update documentation * support for tagging instances and volumes used in jobs, some support for spaces in file paths * corrected workflow id in tagging * redirect exit code 137 to retry-with-more-memory routine --------- Co-authored-by: Henrique Ribeiro <[email protected]> * add gpu count (#41) * add gpu count * fix typo --------- Co-authored-by: quekx <[email protected]> Co-authored-by: Henrique Ribeiro <[email protected]> * WX-1385 Reject blob URLs with external SAS tokens as unparsable (broadinstitute#7347) * WX-1411 Require preinstalled `jq` and `curl` for just-in-time SAS feature (broadinstitute#7350) * WX-1396 Cromwell client allow additional properties (broadinstitute#7352) * WM-2428: Include full error context when failing to abort TES jobs (broadinstitute#7354) * WX-1420 Fix GCP Batch label regex restriction (broadinstitute#7355) Co-authored-by: Beibei Chen <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1385 Remove SAS tokens from TES input log printouts (broadinstitute#7358) * [WX-1184] PostgreSQL Docker Image for Local Cromwell (broadinstitute#7172) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1444 Use MySQL LTS in DBMS tests (broadinstitute#7360) * WX-1449 Add `latest` Docker tag (broadinstitute#7362) * WX-767 Upgrade Cloud SDK to 461.0.0 (broadinstitute#7361) * WX-1443 Adopt `gcloud storage` for localization only (broadinstitute#7359) * WX-964 suffix() (broadinstitute#7363) * WX-1417 New database role strategy (broadinstitute#7366) * [WM-2291] Callback API contract tests between Cromwell and CBAS (broadinstitute#7251) Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1445 Update docker image regex to handle python:3 correctly (broadinstitute#7367) * WX-1252 Runtime attributes cleanup – CWL runtime attributes (broadinstitute#7370) * WX-1485 Upgrade postgres lib (broadinstitute#7371) * WM-2454: Private GitHub support on describe api (broadinstitute#7365) * WX-1252 Runtime attributes cleanup – CWL runtime environment (broadinstitute#7369) * WX-1462 POSIX-flavored sub() (broadinstitute#7374) * [WX-963] Unzip Engine Function (broadinstitute#7368) * WX-1489 Hopefully reduce CI flakiness by modernizing deadlock test (broadinstitute#7376) * [WX-965] quote() and squote() engine functions. (broadinstitute#7375) Co-authored-by: Ryan Saperstein <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * [WX-1317] Remove Akka 'server' header from all HTTP responses (broadinstitute#7379) * WM-2461] Add support for running private workflows on Azure (broadinstitute#7373) * WX-1252 Per-backend runtime attributes (broadinstitute#7380) * ID-347 Remove Martha References (broadinstitute#7384) * WX-1461 Remove `womtool upgrade` command (broadinstitute#7382) * WX-1488 Supply cloud platform when making DRS requests (broadinstitute#7381) * WX-757 Fix workflow stuck in aborting after WDL type error (broadinstitute#7385) * [WX-1506] Add jobLogger output for subworkflow running (with parent and subworkflow ID) (broadinstitute#7388) Co-authored-by: jlester-msft <[email protected]> * WX-757 Fix `stdout`, `stderr` in workflow body causing crashes (broadinstitute#7386) * WX-1519 Stop leaking DRS Localizer images on every CI run (broadinstitute#7390) * [WX-1460] WDL 1.1 Struct Literal Parsing (broadinstitute#7391) Co-authored-by: Janet Gainer-Dewar <[email protected]> * [WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows (broadinstitute#7392) * WX-1542 Fix workflow cancellation (broadinstitute#7398) * WX-1550 Don't pointlessly continue running integration tests (broadinstitute#7400) * WX-1557 Fix default values from Configuring.md and Scaling.md (broadinstitute#7393) Co-authored-by: Adam Nichols <[email protected]> * WX-1557 Fix actor-factory in Batch101.md (broadinstitute#7377) Co-authored-by: Adam Nichols <[email protected]> * WX-1557 Add more tests to the GCP Batch backend (broadinstitute#7394) Co-authored-by: Adam Nichols <[email protected]> * [WX-1531] Struct Literal Type Checking (broadinstitute#7402) * [WX-1468] Implement `returnCodes` runtime attribute (broadinstitute#7389) Co-authored-by: Ryan Saperstein <[email protected]> Co-authored-by: Ryan Saperstein <[email protected]> * [WM-2555] Cromwell -> ECM contract test (broadinstitute#7405) * [WX-1568] Bump Akka max-response-reason-length limit (broadinstitute#7406) * WX-1530 Strip pesky URL bits when creating local paths for HTTP inputs (broadinstitute#7404) * WX-1538 Add more tests to the GCP Batch backend (broadinstitute#7410) * [WX-1605] Fix Codecov Report Uploads (broadinstitute#7418) * [WX-1410] Sanitize 4 byte UTF-8 characters before inserting into METADATA_ENTRY (broadinstitute#7414) * WX-1566 Special Docker build for debugging (broadinstitute#7417) Run `sbt -Dproject.isDebug=true server/docker` * [WX-1448] Add verbose logging and timeout for getm (broadinstitute#7416) Co-authored-by: Adam Nichols <[email protected]> * WX-1566 Fix Morgan's call cache file hash CPU thrash Cromwell crash (broadinstitute#7419) * WX-1566 Fix RTD so this ticket's docs get published (broadinstitute#7420) * fix womFileType coercion (#47) Co-authored-by: quekx <[email protected]> * PR #44: Multi-aserisk globbing by @kevinRenaersBio. Reapplied selection of changes on aws_develop branch * PR #45 reapplied to aws_develop: support for sharedMemory. skipped EFS fixes for now while waiting for feedback * upstream PR broadinstitute#7488 : fix compilation error due to swagger2markup * Fix issue with caching on EFS by replacing touch with existOrThrow routine * make sure that md5 sibling file of EFS data is newer than main file to be valid * revise sharedmemory attribute to take reguluar GB/MB specification * updated AWS README file --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Saloni Shah <[email protected]> Co-authored-by: Brian Reilly <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Katrina P <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: Justin Variath Thomas <[email protected]> Co-authored-by: Khalid Shakir <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Trevyn Langsford <[email protected]> Co-authored-by: Miguel Covarrubias <[email protected]> Co-authored-by: ekiernan <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Blair Murri <[email protected]> Co-authored-by: Thomas Wiseman <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Stephen Fleming <[email protected]> Co-authored-by: Henrique Ribeiro <[email protected]> Co-authored-by: mspector <[email protected]> Co-authored-by: Tristan Garwood <[email protected]> Co-authored-by: Ivan <[email protected]> Co-authored-by: dspeck1 <[email protected]> Co-authored-by: jarroyo28 <[email protected]> Co-authored-by: Jacob Jennings <[email protected]> Co-authored-by: Dillon Scott <[email protected]> Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: dvoet <[email protected]> Co-authored-by: xquek <[email protected]> Co-authored-by: quekx <[email protected]> Co-authored-by: ScalaFmt Fixer <[email protected]> Co-authored-by: Broad Bot <[email protected]> Co-authored-by: Beibei Chen <[email protected]> Co-authored-by: Maria Yazykova <[email protected]> Co-authored-by: rsaperst <[email protected]> Co-authored-by: Ryan Saperstein <[email protected]> Co-authored-by: jlester-msft <[email protected]> Co-authored-by: Alexis Hernandez <[email protected]> Co-authored-by: dkj <[email protected]> Co-authored-by: Ryan Saperstein <[email protected]> Co-authored-by: Alexis Hernandez <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tested by localizing a Blob file from Terra Prod, on a patched Cromwell in Terra Dev.
This PR with foreign Blob URL:
This PR with a regular HTTPS URL from the 'net:
developwith foreign Blob URL: