Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store workflow execution trace in user's storage account #173

Closed
olesya13 opened this issue Mar 31, 2023 · 3 comments
Closed

Store workflow execution trace in user's storage account #173

olesya13 opened this issue Mar 31, 2023 · 3 comments
Labels
enhancement New feature or request future planning To be prioritized in future needs discussion Team discussion is needed: for example: discuss technical directions, alternatives etc. TES Priority: P2 Groomed to a Priority 2 issue Troubleshooting Enable users to identify and debug errors

Comments

@olesya13
Copy link
Contributor

Problem:
We collect information about TesTask and store it as json in PostgreSQL database. This information might be useful for debugging, workflow performance tuning, etc, but it is not accessible for user when it's stored in database.

Solution:
Store TesTask information in user's storage account.

At this moment for each workflow run we generate 3 files in container 'outputs': metadata.json, outputs.data, timing.html. We might create one more file trace.json that will contain all TesTasks for that workflow run.

In Scheduler.cs (OrchestrateTesTasksOnBatch) when tesTask changes state to terminal state (completed, failed) we append tesTask to trace.json as JSONL.

Additional context
This work overlaps with the work required for having TES as a self-contained executable. Requires discussion of implementation details to fit into future work on self-contained executable.

@olesya13 olesya13 added enhancement New feature or request needs discussion Team discussion is needed: for example: discuss technical directions, alternatives etc. labels Mar 31, 2023
@BMurri
Copy link
Collaborator

BMurri commented Jun 5, 2023

TES doesn't have scope for workflow-level features. This looks like an issue that should instead be in the microsoft/CromwellOnAzure repo instead (TriggerService, specifically, which creates those three files mentioned and has access to the TesTasks in the PostgreSQL db).

@ngambani ngambani added the future planning To be prioritized in future label Jun 16, 2023
@ngambani ngambani added the Troubleshooting Enable users to identify and debug errors label Oct 16, 2023
@MattMcL4475
Copy link
Collaborator

Possible solution: #360

@MattMcL4475 MattMcL4475 added the TES Priority: P2 Groomed to a Priority 2 issue label Dec 11, 2023
@BMurri
Copy link
Collaborator

BMurri commented Feb 29, 2024

More task information is presented to cromwell (so workflows failed by failing tasks now have execution failure metadata provided both to cromwell's caller(s) as well as in cromwell's workflow logs. This information includes links (without SAS tokens) to trace logs from task executions that are today stored in the user's storage account.

@BMurri BMurri closed this as completed Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request future planning To be prioritized in future needs discussion Team discussion is needed: for example: discuss technical directions, alternatives etc. TES Priority: P2 Groomed to a Priority 2 issue Troubleshooting Enable users to identify and debug errors
Projects
None yet
Development

No branches or pull requests

4 participants