Thanks @gcormier . The deployer already adds an Application Insights instance to the resource group, so the work here is to plumb it through for the Azure Batch tasks in the BatchScheduler class. This would also enable the CPU/memory/etc. to be viewed in the Azure Batch Explorer. We'd certainly appreciate the help if you and/or the team (or anyone else) can add this. Otherwise, this is definitely something we want to get done this year.

More info here: https://docs.microsoft.com/en-us/azure/batch/monitor-application-insights

UPDATE - The Genomics team is working on a way to capture and report max cpu, disk, and memory utilization, then upload the info into cosmosdb.

As of 2.2, you can get detailed metrics on disk utilization from cosmosdb. We'll use this item to track progress on feeding the rest of this information to application insights to make accessible after the run.

TODO enable Batch and Application Insights integration. Since we don't use autopools anymore, Pools are longer-lived, and this feature makes more sense to implement.

@MattMcL4475 this looks like a long pending issue open since 2020, can this be closed and reopened later when we decide to work on it or would you say this is something we intend to do in 2024?

I would love to see this in 2024, but I wonder how oversubscribed we might be. However, I fear that if we close this, it's more likely that we'd forget about it rather than remember to reopen/recreate it later.