The Batch VM is currently selected based on resource requirements of the Cromwell task. The code selects the cheapest VM that has CPU count, memory and disk size that are (all three) equal or larger than requested. For tasks requiring large disk, this increases the cost of the VM because larger disk requirement will selected the VM that also has CPU/memory much higher that wanted.

Proposed solution:
In light of the fact that TES now supports out of the box local NVMe drives which are found on more than just series L SKUs but the code does NOT use the NVMe drive presence in the selection code described above, the following is the new proposal:

1. Add total drive size of NVMe drives in any given SKU to the current VM whitelist.
2. Add Standard LRS SSD storage to the PriceApi client, and capture this in the GenerateBatchVmSkus tool
3. Change the logic to select the VM by using the greater of ResourceDiskSizeInGiB and NVMe drives total size for all SKUs where that value is >= the requested disk size, and add the additional cost of the requested disk size to the cost of the SKU for all other SKUs.
4. When generating the pool specification, if the requested disk size is greater than the larger of the ResourceDiskSizeInGiB and NVMe drives total size, add a Standard LRS disk of the requested size to the pool spec and format and mount it to the same mount point as the current NVMe start-task.

Estimated effort: 3 days

Note: Premium SSD is not available for all SKUs, Premium v2 is not available through Batch (same with Ultra). To simplify the implementation and maintain costs, Standard SSD was chosen.

Consider dividing the requested capacity by the number of additional drives that can be attached in order to pool them to improve disk I/O.

Previous proposed solution:

Change the logic to select the VM based on CPU count and memory only, then attach the disk of requested size (up to a limit, settable in the configuration?). Additionally, to further improve disk I/O, attach multiple smaller disks and pool them together into single logical volume.

See https://learn.microsoft.com/en-us/azure/virtual-machines/premium-storage-performance

Example code to run on the batch VM to prepare the volume (assumes that 4 disks are attached):

1. Create physical volumes:
   pvcreate /dev/sdc /dev/sdd /dev/sde /dev/sdf
   OR pvcreate /dev/sd[c-f]

2. Create volume group (group of physical volumes):
   vgcreate vg1 /dev/sdc /dev/sdd /dev/sde /dev/sdf
   OR vgcreate vg1 /dev/sd[c-f]

3. Create logical volume, striped over the volumes in the volume group:
   lvcreate -i 4 -I 512k -n lv1 -L $(vgs vg1 -o vg_size --noheadings) vg1 (4 is the number of stripes, needs to be equal to number of disks, 512k is the stripe size)
   OR
   lvcreate --extents 100%FREE --stripes 4 -I 512k --name lv1 vg1

4. Create the file system on the logical volume:
   mkfs -t xfs /dev/vg1/lv1

5. Mount the volume:
   mkdir -p /mnt1
   mount /dev/vg1/lv1 /mnt1