problem publishing large file to azure blob storage

[kobelavaerts]

Hello everyone,

I hope it's OK for me to ask this here.
I have run into a problem when trying to publish a large file to azure blob storage.

We run our pipeline with executor Local but want to write some of our output files together with the process .command.* files and the report, timeline and trace files to an azure blob storage container.

This works for the most part except for one file.
We have one process that creates a compressed tar.gz archive of intermediate output files. This is a relatively big file of 1.7 TB. When trying to publish this file to the blob storage, nextflow trows error 409. The uncommitted block count cannot exceed the maximum limit of 100,000 blocks.

.nextflow.log excerpt:

Jan-27 19:01:49.779 [FileTransfer-thread-67] ERROR c.a.s.common.StorageOutputStream - com.azure.storage.blob.models.B
lobStorageException: Status code 409, "<U+FEFF><?xml version="1.0" encoding="utf-8"?><Error><Code>BlockCountExceedsLi
mit</Code><Message>The uncommitted block count cannot exceed the maximum limit of 100,000 blocks.
RequestId:1dbd201e-a01e-0035-6db0-132d0d000000
Time:2022-01-27T19:01:49.7131543Z</Message></Error>"

All other output files and log files of other processes are successfully published to the azure blob container.

We thought maybe the file was to big, but when we try to upload the file manually to the same azure blob container using azcopy we don't get this error and the file is successfully uploaded.

As I'm not able to resolve the problem I would appreciate any suggestions for fixing this issue.

more information below:

Nextflow version 21.10.0, build 5640 (11-11-2021 18:27 UTC)

nextflow.config file excerpt

azure {
  storage {
    accountName = params.azstorage
    sasToken = params.azsastoken
  }
}

// specify the output folders
params.tostoredir = "az://" + params.azblob + "/ToStore"
params.tosenddir = "az://" + params.azblob + "/ToSend"
params.logdir = params.tostoredir + "/log"

// Save the run report and trace files in azure blob storage
report {
  enabled = true
  file = "az://${params.azblob}/ToStore/log/${params.runname}-report.html"
}
trace {
  enabled = true
  file = "az://${params.azblob}/ToStore/log/${params.runname}-trace.txt"
}
timeline {
  enabled = true
  file = "az://${params.azblob}/ToStore/log/${params.runname}-timeline.html"
}

nextflow process:

process create_archive{
    publishDir "${params.tosenddir}", pattern: "*.{gz,txt}", mode: "copy"
    publishDir "${params.logdir}/${task.process}/${task.hash}", pattern: ".*", mode: "copy"

    errorStrategy "retry"

    input:
    val(project_name)
    val(demuxdone)

    output:
    tuple val(project_name), file("**")
    path(".*")

    script:
    """
    tar --use-compress-program="pigz -p ${task.cpus}" -cf ${params.runname}-RawData-${project_name}.tar.gz -C ${params.rawdata} ${project_name}
    md5sum ${params.runname}-RawData-${project_name}.tar.gz > ${params.runname}-RawData-${project_name}.md5sum.txt
    """
}

.command.sh:

#!/bin/bash -ue
tar --use-compress-program="pigz -p 12" -cf 220124_A01044_0162_BH5VJYDSX3-RawData-3971.tar.gz -C /mnt/sdb1/220124_A01044_0162_BH5VJYDSX3/RawData 3971
md5sum 220124_A01044_0162_BH5VJYDSX3-RawData-3971.tar.gz > 220124_A01044_0162_BH5VJYDSX3-RawData-3971.md5sum.txt

Please let me know if you want additional information.

[kobelavaerts]

I did some more digging and have a bit of additional information about the issue.

We did some calls to the azure Blob Service REST API to look at the uncommitted and committed blocks of the file.

When listing the blocks from the blob where the tar.gz file should have been written to, we see that there are no committed blocks and there are indeed 100 000 uncommitted blocks. We also see that each block is 4194304 B, corresponding to 4.0 MB. If we have 100k blocks of this size, we get 390 GB. This explains why the tar.gz file failed to upload since 1.7 TB is more than the maximum allowed 390 GB.

example command used:

curl \
'https://[my-account].blob.core.windows.net/[blob-container]/[path/to/blob/tar.gz]?comp=blocklist&blocklisttype=all&[SAS-token] \
-o azure_blocks.txt \
-H "Accept: application/xml"

When we use the same commands on a successfully uploaded blob, we see only committed blocks and no uncommitted blocks. The committed blocks here are again 4.0 MB.

When we look at the blob of the same tar.gz file that we uploaded using azcopy ourselves, we see that there are 27746 committed blocks with a size of 67108864 B or 64 MB per block . In total this corresponds to 1.69 TB which is the correct size of the file.

When going through the source code of the nf-azure plugin it seems like nextflow uses azcopy as well to transfer files to and from blob containers. link. The plugin seems to use azcopy version 10.13.0 which is the same version we have installed.

My question now is why is the block size limited to 4 MB and can the block size be increased to allow us to publish larger files to our blob container?
@pditommaso would you be able to help with this?

[Tomekke]

@vsmalladi You're correct. We go from Azure VM -> Blob storage but then we bring back our data to the local server. And due to security reasons outside of our control we can't mount a file share on that local server. So in short we need azcopy to copy from Azure VM to blob storage, hence our request.
Cheers
Edit: Replying directly to thread.

[abhi18av]

Another way is to use Azure File share to mount to VM and just deposit all the files in the storage account.

Agreed, for transferring large files generated within a task, to blob storage azcopy might not be the optimal choice - instead you can make use of Azure File Share integration with Nextflow which should alleviate these transfer pains.

However, given the permission constraint mentioned above, I think that blob fuse might be worth exploring since it can be mounted on the VM and offers further customizations and caching.

That being said, I'm still looking into why the block size parameter isn't as expected. Apologies for the delay in update here, got caught up in something.

[kobelavaerts]

@vsmalladi Thank you for the suggestion, like @Tomekke mentioned this is unfortunately not an option for us.

@abhi18av Thanks a lot for your efforts, I'm happy you're still looking into it.

Indeed as my colleague mentioned, we run the pipeline not with the azurebatch executor but with local executor. My apologies for the misunderstanding, I did mention it in my original post but I could have been clearer.

We didn't find the .azcopy_log file but when looking at the source code I think it might have something to do with us not using the azurebatch executor. Sadly this means we can't look at the azcopy command used or the output.

For me it's not clear which classes and methods are only loaded when using azurebatch executor vs when the azure scope is used without the azurebatch executor. But if there are differences there, I was thinking if it's maybe possible that an older version of azcopy is used when the azurebatch executor isn't set? The reason I bring this up is because from the documentation it seems that with older versions of azcopy (service versions prior to 2016-05-31), the block-size limit was 4MB which would correspond with the block size of 4MB we see on all our blobs published by nextflow.

I'm a little out of my depth here but what do you think about it? Maybe it's worth checking out.

[abhi18av]

Thanks for the link @kobelavaerts , using --log-level debug (the default is info) - I was able to confirm that the API version used is 2019-12-12

   Content-Length: [10485760]
   User-Agent: [AzCopy/10.13.0 Azure-Storage/0.14 (go1.17.2; darwin)]
   X-Ms-Client-Request-Id: [xxxxxxxxxxxxxxxxxxxxxxxxxxx]
   X-Ms-Version: [2019-12-12]

Given that you are also using the same version of azcopy as per #2610 (comment) . Therefore, I am inclined to think that might not be the reason why this is happening.

In any case, I have been able to at least reproduce this issue on my end and am currently working on enhancements to the PR to address this issue.

[abhi18av]

Updates:

How I reproduced it locally. Basically, have a process which produces a file of a specific size

process PUBLISH_REPORTS {
    publishDir "${params.outdir}"

    output:
        path("*.tmp")

    script:
        """
        mkfile -n 50m file_50mb.tmp
        """

}

• Setup the creds in nextflow.config for the following

profiles {

    azurebatch {

        params {
            outdir = "az://${AZURE_STORAGE_CONTAINER_NAME}/results"
        }

        azure {
            azcopy {
                blockSize = "10"
                blobTier = "Hot"
            }

            storage {
                accountName = "$AZURE_STORAGE_ACCOUNT_NAME"
                accountKey = "$AZURE_STORAGE_ACCOUNT_KEY"
            }
        }
    }
}

• Then using the compiled abhinav/azcopy-config branch (mentioned in the thread above) you can launch the pipeline

$ launch.sh run main.nf -profile azurebatch

... we run the pipeline not with the azurebatch executor but with local executor.

So, the crucial bit here is that when the Azure Batch executor isn't used, which is the only place where AzBashLib is used and consequently, the AzCopyOpts isn't being invoked at all.

Instead, the Java SDK for Azure Storage is used for transferring the file to the blob container using this code here.

TLDR: The azcopy isn't used to transfer files from local executor to the blob storage. However, if the executor is azurebatch then azcopy will be invoked.

Given that there is a BLOB_DEFAULT_UPLOAD_BLOCK_SIZE field within the BlobClient class, I think those are the options we need to look into, in order to accomplish what you're requesting wrt the local executor and publishing files to Blob container.

This might be a major refactor for the copy strategy to allow further customizability, however, that's beyond the time I have on my hands atm 😅

I do suggest that you look into the blobfuse for alleviating this need in the meantime and mount the blob container in your VM.

[abhi18av]

(Reposting the response in the thread here to summarize the overall issue.)

Updates:

How I reproduced it locally. Basically, have a process which produces a file of a specific size

process PUBLISH_REPORTS {
    publishDir "${params.outdir}"

    output:
        path("*.tmp")

    script:
        """
        mkfile -n 50m file_50mb.tmp
        """

}

• Setup the creds in nextflow.config for the following

profiles {

    azurebatch {

        params {
            outdir = "az://${AZURE_STORAGE_CONTAINER_NAME}/results"
        }

        azure {
            azcopy {
                blockSize = "10"
                blobTier = "Hot"
            }

            storage {
                accountName = "$AZURE_STORAGE_ACCOUNT_NAME"
                accountKey = "$AZURE_STORAGE_ACCOUNT_KEY"
            }
        }
    }
}

• Then using the compiled abhinav/azcopy-config branch (mentioned in the thread above) you can launch the pipeline

$ launch.sh run main.nf -profile azurebatch

... we run the pipeline not with the azurebatch executor but with local executor.

So, the crucial bit here is that when the Azure Batch executor isn't used, which is the only place where AzBashLib is used and consequently, the AzCopyOpts isn't being invoked at all.

Instead, the Java SDK for Azure Storage is used for transferring the file to the blob container using this code here.

TLDR: The azcopy isn't used to transfer files from local executor to the blob storage. However, if the executor is azurebatch then azcopy will be invoked.

Given that there is a BLOB_DEFAULT_UPLOAD_BLOCK_SIZE field within the BlobClient class, I think those are the options we need to look into, in order to accomplish what you're requesting wrt the local executor and publishing files to Blob container.

This might be a major refactor for the copy strategy to allow further customizability, however, that's beyond the time I have on my hands atm 😅

I do suggest that you look into the blobfuse for alleviating this need in the meantime and mount the blob container in your VM.