dtar failed to extract entry - line too long

[charles-d-it]

I am trying to extract a .dtar archive but getting the following:

ERROR: Failed to extract entry XX at offset YY Line too long

I am using mpifileutils/0.11.1 with openmpi/5.0.3 with the command line:

mpiexec --allow-run-as-root -n 8 dtar --progress 15 -xf .dtar

I get errors on several files, then the mpiexec process terminates.
Any assistance would be appreciated.

[charles-d-it]

Any suggestions or help on this?

[carbonneau1]

I wasn't able to replicate this one but I'm getting back on it. Do you have an idea of the directories depth and sting length by any chance?

[charles-d-it]

No, I not. Chuck Dickerson Amentum 2224 Bay Area Blvd, Houston, TX 77058 amentum.com<http://www.amentum.com/> O: 281.244.5895 ***@***.***[Logo Description automatically generated]<http://www.amentum.com/> From: Eric Carbonneau ***@***.***> Sent: Tuesday, February 25, 2025 2:05 PM To: hpc/mpifileutils ***@***.***> Cc: Dickerson, Charles {Chuck} (JSC-EG)[Jacobs Technology, Inc.] ***@***.***>; Author ***@***.***> Subject: [EXTERNAL] [BULK] Re: [hpc/mpifileutils] dtar failed to extract entry - line too long (Issue #621) CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. I wasn't able to replicate this one but I'm getting back on it. Do you have an idea of the directories depth and sting length by any chance? - Reply to this email directly, view it on GitHub<#621 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BOSOH2IV3WEPW7MUWKZFWSL2RTEGVAVCNFSM6AAAAABVKPXYCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTGE3DIOBXGU>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>> [carbonneau1]carbonneau1 left a comment (hpc/mpifileutils#621)<#621 (comment)> I wasn't able to replicate this one but I'm getting back on it. Do you have an idea of the directories depth and sting length by any chance? - Reply to this email directly, view it on GitHub<#621 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BOSOH2IV3WEPW7MUWKZFWSL2RTEGVAVCNFSM6AAAAABVKPXYCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTGE3DIOBXGU>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[carbonneau1]

The only way I was able to reproduce the issue was to create an archive with tar (not dtar). Which leads me to believe your archive was not created with dtar as the issue happens during creation and fails to create the archive. I need to fix the issue for both creation and extraction. At this point, for now, try to use tar -xf on your archive and let me know.

[charles-d-it]

I attempted to extract the file using tar -xvf, approximately 436G of the 17T *.dtar file was extracted, then the process crashed with the error message

tar: Skipping to next header
tar: Exiting with failure status due to previous errors`.

The only messages I saw were many tar: Ignoring unknown extended header keyword, I believe this may have come from extended attributes on the files because they were on a lustre filesystem.

[carbonneau1]

My findings: dtar is built on top of libcircle which limits buffers to 4KiB to sends between ranks (processes). Incidentally dtar is limited to chunks and buffers of the same size (4KiB). For dtar it means any path longer than 4K results into the error message you got: "ERROR: Failed to extract entry XX at offset YY Line too long". There is no way around this as it stands.

Even if you try to dsync both file systems, you'll encounter the same issue. How was that archive created? which tool was used?

Still looking.