Unable to unzip archives 4GB or greater in size (Zip64 Issue)

[pnicholls]

I have stumbled on a strange bug that I am not even sure how to start debugging so I thought I'd throw up an issue.

I have a largish (9.7GB) packmatic created zip which contains a bunch of images that will not open if the file is clicked in finder however if the file is automatically launched from Safari once the download is compete then it has no problem unzipping the file.

Sorry about such a vague issue, I would be more than happy to send the zip over to someone or do anything else to help debug this one, just let me know.

Thanks for the great package btw.

[pnicholls]

Did some more digging, it seems like once a zip reaches a certain size that it becomes unable to be opened.

[pnicholls]

Ok so it looks like 4gb is the "certain size"... this seems like it might be an issue with how this lib handles zip64 based files.

@alvises reading the readme it sounds like you were originally a big help in getting the encoding built, is there any tips you might have that would be me fix this issue? Thanks!

[pnicholls]

This test exists

packmatic/test/packmatic/packmatic_test.exs

Line 170 in 2774fb9

assert {_, 0} = System.cmd("zipinfo", [context.file_path])

however I don't believe it is actually testing the validity of the file produced

[pnicholls]

I think something similar like this might be required? Touffy/client-zip#34

[evadne]

@pnicholls I wrote the encoding engine.

Will investigate, it can either be a macOS issue, or more likely, indeed an issue within Packmatic and its emitted headers.

[evadne]

AppNote states, regarding the compressed/uncompressed sizes, https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.2.0.txt,

           The size of the file compressed and uncompressed,
          respectively.  If bit 3 of the general purpose bit flag
          is set, these fields are set to zero in the local header
          and the correct values are put in the data descriptor and
          in the central directory.  If an archive is in zip64 format
          and the value in this field is 0xFFFFFFFF, the size will be
          in the corresponding 8 byte zip64 extended information 
          extra field.  If encrypting the central directory and general 
          purpose bit flag 13 is set indicating masking, the value stored 
          for the uncompressed size in the Local Header will be zero. 

Packmatic always used placeholder and emitted an additional Zip64 Extended Information extra field for the files. I agree, this can be changed, so if the file size is within the expressible range then it should also be emitted here. Whether to omit Zip64 header however is another story.

I remember a similar issue detected on earlier releases of macOS.

[sgessa]

By the way OTP 27.1 adds support for:

• zip64 archives - Archives larger than 4GB or with more than 2^32 entries

[hrd-2]

Facing same issue. my zip file size 8.2GB. Any solution?