[Feature Request] Create DWARFS Archive From TAR/STDIN

[gudenau]

This is a very minor thing, but it would be super useful to add a way to create DWARFS images directly from tar files or from the -O/--to-stdout flag from TAR.

(Sorry if I missed this, I didn't see a mention of it in the readme or a quick search)

Edit:
A bit more explanation for this, as it's likely in order.

If you create a dwarfs image of something that has root owned 600 files as well as non-root owned files you would need root to create the DWARFS archive.

[mhx]

Hi and thanks for your suggestion.

I believe something like this came up already in an earlier discussion, so you're not alone. :)

I agree this would be a nice feature, however it's definitely not trivial to implement. At the moment, mkdwarfs likes to memory-map files for basically any input operation, which makes it really hard to use anything other than "real" files for input.

I'll keep this open, but please don't expect any progress anytime soon.

[gudenau]

For plain TAR files you could read in the metadata and memory map the files as they are in the archive, they are back to back. As for using stdin or compressed archives it might be possible to just allocate anonymous maps as you go.

I understand that these are potentially quite invasive changes to make, I'm just trying to float some ideas.

[mhx]

Yeah, plain TAR would probably be easy enough, but usually you have .tar.{xz,bz2,gz} and it'd be hard to argue to support one but not the other. Anonymous maps would work up to a point, but probably not if we're talking tens of gigabytes of data or more. Also, unless you hold on to these mappings, you'd need to be able to rewind (hard with stdin), as mkdwarfs usually needs more than one pass over the input data.

[gudenau]

I understand that it isn't super simple, at the minimum plain old TAR would be nice because that alone would eliminate the need for root in some edge cases like I described.

[mhx]

I assume the easiest way to shoehorn this into mkdwarfs would be to provide an os_access implementation on top of a plain TAR file. I'm also leaving this gist here for reference.

[mhx]

I'm still not sure whether it should be done, but at least this would seem like a solution that wouldn't increase the complexity of the code base too much.