Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Create DWARFS Archive From TAR/STDIN #181

Open
gudenau opened this issue Nov 29, 2023 · 6 comments
Open

[Feature Request] Create DWARFS Archive From TAR/STDIN #181

gudenau opened this issue Nov 29, 2023 · 6 comments

Comments

@gudenau
Copy link

gudenau commented Nov 29, 2023

This is a very minor thing, but it would be super useful to add a way to create DWARFS images directly from tar files or from the -O/--to-stdout flag from TAR.

(Sorry if I missed this, I didn't see a mention of it in the readme or a quick search)

Edit:
A bit more explanation for this, as it's likely in order.

If you create a dwarfs image of something that has root owned 600 files as well as non-root owned files you would need root to create the DWARFS archive.

@mhx
Copy link
Owner

mhx commented Nov 30, 2023

Hi and thanks for your suggestion.

I believe something like this came up already in an earlier discussion, so you're not alone. :)

I agree this would be a nice feature, however it's definitely not trivial to implement. At the moment, mkdwarfs likes to memory-map files for basically any input operation, which makes it really hard to use anything other than "real" files for input.

I'll keep this open, but please don't expect any progress anytime soon.

@gudenau
Copy link
Author

gudenau commented Nov 30, 2023

For plain TAR files you could read in the metadata and memory map the files as they are in the archive, they are back to back. As for using stdin or compressed archives it might be possible to just allocate anonymous maps as you go.

I understand that these are potentially quite invasive changes to make, I'm just trying to float some ideas.

@mhx
Copy link
Owner

mhx commented Nov 30, 2023

Yeah, plain TAR would probably be easy enough, but usually you have .tar.{xz,bz2,gz} and it'd be hard to argue to support one but not the other. Anonymous maps would work up to a point, but probably not if we're talking tens of gigabytes of data or more. Also, unless you hold on to these mappings, you'd need to be able to rewind (hard with stdin), as mkdwarfs usually needs more than one pass over the input data.

@gudenau
Copy link
Author

gudenau commented Dec 1, 2023

I understand that it isn't super simple, at the minimum plain old TAR would be nice because that alone would eliminate the need for root in some edge cases like I described.

@mhx
Copy link
Owner

mhx commented Dec 1, 2023

I assume the easiest way to shoehorn this into mkdwarfs would be to provide an os_access implementation on top of a plain TAR file. I'm also leaving this gist here for reference.

@mhx
Copy link
Owner

mhx commented Dec 1, 2023

I'm still not sure whether it should be done, but at least this would seem like a solution that wouldn't increase the complexity of the code base too much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants