ipfs-car CLI hangs with large directory

[bryanblaz]

Why when I type in this function it doesn't do anything but sits there

[vasco-santos]

Hello @bryanblaz

What do you mean by sits there? The usage of CLI should be:

--pack <path> --output <my.car>
--unpack <my.car> --output <path>

[bryanblaz]

Hello vasco-santos,
Thanks for replying, but it just sits there whenever I run the commend

[vasco-santos]

@bryanblaz can you try with a single file to understand if it is an issue with large data?

How big is your images folder and how many files you have there?

[bryanblaz]

@vasco-santos I've tried a singles file, something happens.

The images folder is 6GB and there are 4000 images in it, but I thought the size didn't matter.

[Ysh-22]

Same issue here, but it's a little different from yours.
I was somehow able to generate my CAR file, but my issue happened while uploading to nft.storage
I got the following error.
Error uploading: The requested file could not be read, typically due to permission problems that have occurred after a reference to a file was acquired.
My file is a little over 9 gigs and it has around 11k files

[vasco-santos]

The images folder is 6GB and there are 4000 images in it, but I thought the size didn't matter.

It doesn't "matter" in Node.js because we write temporary files in disk. However, we have some known issues with large directories with lot's of files. We are tracking this in nftstorage/nft.storage#928 and aim to solve soon.

@Ysh-22

You are trying to upload in the browser, right? It's browser dependant but I believe Chrome has 1-2GB memory limit and I believe this is the root of the the error message you're receiving. This is because we can currently only stream data with Node.js, not browser

We're building a desktop uploader and also making changes under the hood to allow full streaming of the content.

[blahah]

The example in the screenshot above 'hangs' because the command includes a single unescaped quote, so the command is not executed. The > on its own line indicates that the command is still being entered. You need to double-quote escape the filename, then it will run.