Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run zip import in a transaction. #3031

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

losfair
Copy link

@losfair losfair commented Aug 1, 2022

On SQLite VFSes where transaction start and commit are high-latency operations, this improves ZIP import performance a lot. Should help under the default settings too, as transaction commit involves at least one fsync.

With mvsqlite this reduces the time for importing demo.zip from tens of seconds to ~0.6sec.

Related: #3029

@zadam
Copy link
Owner

zadam commented Aug 1, 2022

Hi, in general having transactional import would be beneficial. But one significant problem is that this approach forces the import mechanism to first read the whole ZIP into memory and removes the streaming aspect from the equation. This will cause problems when importing large ZIPs.

@zDEFz
Copy link

zDEFz commented Aug 3, 2022

Hi, in general having transactional import would be beneficial. But one significant problem is that this approach forces the import mechanism to first read the whole ZIP into memory and removes the streaming aspect from the equation. This will cause problems when importing large ZIPs.

Can't you just add more swap then or memory?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants