Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use DirectIO on linux #16

Open
fulmicoton opened this issue Dec 6, 2022 · 9 comments
Open

Use DirectIO on linux #16

fulmicoton opened this issue Dec 6, 2022 · 9 comments
Assignees

Comments

@fulmicoton
Copy link
Contributor

fulmicoton commented Dec 6, 2022

We don't need to stress the pagecache with the recordlog. Just adding the custom flag on the OpenOptions should do the trick.

@evanxg852000 evanxg852000 self-assigned this Dec 7, 2022
@evanxg852000
Copy link
Collaborator

evanxg852000 commented Dec 7, 2022

@fulmicoton from my understanding to bypass the pagecache we need this .custom_flags(libc::O_DIRECT | libc::O_SYNC).

  • O_DIRECT have some alignment requirement that needs to be satisfied (I have been working on a solution with std::fs)
  • I still need to confirm O_DIRECT works in tokio::fs am not sure yet this works in tokio.

Right now I want to confirm with you if it's these flags you were thinking of or any other flag. As I am still exploring the possibilities on this issue.

@fulmicoton
Copy link
Contributor Author

We call fsync explicitly (and not always). No need for O_SYNC.

@fulmicoton
Copy link
Contributor Author

I still need to confirm O_DIRECT works in tokio::fs am not sure yet this works in tokio.

Stock tokio does not do anything special with regard to io (unfortunately).

This is just regular file descriptor running regular blocking io.
tokio simply runs them on its blocking thread pool.

@evanxg852000
Copy link
Collaborator

@fulmicoton I have been trying to figure out why the DIRECT_IO was failing.
The record log is designed to do IO on 32Kib, which is indeed device page memory-aligned but the issue is that currently we use a BufWriter in between and call sync -> buffer.flush at few places where the data is not necessarily aligned. We cannot also rely on the BufWritter flush mechanism (when full) since it can happen any time.

What we could do is remove the BufWriter in between and handle the alignment with our own buffering (more involved).

After doing a small test on this, I wonder if it's worth it as It seems the direct_IO is slower, this implementation has extra allocation but the difference is too much. See and Run https://replit.com/@evanxg852000/DirectIO

@fulmicoton
Copy link
Contributor Author

What do you mean by failing?
Can you share your branch?

Your replit is on an unoptimized build and I don't know what you are running.

@evanxg852000
Copy link
Collaborator

evanxg852000 commented Jan 9, 2023

What do you mean by failing?

DIRECT_IO should be memory aligned so flush operations fail with Invalid input meaning the problem of memory alignment.

Can you share your branch?

#19

Yes, the replit is an unoptimized build, but I was thinking both are running in the same context. I am still experimenting with measuring.
The replit is running on (the same config as my machine):

VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

@evanxg852000
Copy link
Collaborator

I have updated the replit to run the release build release
The margin is still large on both local box and replit with release

@fulmicoton
Copy link
Contributor Author

ok let's not use direct io then. Can you see if we can fadvise that we don't need this to be page cached for too long?

@evanxg852000
Copy link
Collaborator

Ok will check that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants