-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
questions regarding low req/s write and how to sync between them as ipc use #11
Comments
Thank you for your feedback and for using our library! Regarding the slow write performance at 2500 req/s, using a
This change points the synchronizer to use a shared memory object located in a If
Additionally, for optimizing the performance further, make sure to compile and run your program in release mode:
After implementing these adjustments and running in release mode, I observed a substantial increase in the server requests per second:
It's also important to note that this library is optimized for a read-heavy data access pattern, where typically 99% of operations are reads from many processes, and only about 1% are writes from a single writer. Nonetheless, by utilizing a |
@bocharov the write speed is too slow. can u check this out? the read write speed is rather balanced. however i get millions of read req/s but only thousands of writes / s |
@bocharov reader speed reader
|
@sprappcom I've looked with profiler at If your use case permits, you can consider using
With this change write requests went from 35k to 115k per second on my laptop. More performance gains are possible by optimizing https://github.com/cloudwego/shmipc-go looks interesting too, we could borrow few things from there for the future versions, although it's implemented in Go and will need to adopt logic to Rust. |
@bocharov golang is slower than rust. hope to see it faster than shmipc-go. thx by the way. p.s. : 115k writes/s vs 80 mil reads/s shmipc-go (golang using shm) does 4mil reads/writes per second. do u think there's an ETA on when this speed up is possible? |
@sprappcom good news, I was able to optimize writer performance via #12 You can expect ~4M writes/s with
On my Linux laptop with 13th Gen Intel(R) Core(TM) i7-13800H processor I'm getting:
|
@bocharov thx for the efficiency! u are great! it's really amazing. thx again. |
@bocharov given the amount of wyhash2 collision here, wont it be better to make it xxh3? possible to provide a way for users to define the hash function? i mean they can define whatever function they want including xxh3, fnv1a etc coming from golang background, something along the lines of this hash feature: some cases xxh3 is better, some fnv1a, wyhash2 is good (as mentioned online somewhere) for length < 192. p.s. : alternatively, i suggest an option for users to choose between xxh3, fnv1a or wyhash2 |
@sprappcom default hasher is wyhash not wyhash2, which has decent collision properties as per https://medium.com/@tprodanov/benchmarking-non-cryptographic-hash-functions-in-rust-2e6091077d11 It's still possible to provide different hasher to be used for checksum calculation by |
thx for the great work by the way
The text was updated successfully, but these errors were encountered: