Multi-threading

[thehydroimpulse]

I haven't been able to dig down into it but I want to figure out a way we can get this library to be multi-threaded. As far as I can tell, we aren't using threads/tasks at all in the tests anymore. Nanomsg in theory should be thread-safe (and that was one of the original goals versus something like ZeroMQ where it was strictly thread-unsafe.

[blabaere]

I created a multithreaded test and put the test runner inside a script loop.
See libnanomsg https://github.com/blabaere/nanomsg.rs/tree/multithreadtest
I found the following problems :

• Sometime, the test runner would just refuse to die, gdb reports: No threads. Linux Process Explorer does not show any thread and displays the process as stuck in 'futex_wait_queue_me'.
• Sometime, an assertion fails inside nanomsg :Bad file descriptor 9
• Sometime, this one : Resource temporarily unavailable 11

[thehydroimpulse]

Thanks @blabaere for digging into this. Yeah, it seems really odd. I only started hitting these issues when I had gotten far enough in implementing this library and now it seems it's never not hitting those issues. I know nanomsg said they tried to be thread-safe but there may be a few areas which are not.

[blabaere]

Finally got one example where it would hang forever but let me debug it:
#0 0x00007f51cf52c9b3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f51d0049932 in nn_poller_wait (self=0x7f51d026c644, timeout=100) at src/aio/poller_epoll.inc:186
#2 0x00007f51d004c35e in nn_worker_routine (arg=0x7f51d026c600) at src/aio/worker_posix.inc:176
#3 0x00007f51d004e67d in nn_thread_main_routine (arg=) at src/utils/thread_posix.inc:35

[blabaere]

My bad, it looks like I should add some more sync between the tasks.
To sum up, I feel like to make this work, one must make sure bind is effective before calling connect, and make sure receive is effective before calling send. And none of them is effective right away when called (this is the reason the sleep calls in the original tests)

Here, the sender task is waiting for the end of the test, and the receiver task is waiting for the message.

Receiver:

#0  0x00007ff33a09b963 in poll () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ff33abc83f8 in nn_efd_wait (self=<optimized out>, timeout=-1) at src/utils/efd.c:48
#2  0x00007ff33abc36c9 in nn_sock_recv (self=0x7ff320000930, msg=0x7ff333ffe7d0, flags=<optimized out>) at src/core/sock.c:671
#3  0x00007ff33abc0e07 in nn_recvmsg (s=<optimized out>, msghdr=0x7ff333ffe880, flags=<optimized out>) at src/core/global.c:872
#4  0x00007ff33abc110e in nn_recv (s=<optimized out>, buf=<optimized out>, len=<optimized out>, flags=<optimized out>) at src/core/global.c:707
#5  0x00007ff33b02fd6e in tests::test_receive::h0ec28f9462429bc7fqa () at src/lib.rs:335
#6  0x00007ff33b032c5d in tests::should_create_a_pipeline_mt::closure.3416 () at src/lib.rs:390

Sender:

#0  0x00007ff33a593d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007ff33b0d8eef in sync::barrier::Barrier::wait::h8f2657fc8601dd6brVp ()
#2  0x00007ff33b032311 in tests::should_create_a_pipeline_mt::closure.3377 () at src/lib.rs:378

[blabaere]

That blog also uses sleep between bind and connect, in seconds:
http://tim.dysinger.net/posts/2013-09-16-getting-started-with-nanomsg.html

[thehydroimpulse]

Interesting. I'm wondering if that's the expected procedure for nanomsg, to wait an arbitrary length of time.

[blabaere]

It seems that thread-safety is bought with asynchronous behavior in some
places. I will digg a little bit more.

Le mar. 9 déc. 2014 22:37, Daniel Fagnan notifications@github.com a
écrit :

Interesting. I'm wondering if that's the expected procedure for nanomsg,
to wait an arbitrary length of time.

—
Reply to this email directly or view it on GitHub
#70 (comment)
.

[blabaere]

Indeed, using barrier and sleep all over the place to ensure that bind/connect and send/receive correct timings fixes the problem where the test runner would hang forever.

A call to nn_term and one more sleep fixes the nanomsg assertion failures.

The ugliness can be witnessed here: https://github.com/blabaere/nanomsg.rs/blob/multithreadtest/libnanomsg/src/lib.rs

@thehydroimpulse, are we sure we want to show this kind of stuff ?

[thehydroimpulse]

@blabaere mmmm, that's a bit unfortunate.

[blabaere]

Got it right this time !

The send/receive order issue I saw imagined was only the requirement to not close the sender too early, that is before the receiver ever gets a chance to see the message. And now it works without any sleep calls, even with several parallel tests.

And the bind/connect order requirements can also be dropped. But if you implement that ordering "manually" with barriers the test runs way faster than if you let nanomsg handle the "retry" (wild guess).

And guess who looks all nice and tidy now ?https://github.com/blabaere/nanomsg.rs/blob/multithreadtest/libnanomsg/src/lib.rs

[thehydroimpulse]

@blabaere Wow, that's awesome! That looks sooo much cleaner.

[blabaere]

To check the thread-safety of individual sockets, one should be shared between several tasks. But of course, Rust will not let you do that since all the read/write operations require a mutable socket.
The mutability being specified in the traits, the only way I see to make this possible is to allow the cloning off the sockets.

@thehydroimpulse, do you see any problem with Socket having the Clone trait, or do you think there is another way to enable socket sharing between tasks ?

[thehydroimpulse]

@blabaere I don't see a huge problem with having Clone implemented for the socket. This is how the Tcp sockets in Rust work so that you can have a thread dedicated to blocking on receiving new connections and another thread can close the socket over the boundary. Otherwise there's no way to shutdown that thread.

[thehydroimpulse]

There's an implied ownership over sockets, however. Less so with nanomsg because they're not as generic as raw tcp. For example, if two threads can read and write on the same socket, you have to have some sort of implied ownership on what operations should be allowed on either thread (otherwise both threads can be reading and they both won't get the full message, it'll be split between the two)

[blabaere]

I didn't think of that message split problem, but it must be fun to debug too.

Another potential problem I see now is that nanomsg recycles the int values returned by nn_socket.
If a user clones a Rust socket, drops one of the clones and then recreates another socket, there is a good chance he will end up with two different Rust sockets using the same C socket, with funny effects. This would probably be quite difficult to diagnose.

The choice we have is between :

• a safe one (no change), with the downside of not allowing any sharing of sockets between tasks.
• a powerful but risky one (clone), with the downside of letting a user shoot himself in the foot.
• a middle ground through a pub unsafe fn to create a Rust socket from a C socket with a huge warning in the doc comments.

I'm not in favor of option 2 because I don't want to find myself in the situation mentioned above. And it looks like Rust is all about safety, but that's just my personal opinion here.
From the nanomsg documentation ... using a single socket from multiple threads in parallel is still discouraged ....
Maybe we could just wait for someone to come up with a real world use case and see then how to handle it.