Generators in Actor method

[callmephilip]

Hello,

Is this a valid way of interacting with an agent producing a generator or am I missing smth? This does seem to work "on my machine":

import pykka
from typing import Iterable

class PlainActor(pykka.ThreadingActor):
    def __init__(self):
        super().__init__()
        self.stored_messages = []

    def say_smth(self) -> Iterable[str]:
        for i in range(10):
            yield f"Hello {i}"

actor = PlainActor.start().proxy()
for m in actor.say_smth().get():
    print(m)

actor.stop().get()

[jodal]

If one wanted to implement the actor pattern in the safest way possible, the actor would serialize its messages to e.g. the pickle format or JSON before sending it. For performance reasons, Pykka does not do that, and leaves it to use as the user to only return objects that can safely be shared between threads.

Your example works because you simply return a generator object, and the caller can then iterate of the generator. On the surface, this seems to work. I think the big question is, where does the generator work happen? In the actor, or on the main thread?

If we change the example briefly:

import threading
from typing import Iterable

import pykka

class PlainActor(pykka.ThreadingActor):
    def __init__(self):
        super().__init__()
        self.stored_messages = []

    def say_smth(self) -> Iterable[str]:
        for i in range(10):
            yield f"Hello {i} from {threading.current_thread()}"

actor = PlainActor.start().proxy()
for m in actor.say_smth().get():
    print(m)

actor.stop().get()

And then run it we get:

Hello 0 from <_MainThread(MainThread, started 138697826586752)>
Hello 1 from <_MainThread(MainThread, started 138697826586752)>
Hello 2 from <_MainThread(MainThread, started 138697826586752)>
Hello 3 from <_MainThread(MainThread, started 138697826586752)>
Hello 4 from <_MainThread(MainThread, started 138697826586752)>
Hello 5 from <_MainThread(MainThread, started 138697826586752)>
Hello 6 from <_MainThread(MainThread, started 138697826586752)>
Hello 7 from <_MainThread(MainThread, started 138697826586752)>
Hello 8 from <_MainThread(MainThread, started 138697826586752)>
Hello 9 from <_MainThread(MainThread, started 138697826586752)>

In other words, the work does not happen in the actor, but on the main thread. If you in the actor inbetween each yield call modified state, that would now happen from the main thread. If the actor's state used data structures that was not thread-safe you could potentially run into issues.

[callmephilip]

@jodal thank you very much for this explanation - something was telling me i will end up picking up some oopsie daisies lol.

i guess, what i am wondering now if there is way to make generators run inside actor's thread. this being said, it does make sense that what is essentially a "callback" ends up running on the main thread. but it does mess things up big time obviously