How to call set_result() of future after return py::object to Python?

[MarshalX]

I have class with method that return Python future. Before the return I create observer and call some C++ function that get my observer and call callback on success.

With other words I pass py::object (future) to my observer, run some C++ method that takes a time to complete, and in OnSuccess of my observer I try to call set_result() of passed future. So when I call set_result() in my observer I got SIGABRT.

This works fine and I get it from test files of pybind11 (ref)

static py::object async_test() {
  py::object loop = py::module_::import("asyncio.events").attr("get_event_loop")();
  py::object f = loop.attr("create_future")();

  f.attr("set_result")(5);

  return f;
}

PYBIND11_MODULE(test, m) {
  m.def("async_test", &async_test);
}

This is what I try to do:

py::object Connection::Create() {
    pybind11::object loop = pybind11::module_::import("asyncio.events").attr("get_event_loop")();
    pybind11::object f = loop.attr("create_future")();

    auto observer = new Observer(this, &f);

    _connection->create(observer);

// This it will work fine!
//    f.attr("set_result")(10);

    return f;
  }

My observer:

// .h
class Observer {
public:
    explicit Observer(pybind11::object* future);

    void OnSuccess();
private:
    pybind11::object* _future;
}


// .cpp

Observer::Observer(pybind11::object* future) {
  _future = future;
}

void Observer::OnSuccess() {
   _future->attr("set_result")(10); // throw SIGABRT
}

My abort is terminate_handler unexpectedly returned by libc++abi.

So is it possible to do? How to bypass SIGABRT? I also try to play with return value policy. Nothing new.

UPD. I get more info about the exception. Its

pybind11/include/pybind11/cast.h

Line 1385 in 39fbc79

pybind11_fail("pybind11::object_api<>::operator() PyGILState_Check() failure.");

[MarshalX]

Before working with python object we need to asquire GIL, because it was released after finish of my method that returned future to Python.

My updated OnSuccess:

void Observer::OnSuccess() {
  pybind11::gil_scoped_acquire acquire;
   _future->attr("set_result")(10);
  pybind11::gil_scoped_release release;
}

Issue 2. Now I got Exception: EXC_BAD_ACCESS (code=1, address=0x8) during the access to set_result.
Solution: we need make a copy of object with increasing of ref count. It's possible using py::object constructor. For example return py::object{f};

Issue 3. I set result for some strange future that is not the same that I returned to Python (but should be?). In this case my future in python in pending status always almost after setting the result...

upd. Using a debugger I double check that addresses of m_ptr of handle of my future in observer is the same with return result to python. I have no idea why pybind11 not notice python that there is result in the future

upd2. I figure out that await my_method() will block and will not noticed about finished status of future! But using the endless loop the status of future is updated! See the code below:

await statement:

f = my_method()
print(f) # <Future pending>
r = await f
print(r) # the code does't reach this line!

endless loop:

f = my_method()
while True:
        print(f) # for the first iteration will be <Future pending>, for the second one and the rest <Future finished result=10>!
        await asyncio.sleep(0.1)

Why await statement doesn't work properly? Why it doesn't catch the changed state of future and doesn't return the result?

upd3. Future is awaitable object and should works fine with await statement but it doesn't. I checked add_done_callback and it works fine. So I wrote a simple wrapper for future.... just a hack that works too slow.

async def wrap(future):
        on_done_event = asyncio.Event()

        def _done_callback(_):
            on_done_event.set()

        future.add_done_callback(_done_callback)
        await asyncio.wait_for(on_done_event.wait(), timeout=5)

        return future.result()

print(await wrap(my_method()))

My last question is what the hell with await statement and returned future from pybind11 and how I can get clean solution instead of hacks?)

UPD. Use set_result directly in another thread is bad idea. More info here: #3597 (comment)

[MarshalX]

I created the issue with project for reproduce: #3597