Ctrl+C Cancel clean up issue

[mattrunchey]

I've been struggling to figure out how to clean up my close logic on a while loop using this client. Using Python 3.5.6, installed hbmqtt locally from a cloned master branch (pip install -e hbmqtt), but the same issue plagued me on 0.9.5.

In my code, I create a client connection with

    async def _get_client(self):
        """Connect the client to the configured broker"""
        con_str = "mqtt:{}:{}".format(self.url, self.port)
        await self._c.connect(con_str)

My main loop is essentially a while loop with a handler for KeyboardInterrupts and a task canceller, which eventually throws me to the stop() function designed to stop all of my running tasks before the event loop closes (using the cancel code from the last comment in #94 , but I have the same type of cancel logic elsewhere):

    async def stop(self):
        await self._c.disconnect()

        await asyncio.sleep(2)
        tasks = [task for task in asyncio.Task.all_tasks() if task is not
                 asyncio.tasks.Task.current_task()]
        list(map(lambda task: task.cancel(), tasks))
        results = await asyncio.gather(*tasks, return_exceptions=True)
        self._logger.debug('Finished cancelling tasks, result: {}'.format(results))

However, this isn't actually cancelling all of my tasks. After stop() completes, and my EventLoop closes down, I get multiple warnings that there were additional pending tasks that failed to cancel:

19-04-16 23:05:19.261 ERR asyncio Task was destroyed but it is pending!
task: <Task pending coro=<coro() running at /usr/local/Cellar/python35/3.5.6_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/coroutines.py:208> cb=[PluginManager.fire_event.<locals>.clean_fired_events() at /Users/mrunchey/gitrepos/hbmqtt/hbmqtt/plugins/manager.py:137]>

There are generally multiple of these (like, a minimum of 3), even if I don't do anything other than connect and disconnect (I commented out every subscription and publish to try and get a basic idle unconfigured client).

For some reason it doesn't seem like these tasks are part of the list of tasks when the cancel process is happening, but I'm not sure how to stop them from getting created before running my "cancel everything" routine (I was trying the sleep after disconnect, but that doesn't seem to be achieving my desired result).

Any ideas on what I'm doing wrong?

[mattrunchey]

For more info, the actual client seems to be correctly transitioning to a closed state through these messages:

19-04-16 23:32:10.144 DEB hbmqtt.mqtt.protocol.handler Stopping 0 puback waiters
19-04-16 23:32:10.144 DEB hbmqtt.mqtt.protocol.handler Stopping 0 pucomp waiters
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler Stopping 0 purec waiters
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler Stopping 0 purel waiters
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler waiting for tasks to be stopped
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler Task cancelled, reader loop ending
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler Broker closed connection
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler hbmqtt/o?iUew6]HjY\u=e8 Reader coro stopped
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler Stopping 0 puback waiters
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler Stopping 0 pucomp waiters
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler Stopping 0 purec waiters
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler Stopping 0 purel waiters
19-04-16 23:32:10.145 DEB hbmqtt.mqtt.protocol.handler waiting for tasks to be stopped
19-04-16 23:32:10.146 DEB hbmqtt.mqtt.protocol.handler closing writer

[mattrunchey]

Figured out why this was happening, not sure if it's a bug or expected behavior in HBMQTT:

My program uses an EventLoop, but it exists in a different thread than the one I've been instantiating my MQTTClient in. While I do pass this loop in to the MQTTClient, the thread context doesn't actually have a loop, so calling asyncio.get_event_loop() creates a loop for the current thread (now causing me to have two event loops). When I close my background loop, but don't clean up the tasks in the other loop, I get the obvious "task destroyed but pending".

The path that this happens through is in https://github.com/beerfactory/hbmqtt/blob/master/hbmqtt/client.py#L116, because the PluginManager (https://github.com/beerfactory/hbmqtt/blob/master/hbmqtt/plugins/manager.py#L43) isn't getting passed the loop that I provided to the Client.

In theory, shouldn't the PluginManager inherit the same loop that the Client created / was given?

[mattrunchey]

Just realized that there's an open PR to address this exact thing #126 ...

[smurfix]

Another solution would be to use #178.