MQTT client: limit outbox size (IDFGH-7527)

[kotyara12]

Good day!

It is necessary to add the ability to limit the size of the outbox for mqtt_client or block it altogether (disable it).

The problem is the following. On a "good" WiFi connection, everything works fine, no problems.

But not all devices have a good WiFi connection, some of my devices are at the Wifi limit, -90dBi and below. And sometimes neighbors clog the channel so tightly that packet loss reaches 100%. In this case, mqtt queues messages. And this is good. But not much. The fact is that rather large JSON packets go to MQTT, from 512 bytes and above, sometimes 2-3 kilobytes. I can't quickly determine if there is a connection to the server or not. And the messages are queued. And the memory runs out very, very quickly, the device either freezes or reboots.

It is necessary to somehow limit the size of the outgoing mailbox, after which the "old" messages will be overwritten by new ones. I haven't found such an option yet.
And besides, it is very important that messages in the same topic are not duplicated in the outbox, but are filled with new ones.

[euripedesrocha]

Hi @kotyara12 thanks for the suggestion. I'm looking into adding more control over the outbox, but you can already disable it via menuconfig providing a custom implementation. It's also possible to define a timeout for messages, using these two options you may have a temporary solution for your issue.

[kotyara12]

Hello @euripedesrocha.

I don't quite understand how to disable the send queue via menuconfig. I understand that in order to use a custom implementation, you need to write it, and I do not have enough knowledge in this matter.
But this is not required. In fact, the send queue is a great feature and avoids problems in some situations when the device generates several packets at once to send to mqtt and "does not have time" to send them all. But there are times when this functionality brings problems. For example, right now the power was turned off, the devices switched to battery operation, but unfortunately the provider's equipment is apparently de-energized.
I've gotten around so far by checking the amount of free heap before JSON is generated and sent. Devices began to work much more stable, failures due to heap exhaustion were no longer observed.
However, if such functionality is at the library level, it will be "much more correct", in my opinion.
Thank you.

PS: In general, I am completely delighted with the multitasking ESP-IDF, after Arduino.

[euripedesrocha]

Hi @kotyara12,
we agree that this a good feature to have in esp-mqtt client, and I'm working on it already.

I only pointed the custom outbox route because it's a way for you to not be blocked by our implementation time.

[kotyara12]

Hello @euripedesrocha.

Pay attention to this mechanism:

Let's say we send a message "{big-big-json}" to the topic "device/topic1"
But there is no connection with the broker (for any reason), it is put in the outbox queue. After some time (but before the message storage timeout expires), another message is added to the SAME topic "device/topic1", but with different data.
In order not to know the extra space in the outbox, you need to somehow replace the data in the queue (deleting the old ones) and update the addition time.

Thank you.

[matiasstack]

Hello, I also agree that this would be a great feature to have.
@euripedesrocha you mentioned the custom outbox option, which looks quite interesting. Could you point me to some existing implementation to use as "inspiration" for a custom one?
Thanks!

[euripedesrocha]

@matiasstack unfortunately we don't have an example, but I suggest that you take our implementation as inspiration. If you look into our documentation, what is needed is to provide your version of the implementation of the API.

[AxelLin]

@euripedesrocha
Looks like this feature is implemented in esp-mqtt.
However, the esp-idf still uses a very old version of esp-mqtt (5afa3d3).
As a result, people cannot test latest esp-mqtt code. Any chance to update esp-mqtt submodule in esp-idf?

[euripedesrocha]

@AxelLin, mqtt submodule is in process of update in the latest idf master and 5.1.

For 5.0, that you are referring to, there is still an ongoing discussion about which features will be backported.

[kotyara12]

(Top) → Component config → ESP-MQTT Configurations → Number of queued events

This is it? How this option works (there is nothing about it in the help yet)

[euripedesrocha]

@kotyara12 the outbox limitation is a runtime configuration. You can find the limit configuration in the mqtt client config structure. The availability will depend on which IDF version you are using and the version of esp_mqtt.

[kotyara12]

@euripedesrocha, many thanks!