Home Assistant Autodiscover MQTT messages do not have retain flag set

[makuip]

The Home Assistant (HA) integration using MQTT Autodiscover works as expected. But the device disappears from HA when HA is restartet. I know that this behavior is described in the documentation (https://mp-se.github.io/gravitymon//services.html#home-assistant, option 3). I still think that this is a bug because the root cause is that the Autodiscover MQTT messages "homeassistant/sensor/..." do not have the MQTT retain flag set.

As a feature enhancement one could send the Autodiscover MQTT messages not only during "push test" but also after each power on.

[mp-se]

I can add an option so that the MQTT retain flag can be set if wanted. There are pros and cons of having retain on.

If retain is used then the new feature would not really be needed and you can always set the sensors manual in the HA configuration if persistence is wanted.

[makuip]

Thanks. An option to set the MQTT retain flag would be great.

I still think that sending Autodiscover MQTT messages after each power on is useful. Many people run the MQTT broker on the same device as Home Assistant. A restart of the device will affect both and retained messages are not persistent in Mosquitto using the default configuration.

[mp-se]

Well, I cant do it at every power on, in gravity mode there you don't want to send the data since it will take too much time and trying to send to much data when in config mode can cause crashes on the 8266 due to low memory.

If you want it always persisted the better option is to define the sensors in Home Assistant and let gravitymon send the data. That is the most effective and safe method.

[mp-se]

I have had a deeper look on the code part for MQTT and I believe the docs are wrong. Autoregistration is no longer limited to push testing. This was not properly migrated to 2.x stream. The problem is that on an esp8266 it crashes since there is not enough memory for handling such a big payload which is not something that can be fixed. I will update the docs for that part.

Its probably a better approach to have some external script send the autoregistration data and let gravitymon send the values.

[mp-se]

The current limit on the esp8266 is around 1000 characters in total, more than that then it crashes due to lack of memory. In most cases the auto registration data is more than that so it would not help to do this during startup. I would recommend using an ESP32 instead that would solve the problem.

[makuip]

I'm considering upgrading my devices to ESP32.

Regarding the ESP8266 limit of 1000 characters: wouldn't it be possible to do the auto registration in multiple steps? The MQTT messages are static and only change with a different configuration. They could be stored completely formatted together with the config data.

[mp-se]

I'm considering upgrading my devices to ESP32.

Regarding the ESP8266 limit of 1000 characters: wouldn't it be possible to do the auto registration in multiple steps? The MQTT messages are static and only change with a different configuration. They could be stored completely formatted together with the config data.

Probably yes, but that would require a lot of refactoring and changing how the push code works. It might also require multiple connection to the server which will increase power consumption and reduce battery life.

I dont think it worth the effort when there are other options that will work and the esp8266 is not really well supported anymore.

[makuip]

Understood, I will upgrade my devices to ESP32. Right now my devices use an ESP8266 D1 mini. Which ESP32 is the best pin compatible replacement and most recommended? ESP32c3 mini or ESP32s3 mini?

[mp-se]

All the mini will work but some C3 boards has had issues with the antenna and poor wifi connection. Both of them are good candidates since they also have BLE. The s2 mini will also work, even the older ESP32 d1 mini even though this board is wider and can be hard to fit into the tube.

[mp-se]

You can try the build i the dev branch, that has the option to set the retain flag for messages and fixed the faulty link

[makuip]

I did a very brief test. When I do a test push test the messages do have a retain flag. However, all of them including the measurement values do have a retain flag. This is not desirable since old measurement values may be reported by the MQTT broker. I had also issues with the gravity mode. No measurements were sent. I still have to verify this.

[mp-se]

All messages will have the retain flag set, this is the reason for forcing a user to activate it. This integration is not specific for Home Assistant and I use that for many more projects and I dont want to put any Home Assistant logic into the framework code.

What I sense you are asking for is a specific home assistant integration which is possible to add but will require some work to implement. It would probably be easier to use the REST api rather than the MQTT integration for and also get around the current memory limitations with the esp8266.

I dont really see any need for having the retain flag since all the defined messages are sent during every push, So I could remove feature from the software all together.

[makuip]

Regarding the retain flag, I do agree with you that this is not needed if all defined messages are sent during every push. However, this is currently not the case for my esp8266. I will definitely upgrade to esp32. Having the retain flag only on a part of the messages is not so unusual. Integrations like zigbee2mqtt does it also the same way.
With the current beta firmware, I do have the problem that the push test is working. However, I do not see any MQTT messages send during normal operation. I don't know how to debug this.

[mp-se]

Well, it could be due to a crash. Its possible to see the serial debug data if you connect direct to the esp and remove the battery, when you tilt the device it should behave as in gravity mode.

One option that would not be to difficult to add would be to tag a message/topic as persistent. One way could be to add a special char before the topic name on those topics that should be persisted. For example start with a hash or exclamation mark.

[makuip]

gravmon.log
An option to tag a persistent message with a special character to get a retain flag would be great and a perfect solution.

On the other aspect that MQTT push is not working for me, I attached a log from the serial console. I looks like that the device crash.

[mp-se]

Yes, thats a crash, most likely from low stack space = to large mqtt payload that cannot be handled. If you get the payload to below 1000 then it might work, so delete one or more of the sensor values.

Not sure if i can refactor some code to handle it or this is in the arduino/dependant libraries.

[makuip]

I removed the Home Assistant MQTT auto-discovery messages from the config which shortens the message set drastically. In this case MQTT push is working as expected.
The complete message set in the template had a size of 1387 characters.

[mp-se]

I have done some refactoring to the push code to free up some stack space +1kb. This should be enough to handle the full payload on an esp8266. I will need to do some more testing to see that I haven't broken anything else...