Rework backend add MQTT and WebSocket support

[rjwats]

• Update back end to add MQTT and WebSocket support
• Update demo project to demonstrate MQTT and WebSockets

[saniyo]

Is there a common way how to stream data via WS, as you did in your old led project?
Sometimes it's necessary to stream and receive data from WS and bypass the settings module. it's just thoughts aloud.

[rjwats]

My old project wasn't so much "streaming" rather than reacting to changes to the "state" from other sources. For example a change to the color of the LED strip made via MQTT was being reflected immediately in the UI via WebSockets or a change to a setting via HTTP was being propagated to both the MQTT broker and WebSocket clients. I think there might be some confusion in the naming in the project. The term "settings" should perhaps be "state" to make it clear that the features can be used for more than just configuration. If you have a requirement outside the scope of observing and reacting to state changes (for example one of my previous projects for example continuiously "streamed" audio frequency information to the client) there's nothing stopping you creating your own WebSocket connection and implementing your own functionality. The framework is trying to facilitate the most common use cases which is reacting to state change in a configurable manner. I do think a naming change might help make this more clear.

…

On Tue, Apr 7, 2020 at 9:45 PM saniyo ***@***.***> wrote: Is there a common way how to stream data through WS, as you did in your old led project? Sometimes it's necessary to stream and receive data from WS and bypass the settings module. it's just thoughts aloud. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKE4VFBKVN7UDNI5IHA2M3RLOGHZANCNFSM4MDNABLA> .

[rjwats]

I still have to implement security on the WS connection (probably using the "JWT as query parameter" approach) and improve the documentation.

Any input at this stage is really welcomed!

[saniyo]

I still have to implement security on the WS connection (probably using the "JWT as query parameter" approach) and improve the documentation.

Any input at this stage is really welcomed!

Having added the security, please leave an ability to switch off the security feature for WS packets debug. Thanks.

[philip-searle]

Looks good to me; just a few minor typos that caught my eye. Splitting the serialization out from the service/endpoint code is a good change; significantly simplifies them and allows the serializer to be reused across REST/WS/MQTT handlers.

[rjwats]

Working on a weather station project this weekend I realized that I missed a significant use-case.

When reading from sensors you may have Endpoints/WebSockets/Brokers which will only ever source data - GET only Endpoint/Transmit only WebSocket/Publish only MQTT. In this situation you never externally update the sensor data, only read updates from it. The framework does not facilitate this nicely.

In my weather station branch I played around with improving this by separating out the GET from the POST and the Pub from the Sub (introducing more class types). I think this might be a good change and provides another decoupling improvement.

Going to continue looking at this after work today.

[rjwats]

Sounds reasonable. I had a similar requirement in a LED lighting project I had done in the past.

…

On Thu, May 7, 2020 at 10:58 PM kasedy ***@***.***> wrote: ***@***.**** requested changes on this pull request. ------------------------------ In lib/framework/SettingsService.h <#102 (comment)>: > template <class T> -class SettingsService : public SettingsPersistence { +class SettingsService { public: The struct T that holds settings could be quite complicated. We need a constructor that passes arguments from SettingsService to "T _settings". In my project for example it is a list of effects that current device supports. ⬇️ Suggested change - public: + public: + template<typename... Args> + SettingsService::SettingsService(Args&&... args) : _settings(std::forward<Args>(args)...) {} — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#102 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKE4VFJEXLNWD5HKGUG5I3RQMVHTANCNFSM4MDNABLA> .

[rjwats]

@kasedy Thanks for the great feedback - I've made your suggested changes in this branch.

[rjwats]

A naming question, for those with opinions on the subject:

Do the names "SettingsService", "SettingsEndpoint", "SettingsXxx" still make sense?

Early on in the framework, "settings" were just about the only thing I was dealing with in the SettingsService class. Now, "SettingsService" has become more of an event-driven conduit for data (might be: settings, sensor data, state etc...) I'm not sure the names really make sense now.

I thought "DataBus" (sounds a bit "enterprisey") or "DataService"? I'm quite familiar with the current names now but I wonder if there are better ones. Seeing as this PR introduces such a lot of changes this might be the right time to do that too if we can find better names.

[rjwats]

LGTM

I eager to see WebSocketTx / WebSocketRx changes. It would be nice to migrate MQTT / WiFi / AP status pages to sockets. Right now it annoying to refresh them constantly.

Regarding the status endpoints: I hadn't thought about that really, but yes, we could push status changes to the UI. Currently most of the status endpoints are read on request rather than managed with events, though this could be changed.

I've added an issue (#104) to the backlog because I think this might be a nice change, probably after this PR goes back to master.

[kasedy]

@rjwats, can you give me contributor access? I would like to participate in the development. It would reduce turn around time for small fixes and improvements. If you have questions please find my email in profile.

[kasedy]

Proposal: move base classes to separate directory lib/framework/base
SettingsDeserializer.h
SettingsEndpoint.h
SettingsPersistence.h
SettingsSerializer.h
SettingsService.h
SettingsSocket.h

[rjwats]

Proposal: move base classes to separate directory lib/framework/base
SettingsDeserializer.h
SettingsEndpoint.h
SettingsPersistence.h
SettingsSerializer.h
SettingsService.h
SettingsSocket.h

Yes, I had that in mind too. Have split "core" and "util" from "framework".

That, and some of renames discussed above (PR back to this branch) for separate review:

#105

[rjwats]

Think that's all the renames done for now... I'll get the README updated tomorrow then this PR can go back to master.

Shout if you spot anything super critical. After this it would be good to do a round of tidyups, then get on with the "feature service" to allow features to be disabled at build time.

[rjwats]

@proddy @kasedy I'm thinking PROGMEM_WWW should be the default now. I rarely deploy the interface separately any more. It would mean also enabling "min_spiffs.csv" for env:node32s by default.

What do you think?

[proddy]

yup, do it. progmem all the way! BTW I don't think you ever checked-in your min_spiffs.csv. Also note SPIFFS will be depreciated on the ESP8266 soon and replaced with LittleFS. What I did on one of my projects is replace all the SPIFFS.* calls with a nasty define, like:

#if defined(ESP8266)
#include <FS.h>
#include <LittleFS.h>
#define EMSESP_FS LittleFS
#elif defined(ESP32)
#include <SPIFFS.h>
#define EMSESP_FS SPIFFS
#endif

[rjwats]

Thanks @proddy, agreed - PROGMEM is the way!

min_spiffs.csv is provided by the arduino-esp32 core thankfully so you can just reference it. I had however, carefully calculated a suitable partition table before discovering it though, wasted effort :(

Thx for the info on the deprecation of SPIFFS. The internals rely on the FS abstraction so, like you say, it's easy to swap out.