Abstraction branch (making software easy to customize)

[MatthewColvin]

@CoretechR Did you plan for LVGL to be the long term solution I see you talking about a webserver on the esp to allow modification of the UI which sounds user friendly and easy and I was wondering if LVGL will be able to support this.

[CoretechR]

Hi, yeah, the UI would never run this fast without LVGL. So that would be one of the main components of the software that I want to keep.

[MatthewColvin]

Awesome as long as we are 100% set on LVGL I think we can build the Hardware Abstraction layer with that dependency in mind.

My pull request that allows UI to be shared it coming along but it will be good to talk about what our hardware abstraction interface needs to have.
https://github.com/CoretechR/OMOTE/pull/15/files#diff-65fb8b5ec34d4b9d0342eb875debd8b115428d7240dfbd02c6812723274657b2
Basically we need everything the remote is able to do with the hardware and the data to do it to be in this interface.
This design will make for a really clean and controllable solution for end users since they will basically just update the OmoteUI.cpp & .hpp.
Then if there are bugs or improvements to the way we do those things with hardware end users will just have to do a quick sync/merge
from the upstream.

[Dark1886]

Just adding some appreciation for integrating the LVGL more directly into platformio. I'm still learning how to use LVGL and have components about to drop on my doorstep, but removing steps for end users to create their own UI will be a huge benefit.

[MatthewColvin]

I really want to build one of these but unfortunately I don't think I am going to have the skills or equipment to be able to solder the components onto the PCB.

I can however make the software very easy to update and interact with so for now that is going to be my focus when I have time to work on it.

[Dark1886]

If you would like I may be able to send you a board once I confirm mine are working. I won't have a screen or battery to send with it but those are easy to source and install. My boards aren't schedule to arrive until ~AUG 1st though.

[MatthewColvin]

Oh that would be awesome!
I don't think GitHub has any private messages does it?

[Dark1886]

I don't think it does, but you can reach out to me on discord if you would like.

[MatthewColvin]

Figured I could high Jack this thread to talk about the abstraction of the code and allow people to weigh in on my current design.

#15

[CoretechR]

@MatthewColvin I still have some questions about your implementation:

• The Simulator now uses the OmoteUI.hpp from the Platformio folder. How does the include work when these are completely different directories?
• What would be the best workflow for customizing the interface (UI, images, IR codes, MQTT) right now? There are probably a few separate files that need to be edited.

I'm still wondering what the best way for easier customization would be. Maybe some sort of json config file that's loaded into the SPIFF?

[MatthewColvin]

Hey @CoretechR,

• The Visual studio solution config was modified to add the files that are needed.

• Currently the only way to interact with the hardware is through the HardwareInterface and so it will need to be flushed out to a point where it is a lot more capable.

  • I think we need to try to understand how much coding experience people may have when they attempt to customize this.

  • There are kind of 2 extremes we can go

    • No code - Users customize through config files and #defines only
      • main drawback here is that it will limit the users ability to customize it and will take a while get this system in place
      • we could have a static UI that we then give a custom config to drive data like IR codes and Mqtt commands.
      • we could create a config system that allows generation of UI but that could be a pretty difficult structure to build up depending on how customizable you want it.
    • Full code
      • User must implement their own LVGL code and if they want to add hardware features they must do it through the interface or break the sim.

My thought was that we have a hybrid of sorts where the maintainers focus on the HardwareInterface and then allow the end user to use that HardwareInterface inside their custom LVGL code (OmoteUI). I still have a bit of work to get that HardwareInterface into a place that would make good use of the hardware. But we may not want to assume that amount of coding knowledge?

• I am unsure how images are getting added to the project I assumed you used some kind of tool for those huge structures.
  • I recently did add a header file for those though to clean up the OmoteUI class.

After I flush out the HardwareInterface the steps to customize would look something like this

1. Fork the repo
2. Modify the OmoteUI.cpp and .hpp to your liking to get the UI to look how you want in the Layout_ui function
3. Create callbacks that hookup to the UI and use the hardwareInterface to do things like send IR and mqtt messages.
   a. There is a variable inside of OmoteUI called mHardware and that is how end users will interact with the hardware.
   b. Example mHardware->MQTTPublish("topic", payload);
4. Flash it or run the sim

Might be good for us to talk sometime so I can kinda get an idea of where you want this to go.

[CoretechR]

The Visual studio solution config was modified to add the files that are needed.

Ah, now I see that you've added Platformio as an extra include directory. Perfect!

After I flush out the HardwareInterface the steps to customize would look something like this
Fork the repo
Modify the OmoteUI.cpp and .hpp to your liking to get the UI to look how you want in the Layout_ui function
Create callbacks that hookup to the UI and use the hardwareInterface to do things like send IR and mqtt messages.
a. There is a variable inside of OmoteUI called mHardware and that is how end users will interact with the hardware.
b. Example mHardware->MQTTPublish("topic", payload);
Flash it or run the sim

That makes sense and should be doable for the average person with some moderate programming experience. At least with some help in terms of tutorials and examples. And the way you linked the simulator and Platformio makes it much quicker to test new configurations without the need for any hardware.

In the long term, this could be expanded to some kind of GUI configurator or yaml/json file. But my immediate goal is to make the software stable and serviceable. I think you're already on the right track with your Hardware Interface and I really appreciate your work on this. The only thing that I'm worried about, is that I'm not able to maintain this in the future. I'm not a software engineer and the way you built the OmoteUI and HardwareRev classes is something I can't follow entirely. This is surely the professional programming style, but is there a way to lower the complexity a little? At least to the level of the average Arduino library code?

[MatthewColvin]

@CoretechR yeah I for sure understand your worry on not being able to service the code. I have made massive changes to the way things work.

I think that from the customizers perspective it will be at that Arduino library level. Unfortunately, if we remove the hardwareInterface I think we are going to lose the simulator and HW simultaneous editing. Which I think really could help with development speed (especially newer people).

If you would be up for a meeting where I walk through some of the code and maybe we can brain storm some ideas on how to make it simpler. I also planned to make documents on how to add hardware features and expand this current ecosystem but I was just trying to flush out some basics first.

Also I want to help with this project. I don't know how much time I can commit but I do plan to continue work on it.😁

[CoretechR]

@MatthewColvin There was another pull request by bittnert today. This seems like a good approach that won't overwhelm people in terms of code complexity. At least it's something that I can still follow. Maybe there's some way to combine this with your work. If the GUI and display are separated, the simultaneous hardware and sim editing should still be possible.

[MatthewColvin]

Hey @CoretechR I really like that pull request and how it spreads out the hardware into simple modules but in order to keep the simulator similar to the hardware we need an interface for the UI to rely on as opposed to direct hardware dependencies inside the UI.

Let me chat on that PR I wanted to break up my interface into smaller ones based on individual hardware components eventually.

[MatthewColvin]

abstraction...MatthewColvin:OMOTE:RefineAndImplementHardwareInterface

Current work on that interface.

[GaryTill]

On the subject of full-code vs no-code GUI customisation, just an example the ESPHome-gui project are using YAML to define the layout and contents of an LVGL GUI as a bit of a middle ground, you describe your GUI components in YAML with simple blocks with basic customisation and tag them with whatever their function is, this requires something to interpret (in this case I think it's just some python) and piece together blocks of pre-defined LVGL code with limited customisability but it could work in this situation.

Limiting the GUI to a selection of pre-defined blocks makes it less configurable (unless people want to tinker with the raw LVGL) but it is much easier than building some kind of graphical GUI builder from the ground up with the LVGL simulator at its heart.

A GUI for this product would likely comprise of only a few fixed parts for most users, a settings page more or less hardcoded, then a small number of device pages with simple arrangements likely just 3x4 button grids, with blank buttons being hidden, just that one layout option allows for numpads, D-pads, transport controls, input selections etc, all with one predefined layout.

That keeps the interpreter code fairly simple. just that on it's own would cover 90% of users needs before you get into home automation control with sliders and toggles, or display readouts for sensors etc.

[bittnert]

I would like to add my idea about a no-code GUI customization. In my opinion, for the remote there are two groups of configuration which are interesting to be done. One would be a "device" configuration which more or less describes how and what a device (like TV, blue ray player, amazon fire tv stick etc) is capable of doing and how to control it in priciple. With this information, the remote could already replace every single remote control.

The second group would be "actions" (or how ever you want to call it). An action would define a series of commands to be send to different (configured) devices to enable a sort of sequencing. For example, I would press a button and the remote would turn on the tv, the amazon fire tv stick, the sound bar etc.

I would like to describe the configuration structure bottom up (how the firmware would handle them). I think it would be possible to configure this then in a web interface and then store it for the firmware in a usable way without a complex GUI customization interface.

I will start with the configuration structure for devices as actions would rely on that.

The device configuration would consist of an array or a structure with one entry per configured device. Some of the configuration is fixed in size, others can have different sizes which would be referenced by a pointer.

typedef struct {
/* Type describes what type of device it is. For example, this could be 
a specific http or MQTT protocol or IR Sony encoding or IR RC-5. So 
the remote it self does not support TV XY but IR Sony which is used 
by the specific TV and which codes to send when will be specified 
in this structure*/
    uint32_t type;
/* The name is a user specified string which identifies the device.
This name will only be used for the UI and will not have additional 
functionality*/
    const char* name;
/* This points to type specific configuration. For example, for http 
or MQTT devices, this could include the IP address or similar information*/
    void *type_specific_config;
/* Physical button codes. When a physical button is pressed, the firmware would
take the code specified here and pass it on to the function specific for the specified 
type. For example, the play button can have the IR code for the play functionality 
stored in the button codes and the code would then be send out via the IR transmitter*/
    button_codes buttons;
/* Flag to indicate if the numpad can be used by this devices */
    bool use_numpad;
/* Array containing the codes used for the numpads*/
    uint32_t numpad_codes[10];
/*Number of ui buttons. This specifies how many elements the array below has.*/
   uint32_t no_ui_buttons;
/* List of the UI buttons which should be provided in the ui interface*/
   UI_Buttons*  ui_buttons;
} Device_Config;

typedef struct {
    uint32_t play;
    uint32_t reverse;
    uint32_t stop;
    uint32_t forward;
    uint32_t volUp;
    uint32_t volDown;
    uint32_t up;
    uint32_t down;
    uint32_t left;
    uint32_t right;
    uint32_t channel_down;
    uint32_t channel_up;
    uint32_t red;
    uint32_t green;
    uint32_t yellow;
    uint32_t blue;
} button_codes;

typedef struct {
    uint32_t code;
    const char* name; 
} UI_Buttons;


/**/
typedef struct {
/* Amount of devices configured*/
    uint32_t count;
/* Pointer to the array of configuration structures*/
    Device_Config* config;
} Device_Config_Wrapper;

/*The Firmware would then use an object of Device_Config_Wrapper to 
talk to the supported devices*/
Device_Config_Wrapper device_config;

To configure actions, the configuration structure could have the following structure in its easiest form

typedef struct {
/*Index of the devcie in the device_config*/
   uint32_t device;
/* Code to be send*/
   uint32_t code;
/*Delay after the code is send*/
   uint32_t delay;
} Single_Action;

typedef struct {
/* Name of the action to be shown to the user in UI*/
    const char* name;
/* Number of single actions*/
    uint32_t no_single_actions;
/*Array of single actions which can be looped through sending the different codes*/
    Single_Action* actions;
} Action_Config;

So the Firmware could have an array of Action_Config to support several different actions.

In this configuration, the idea is, that the remote "only" needs to support a specific protocol and the configuration then has to specify how to use it.

This would be my suggestion how this could be implemented. Please let me know what you think about this approach.

BR

[MatthewColvin]

Hay @bittnert I like the idea of this for a fixed GUI layout the only thing that gets me a little is the void* to the type specific data.

I guess my question about the no code approach is do we want to support UI creation through config or just have a fixed config we pipe data?

[bittnert]

Hey @MatthewColvin, I think a general configuration for a UI would be a very tall task.

So I think a more or less fix UI and the content is then configured is more feasible and will be sufficient

I think a configurable UI can be added later if needed.

About the void pointer. I agree with you that this is not ideal. However, the exact structure depends on the type. Would you have a better solution how this could be implemented?

[MatthewColvin]

Oh I agree it would be hard to do layout gen but I was wondering if maybe we could kinda do it by making template pages then in config they could give the page type and the data needed to drive that page.

Usually with void pointers I think about trying to use templates to squash them but I need to think about it way more lol

[CoretechR]

@bittnert I love the idea of having fixed templates. This would make it much easier to customize the remote without having to fully understand and edit the source code. And these templates could always be expanded to support more devices.

[KlausMu]

Hi @MatthewColvin ,

I tried to get your branch running, especially the simulator. I'm using Windows.

Is this the branch to use? https://github.com/MatthewColvin/OMOTE/tree/halloween

I installed
https://www.msys2.org/
(didn't use this before)
added the paths to PATH as described here and then did

pacman -S mingw64/mingw-w64-x86_64-SDL2

This put SDL2 into

c:\msys64\mingw64\include\SDL2

In plaformio.ini I activated env x64_sim

I had to change

-D SDL_INCLUDE_PATH="\"SDL2/SDL.h\""

to

-D SDL_INCLUDE_PATH="\"c:\msys64\mingw64\include\SDL2\SDL.h\""

Now the compiler doesn't find everything Arduino related (Arduino.h, esp_now.h, ...)
What am I doing wrong?

Thanks!

[MatthewColvin]

@KlausMu I think it is odd you had to change the include path that should not be necessary.
The GitHub workflow does install a few more dependencies so maybe grab those too. If those files are on path I think it should be able to find the include.

https://github.com/CoretechR/OMOTE/blob/abstraction/.github%2Fworkflows%2Fbuild-simulator-windows.yml

Could you maybe paste your specific error or attach a lot of your console?

[KlausMu]

In file included from OmoteUI\UIs\BasicRefactored\page\Halloween.cpp:1:
OmoteUI\UIs\BasicRefactored\page\Halloween.hpp:5:10: fatal error: esp_now.h: No such file or directory

---

• Looking for esp_now.h dependency? Check our library registry!
• CLI > platformio lib search "header:esp_now.h"
• Web > https://registry.platformio.org/search?q=header:esp_now.h

---

5 | #include <esp_now.h>
  |          ^~~~~~~~~~~

compilation terminated.

[KlausMu]

Ok, solution was not to take branch "https://github.com/MatthewColvin/OMOTE/tree/halloween", but branch "https://github.com/MatthewColvin/OMOTE/tree/UIRefactor"

[MatthewColvin]

Hey @CoretechR, With the decision to continue forward with @KlausMu branch as the main line would you like to keep the abstraction branch for people who may want a OO approach to GUI customization or just go ahead and drop it?

I think @KlausMu is open to ingesting the HAL into his code which will allow SIM to run the same LVGL GUI code but that will not be a clean merge by any means lol. Probably better off just working it in manually.

[KlausMu]

Correct, I will try to work it in manually.

[CoretechR]

We should probably keep the abstraction branch up in its current stage. That way people can find it easily.

[MatthewColvin]

Hey @KlausMu and @CoretechR I saw a comment on another ticket with the abstraction branch and some confusion on which branch.
If you guys want to archive it or rename it to "OOmote C++ for Omote". Then I can adjust the read me to let people know what is happening with it.
Either works for me but I don't want it to be confusing for anyone.

[KlausMu]

I like the idea of renaming the branch. I think the most important thing about the branch is that it is heavily using classes. A lot of things can be abstract. Non OO code can be abstract. And OO code can be non abstract.

So I think the name "abstraction" is not precise enough.
"OOmote C++ for Omote" would be fine, or simply "OMOTE with C++ classes".

And then in the README a comment about what this branch is for and what the difference to the main branch is

• consequently using C++ classes
• using a HAL abstraction layer for having the simulator running
• in terms of features not the same as the main branch (no memory optimization, no scenes, simplified key handling, no IR receiver...)
• but maybe some other features not in the main branch? Are there any?

[KlausMu]

@MatthewColvin
I would rename the branch to "OOmote C++ for Omote", as you suggested.
@CoretechR agreed to it.
Do you want to adjust the README afterwards, or shall I do it?
I would add a short explanation on top so that everyone knows what this branch is about.

[MatthewColvin]

If you don't mind adjusting it that would be great, I do have plans to come back and continue to contribute to that version of the code but I am just really bogged down at work at the moment.

[KlausMu]

Branch was renamed to "OOmote C++ for Omote"

Please see if you like the updated README:
https://github.com/CoretechR/OMOTE/tree/OOmote-C%2B%2B-for-Omote

For renaming, I followed this guide:
https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/renaming-a-branch

If you already have a local clone of the repository, you might want to do some or all of the following commands (some of them might not work, depending on how your local clone looks like):

git branch -m abstraction OOmote-C++-for-Omote
git fetch origin
git branch -u remotes/origin/OOmote-C++-for-Omote OOmote-C++-for-Omote
git remote set-head origin -a

git remote prune origin