Using HTML/CSS, QT, or GTK would probably be too much for a game.

I suggest something like tgui or SFGUI as a backing library. We could reflect their API in Lua, and serialize and send to the client using JSON.

i agree with that, those aren't exactly lightweight GUI's
however wxwidgets might be a possibility as it is automatically cross-platform.
but idk how much it is supported

i have a partially working XML format in a branch: https://gitlab.com/project-demeter/core/tree/form_xml
The only missing patch is catching events from XML to core, other parts are managed and just need more and more components, it's XML with HTML-like attributes)
It's using libxml2 which is portable and very performant, used by many softwares

what about this: http://www.clutter-project.org/
it seems to be very lightweight and runs well on Android

nuklear looks good, supports theming and it's very lightweight compared to Qt/GTK.

@weqqr that looks like exactly what openarena uses, and its GUIs look pretty nice
but does it support android?

@ThomasMonroe314 nuklear itself only receives user input and generates a list of shapes to draw, so in theory it can run on everything from a supercomputer to an electric kettle. You need to write a frontend (like this) to make it work with Irrlicht though.

Wanting a CSS themeable, extendable (by CSM) GUI using a completely new GUI toolkit is aiming way too high IMO.
It pretty much guarantees that it won't ever be fully implemented and we'll be stuck on formspecs for a longer time.

Also having something like minetest.register_window removes quite a lot of dynamicness from the gui system.

(The rest of this post is just throwing my ideas in here.)

Instead I would fix "just" these issues:

• coordinate system
• proper layouting, meaning: containers with children
• use nested tables to represent these (not exposed to Lua, should have object-oriented API)
• proper event system: wide variety of events for each element, all optional

This retains the following from formspecs:

• Toolkit-independent
• dynamic ("made up on the spot", no pre-registered layouts)
• events are produced in certain conditions, these can be individually listened for
• uses Irrlicht GUI (probably)

There is likely no existing solution for this, but that's fine.
Since this is quite hard to put into words, some example code:

local window_name = minetest.get_current_modname() .. ":respawn"
local pad = 50

local function show_respawn_dialog(playername)
	local root = gui.Window:new(window_name, 100 + 2*pad, 200 + 2*pad)
	root:listenForEvent("close")
	
	local stack = gui.Stack:new()
	stack.padding = {pad, pad, pad, pad}
	root:add(stack)

	local button1 = gui.Button:new("Respawn", 100, 100)
	button1.id = "b_respawn" -- must be unique
	button1:listenForEvent("click")
	stack:add(button1)

	if creative.is_enabled_for(playername) then
		local button2 = gui.Button:new("Respawn at same position", 100, 100)
		button2.id = "b_respawn2"
		button2:listenForEvent("click")
		stack:add(button2)
	end
	
	minetest.show_gui_dialog(playername, root)
end

local function respawn_normally(player, state)
	assert(state["b_respawn"].pressed == true) -- just an example for state
end

minetest.register_gui_event(window_name, "b_respawn/click", respawn_normally)
minetest.register_gui_event(window_name, "close", respawn_normally)

minetest.register_gui_event(window_name, "b_respawn2/click", function(player, state)
	-- do something else
	return true
end)

internally/when sent via network it would look like this (no modder would ever see this though):

{
	"type": "window",
	"id": "testmod:respawn",
	"events": {"exit": true},
	"children": [
		{"type": "stack", "children": [
			{"type": "button", "id": "b_respawn", "events": {"click": true}},
			{"type": "button", "id": "b_respawn2", "events": {"click": true}}
		]}
	]
}

To achieve backward compatibility you would then need to:

• Convert the formspec string to tables (duh)
• Translate the stupid coordinate system to the new one during conversion
• Give each element a default set of listened events (e.g. buttons always respond to clicks)
• Convert the new events to the old system and call minetest.register_on_player_receive_fields callbacks

nuklear has lua bindings, a thing that i haven't seen anywhere else, and something that is straightforward, simple to use and should be simple to implement

nuklear has lua bindings, a thing that i haven't seen anywhere else, and something that is straightforward, simple to use and should be simple to implement

Doesn't help when we need SSM to be able to make windows/dialogs

@sfan5, all of that sounds good to me. I actually prefer it to mine, so updated first post

It's fine to leave stuff out for an initial implementation, but it needs to be thought about to allow future additions. Yours does however, you could add theming without breaking

pardon my ignorance, but what is SSM?

pardon my ignorance, but what is SSM?

Sorry, server-side mods or server-side modding.

ah ok thanks, but why is that a problem? cant it be used for that?

ah ok thanks, but why is that a problem? cant it be used for that?

Their API will likely be directly manipulating elements, where as we need to send commands over the network to the client.

I suggest keeping both the new API and formspecs in parallel whilst developing, then making a server-side formspec to new GUI converter when stable enough

hmm true, but as weqqr pointed out, we would need to write a frontend, so couldnt we add some that functionality to the front-end?

I really like sfan5's idea, but not the API, so here's my concept.

local function say_hello(window_state, player)
	print("Hello, " .. window_state.name.text)
end

local function show_window(player_name)
	local window = minetest.window("mymod:window", function(w) w
		:size(200, 200)
		:field("name", function(f) f
			:default_text("someone")
			:position(50, 50)
		end)
		:container("my_container", function(c) c
			:position(60, 60)
			:padding(10, 10, 10, 10)
			:button("hello", function(b) b
				:text("Say hello!")
				:position(50, 50)
				:dimensions(100, 100)
				-- Callbacks can be set in-place
				:on_press(say_hello)
			end)
		end)
	end)

	minetest.show_window(player_name, window)
end

-- Complex callbacks can be handled outside of the window definition
minetest.register_on_gui_event("mymod:window/my_container/hello", "press", function(window_state, player)
	print("Your name is " .. player:get_player_name())
end)

minetest.register_on_gui_event("mymod:window", "close", function(window_state, player)
	print("Window closed")
end)

I feel 'themeable' is low priority and should not count against a good, simple, practical solution.
MTs neutral grey boxes are visually fine and no problem.

MTs neutral grey boxes are visually fine and no problem.

MT's current GUI is ugly. The new GUI can be tweaked to look better without breaking things, and margins will be better as you would be able to specify exact margins. Theming is relatively low priority, and should not be in a first implementation - but it should be considered to make sure any first implementation could support it in the future. I think any good consistent and DRY solution would be able to, anyway

Hmm my statement there was too strong, even i like to have colourable boxes instead of only grey / black. Being able to design the background graphic is a good feature to have.

@weqqr That jQuery-like syntax is awful IMO.
@sfan5 What’s the problem of needing to register windows before showing them? That would be handy IMO, as long as the registration may be done at any time and not only on server load: you could setup the whole huge multi-tab window layout, all the callbacks, etc., and only set some content right before showing it.
Also, that way nodes could store window name in the meta instead of its full specification, thus easing updates.

My thoughts (Some of these have been mentioned already in the thread):

• There should only be a CSM GUI API (maybe with a compatibility layer for old-style formspecs). This would need to wait for servers being able to send client mods.
• The API should have some way of drawing every frame so that mods can animate GUI components, making it possible for mods to create new interactive GUI elements, like selection wheels, without hardcoding engine code to handle them.
• Layout should be automatic, i.e. you don't need to manually specify coordinates for everything if you don't want to.
• The same API could be used for client-mod provided HUD elements.
• (Optional) I like the idea of imgui's API. I've never actually used it though so it might not be as good as I imagine.

MTs neutral grey boxes are visually fine and no problem.

Now your really pissing me off

Hmm my statement there was too strong,

Ooook then nevermind

But all jokes aside those "neutral grey boxes" look awful and they have to go they look unprofessional, they are ugly, they make minetest look like test software (the word test in minetest doesn't help with that) and they are NOT CUSTOMIZABLE WITH TEXTUREPACKS!!!!!!

Using HTML/CSS, QT, or GTK would probably be too much for a game.

I'm in with QT and GTK but imagine how awesome it would be having HTML-based dialogs and menus and stuff. Of course not full HTML5 and CSS3, but a base set of HTML syntax (img, input, section, div, and some more) plus some Minetest-specific additions (for the inventories for example) would be friggin awesome!

@dsohler IMO some XML-based language would be way better: it is easier to parse and more flexible. E.g. mods could be able to define their own elements (using XML namespaces... oh yeah, it uses the same namespace:element naming convention as MT does...)

OK, so how much of Unicode is it reasonable to expect for the new GUI? Unless we use some existing GUI (that fully supports Unicode), there's no way we'll get all of it. Worst case, the GUI only supports the bare minimum of understanding Unicode codepoints (i.e. pretend that every index in a UTF-32 string is its own left-to-right character, ignoring combining characters or grapheme clusters and whatever, only little better than Irrlicht) and nothing else. Best case scenario, the entire world starts using exclusively English so we can use ASCII instead. (Actually, the real worst case is that EBCDIC makes a comeback :P)

Of course, even with an advanced Unicode library, it's not feasible to implement everything because a) who has the time to do all of it? and b) our font drawing classes and the fonts we have with Minetest are simply incapable of doing advanced Unicode at all. Unless we embed GTK+, Qt, wxWidgets, HTML, or some other fat thing, we aren't going to get all of it.

OK, so how much of Unicode is it reasonable to expect for the new GUI?

It should be as good as we have now - it should be able to render most characters but doesn't need to deal with right to left text. But support for that would be good eventually

I think we should first abstract the GUI Engine API, such as IMGUI, Nuklear.
Then you can create the GUI parsers to do what you wanna.

local formspec = "formspec_version[json] {button: {...}}"
local formspec = "formspec_version[yaml] button: ..."
local formspec = "formspec_version[html] <button .../>"

Written by nephele. Added annotations and made a few spelling corrections

Closely related to #6805.

It doesn't seem like anyone brought up the idea of a simple low level drawing api? I honestly want something like love2d's API
Minetest can provide a canonical "like formspecs" lib
mods or games can ship their own [drawing libraries] without having to wait on the engine to provide a new styling support or whatever. Some stuff might need engine support still, say, new drawing ops, drawing gradients and the like, but most really should be workable to make it work nicely, as in:

1. a set of API to query stuff (screen size, dpi if known, device type if known, colorspace)
2. a simple set of API to draw "draw line from {x|y} to {x|y}" "Set line width to 2" "set pencil color to 1,0,1"
3. a simple set of API to get events from input devices [such as: ] clicked mouse at X, mouse move events etc
   mostly this [the I/O part] should all be clientside

I mean, it's not a bad idea to have a drawing API. I tried a client-side one: #10801. The biggest difficulty, of course, is not plain drawing: we can have a simple server-side drawing API, even one that supports the screen size with, say, minetest.register_on_window_resize(function(playername)). That's not difficult. Heck, it could even support animation similar to how SVG does it.

The difficulty lies in when you need communication, especially 1) font dimensions and 2) events.

1. It's impossible to predict the width of fonts without actually doing the calculations on the font itself, and the width is crucial for word-wrapping. Only the client has the font, so it's a round trip every single time you need to query font dimensions. Essentially, you'd have to give the server full authority over fonts where the client has no say in the matter, add Freetype to the server code, and have it do all the rendering calculations on the server. Slow, and bad for any semblance of accessibility.

2. Events are an even harder problem, since it's a round trip every event: client sends event, server receives and processes and sends new drawing instructions, repeat. Without SSCSM, it's impossible to make this work well on laggy servers or slow internet connections, not even with any hacks like the above mentioned for fonts. And we all know how soon SSCSM is coming out.

Frankly, I love low-level APIs. I'd provide a low-level API if I could (regardless of whether there would be a high-level GUI API as well), but I don't think it's reasonable without SSCSM.

Yup, so basically my idea is to use RmlUi, it's quite lightweight and its markup is based on HTML and CSS (so it will be familiar to a lot).

It even seems to have premade Lua bindings

It's pretty insane for us to write our own GUI library and API from scratch, fully generic GUI frameworks are some of the most complicated software out there. Layouting, text rendering, animation, accessibility, all that flexibility.

I think the only way that writing our own solution would be viable is if we exposed an input and drawing API to the client-scripting API and allowed modders to implement their own GUI elements/frameworks as needed - this is probably a good API to have in any case, but requiring modders to make their own GUI libraries is not a great situation and will lead to a lot of jank. It also depends on SSCSM which is another thing promised but doesn't seem to be coming

Yup, so basically my idea is to use RmlUi, it's quite lightweight and its markup is based on HTML and CSS (so it will be familiar to a lot).

The problem with using HTML/CSS is that web rendering is insanely complicated, and including chromium/cef in our application would massively bloat the size.

RmlUI isn't actually HTML/CSS, though, it's a subset designed to be lightweight. I'm open into looking into this, the specs look impressive and it comes with a lot of stuff for free - like animation. The fact that it's HTML-inspired with CSS will make it super-customisable.

Indeed, you are right that making our own GUI is insanity (I suffer acutely from this sort of insanity), but I'm afraid using an actual GUI library would only solve part of the issue.

One of the biggest problems is that we need some way to be sending this stuff across the network, and typical GUI paradigms don't work nearly so well that way. GUIs are usually designed to be incrementally updated, but doing so across the network requires batching up changes on the server, sending them to the client, and applying those changes. It's far, far more non-trivial than it sounds, and you end up with disgusting code.

But failing to have incremental changes of some sort makes it so the entire GUI is refreshed for every small change, causing problems with interactive elements. (Updating a formspec every time you move a scrollbar? You lose mouse control of the scrollbar every time the formspec updates.) This eliminates a large class of dynamism in GUIs that you would want in a game. So you've got to implement this on some level. Inevitably, even if you have a nice GUI library lined up, you have to write a wrapper around it to support this whole network thing.

Actually, one of the best GUI paradigms for the job would be an immediate mode GUI, such as Dear ImGui (although that one has its own problems). The API of IMGUIs work by rebuilding the entire GUI every frame, while the framework internally holds state such as which button is pressed, the current cursor location of each input field, etc. Primarily all we would have to do is translate our GUI tree from the server (JSON, for instance) into IMGUI calls, and events are just the reverse. I tried doing something like this in my last attempt at a replacement, and it was more successful than any other.

Of course, there's plenty of other issues, such as how to get a GUI library to play nice without our existing HUD/formspec implementations and conventions. Don't forget about z-indexes having to work between our HUD and the new UI library, double-clicking outside the formspec to close it, GUIs working on mobile, the pause/death menu is a formspec, etc.

I may not have made super visible progress on the GUI replacement front, but I'm willing to bet that I've thought about its design issues more deeply than anyone else has 😁

I believe that the server-side rendering mechanism of Minetest's Formspec needs to be refactored:

Instead of the server sending the entire Formspec code to the client every time a data change occurs, and the client sending every minor data change to the server, the server should only receive the final result. Ideally, data should only be sent to the server when the user clicks the submit button. However, this design would reduce the interactivity of the interface until GUI supports Script (SSCSM).

Inspiration should be taken from the communication between browser JavaScript (React, Vue) pages and servers, and by applying similar concepts and techniques, the implementation of GUI in Minetest can be improved. For example:

• A Formspec can be regarded as a web page, with the data submission part acting as the page's form. Each page should have a unique UUID.
• Data expressions should be used for the data in the form, and the page can use template data expressions to obtain client-side or server-side data.
• The server should only control the display, hiding or closing of the pages it creates.

The client needs a page manager to manage the displayed pages, which can be displayed on a 2D screen or on a 3D object. Pages are incrementally updated on the page display buffer before being rendered.

RmlUI appears to be a good choise, with built-in Lua bindings, animations, flexbox layout, SVG, MVC, and generic event support, making it seemingly ready to use out of the box.

All UIs can be implemented in scripts, and if all three types of Lua scripts are unified into different types of mod libraries: #13654, it would enable all scripts to be shared on contentdb. Unfortunately, it seems no one is willing to implement this feature, despite its potential benefits.

I don't know about the actual gui part, seems that v-rob has that down. But as for the networking side you could use https://en.wikipedia.org/wiki/Xvfb and it would be trivial to include https://github.com/pkulchenko/wxlua to manipulate the network state in gtk2/3 and qt4/5 from lua. Xvfb has stable abi since about September 1987 and its implementation is about 20 years old. The only thing is that gtk2/3 and qt4/5 are extremely bloated toolkits for just displaying menus. The X protocol is designed to work over 52k modems so the network impact would be minimal as would the performance impact be minimal. If a whole toolkit had to be chosen there is https://www.fltk.org/ which is designed to be very much so lighter then gtk2 even. FLTK 1.4 would be a good target for cross compatibility.

I've been thinking about possible approaches to reduce network overhead between client and server, and one idea is that if each datum has a UID then the sender can call to replace or modify by UID reference.

Also, I'm a bit surprised that no one seems to have yet linked to the flow formspec generation library. It has a similar surface syntax as many of the ideas discussed in earlier conversation around this new formspec API.

how is HTML/CSS a problem? its an already working solution that many people are familiar with. I don't see a point to reinventing the wheel here, or making potential modders learn a different system

This has been answered in this thread before. HTML/CSS is very bloated, so you'd be effectively embedding an entire browser rendering engine into Minetest. Moreover, it's not a very good fit for us (for instance, how would we add an HTML inventory list? Not without client-side JavaScript or heavy modification to the HTML engine).

I completely agree that html/css is not a good fit, but I want to point out for completeness' sake that small rendering engines do exist. Litehtml is an example, maybe it could be useful for rich text areas.

i see this has been open for awhile. is there trouble getting a dev to fix this issue? what is blocking this for so long

This is a very big issue to solve, and takes time to implement. However, there is a PR that will close this issue: #14263

Stereoscopic support for the menus please. At the very least, the GUI could be duplicated for each eye instead of half to each eye as is currently the case. Tested with side-by-side stereoscopic mode.