Common issues to consider before finalizing yakui's design

[coderedart]

o/
egui often has some common complaints that pop up in different issues / discussions. I thought, i might put them somewhere so that it will help yakui can consider as features / use cases while it is still in design phase.

didn't want to unnecessarily pollute the issues section by creating multiple issues, as i don't now which of these are actually within the scope of yakui.

most of these issues have duplicates, so i am only linking to the first one that i came across.

Panning / Zooming container.

emilk/egui#1811
scaling a particular ui object only instead of the whole UI.

1. useful for node graph used in projects like blackjack
2. plots / graphs
3. level editors (2d platformers need to move around the level) etc..

Scrolling (variable sized items in particular)

emilk/egui#1376

Docking

emilk/egui#655
detach a tab into an individual window / area and reattach them somewhere else etc..
very useful in editor like apps. game engines, image editors, video editors.

Multi Window

emilk/egui#1044
This feature combines with docking where people want to undock a certain UI tab into a completely separated native window.
that would be great in these days of multi-monitor setup. kinda like GIMP.
this needs a lot of care. imgui has start work in 2017 december and it is still done completely done yet.
this might interact with panning - zooming containers feature too.

Accessibility

emilk/egui#167
we have AccessKit now and trying to be "friendly" to its api will help adding it eventually.
Though, i think it should be optional as it brings in external system dependencies on linux.
IMO, UI being keyboard friendly (or navigation friendly in general) is also helpful. So, allowing shortcuts to trigger certain actions like shifting focus to a certain widget or moving a widget etc.. are all essential.

Context being single threaded by default

emilk/egui#1399

1. obvious performance benefits for users who don't care about multi-threading, which is most users.
2. simpler api for yakui. Users can wrap it over Arc<Mutex<>> if they want multi-threading.
3. avoids issues like mutex deadlocks

Context menu layout

emilk/egui#1176
I don't think yakui will face such issues as it using a layout algorithm. but basically, context menu needs to decide in which direction it will be created (and moved) when faced with size constraints (like right click at bottom of the window should make context menu layout towards top as it won't have enough space to fit in bottom).

Z-order of widgets / Overlapping widgets

emilk/egui#1516
emilk/egui#980

Mnemonics / Navigation features

emilk/egui#429

Text rendering

emilk/egui#56
Issues with blending / gamma issues. text looking blurry or low contrast or jagged edges (anti aliasing) etc..

Modal dialogs

emilk/egui#686
well, i say modal dialogs inside the UI, but i also mean file dialogs like rfd. integrating them without much pain would be nice.
one of my favorite things that i want to see is a command pallette. like vscode's "ctrl + shift + p" or linux user's rofi launcher etc.. and having the command pallette on top of everything is essential to make it possible. reference https://github.com/hnOsmium0001/imgui-command-palette

Finally, don't forget to add yakui to https://www.areweguiyet.com/ .

[LPGhatguy]

This is an excellent writeup, thank you!

[coderedart]

forgot to mention the main reason i made the issue :) :
API / ABI bindings
emilk/egui#1004
closure based API is very hard to make bindings for in a safe manner.

• As yakui will be used in games, people will eventually want to use embedded scripting languages to add UI (atleast for debugging or in games like Garry's mod / roblox).
• Closures usually pass &mut Ui like temporary mut refs of objects, and AFAIK, only mlua supports passing in arguments with lifetimes (mut ref Ui will need to be wrapped in a new struct). that is how i was able to make https://github.com/coderedart/luaegui which covers lots of egui api already. only egui::Window had to use unsafe in 2 places which takes a mut ref to bool for the open status because i was lazy at that time.
• I tried to make bindings for rhai as a proof of concept to script egui on web . but i had to use unsafe every single time to pass in mut ref Ui as structs with lifetimes can't be sent into rhai. well, i say rhai, but basically most bindings don't allow non-static data except for mlua.
• the linked issue above has someone trying to attempt python bindings with no success.
• one way to deal with that is to make a low level C FFI API like imgui or wgpu etc... and the other languages can take them and convert them into safe bindings suitable for them. but this means a large amount of rewrite work for egui.

NOTE: we cannot use lua to script egui on web yet because mlua doesn't work on web yet. there is work going on in the background by the mlua author to port lua vm to rust. if that's the case, the closure based api is good enough for lua bindings.

[LPGhatguy]

I've had some time to think about these. I think we should make issues for the places where we fall short right now. Here's a summary for now:

Panning / Zooming Container

We should definitely support this. I think this requires some thought to do well, but maybe letting widgets set a transform matrix for all their descendants would be a good approach.

We would need the widgets inside the zooming container to render to a higher resolution, like text. This will probably require smarter text glyph caching.

Virtual Lists in Scroll Views

In theory this should be very reasonable to implement as a widget. Implementation will probably resemble virtual list implementations from other UI toolkits.

Docking

Yes, would absolutely love to do this at some point.

Multi Window

There are a lot of good use cases for "multi-surface" UI in general. We should be able to draw some of the UI in a viewport in a game's world, some of it on screen, and some in another native window.

I agree that this could overlap with the panning containers. That's a great observation and might give us a hint on how to do that well.

Accessibility

Accessibility is VERY important to me. This was something I've pushed for at every company I've worked at and I think it's critical to the industry-wide success of a production game UI library.

I had not heard of AccessKit. This looks great! We might have an opportunity to integrate AccessKit sooner than later and help ensure it meets our needs. I would also love to help fund its development if that's something the author(s) would be interested in.

Single-threaded Context

I think we're set here. yakui operates in a single thread for simplicity and I think it gives some big wins. If anything, we would want to use inner parallelism where we can leverage it for interesting algorithms, like parallel layout.

Overlapping Widgets and Z-Ordering

We do a pretty good job of this too. Input events are sunk by the visually highest widget. This is really easy to do because of our layout model! 😃

It would be nice to build more fine-grained controls for layering besides just the order that the widgets appear in code. It's easy to stack things on top of each other right now, but sometimes having z-index is valuable.

Mnemonics / Navigation Features

Interesting! I don't normally use the mnemonics feature of app menus. I think this makes sense to support where we can and we should design our menubar widget with this in mind when we get there.

Text Rendering

Our text rendering is not great on systems with >100% system scaling. Our text rendering pipeline is currently distinct from our main pipeline. I think our blending is better than if we merged them (premultiplied alpha?) but definitely work to do still.

Modal Dialogs

This should be easy. Our API for sinking events should mean that an inline modal dialog can sink clicks outside of it.

External dialogs like rfd is a good use case to keep in mind. It's something we need in our engine. I don't think it would be too bad; we could make a widget that lets you manage external dialogs declaratively.

API / ABI bindings

Oh, thank you for raising this! The DOM has begin_widget and end_widget that everything is implemented in terms of. I hope these methods are helpful for bindings.

I don't have any plans personally to bind yakui anywhere, so if there are any APIs that would help do that, feel free to file specific issues and we can figure out what the best approach to take is.

[coderedart]

Mnemonics / Navigation Features

I never used the Mnemonics feature either. but it felt like vim normal mode, where you can browse the text with jkl; or other keys.
For example, on browser i use extensions like vimium. when i press f (follow link), it will highlight and tag all the links on page with short key combos

and when i press any of those keys in order, it will click / trigger that link.

when i read the linked issue of Mnemonics, i suddenly realized how nice it would be if i can create such actions in egui too. especially for folks who use keyboard 90% of the time (terminal / vim / emacs freaks), navigation using keyboard is a really really god tier feature.

API / ABI bindings

generics is very hard to bind to.
a good example is Into<WidgetText> of egui. it uses such argument types in a lot of places. so, i have to create an enum of IntoWidgetText with all possible types (string, rich text, widgettext) covered to make it work in lua bindings.

It is not completely impossible though. every dynamic language (js/py/lua) has some sort of "Object" struct with fields / values. so, we would just need to implement the Widget trait for that object type and dynamic languages can figure out the rest. for example, i an impl the Into<WidgetText> trait on a table (object) in lua and i can just document that they would need to set a function inside that object / table which when called provides me a WidgetText.

C API

ideally, a C API would be the best. I know that C API will have some unsafe constraints, but the languages who need the bindings will have to create higher level APIs with safe abstractions. this will allow the other language bindings to create their own new from scratch custom widgets.
egui's dev said so too at emilk/egui#1004 (comment)

I would be more interested in investigating a more low-level C-like API (with begin/end) and then build the current Ui interface on top of that as a convenience. But that's a lot of work. It also is not obvious how one would handle third-party containers.

an alternative is to create a C API over the existing safe rust API. reference: emilk/egui#1360 . imgui does it this way https://github.com/cimgui/cimgui

• having structs free of generics (including or rather especially lifetimes) is very important for C. I hated dealing with the egui::Window<'open> struct when binding lua. I plan to request egui to remove that lifetime from one of the most commonly used struct and use some other api.
• #[repr(C)] types so that users can pass around simpler structs without causing undefined behavior.

Reflection / serde.

This is how current bevy-egui-inspector does things. it just creates a default mapping of value type => widget, and when derived, it will recursively keep creating widgets for each field / value combination. it relies on the bevy_reflect::Reflect trait to do this which provides the list and types of all fields within a struct / enum. this can allow us to avoid C API and use some common format like json to enable description of widgets.

plugins are not always used via embedding a scripting language and allowing it to directly modify host's data. sometimes, they can be implemented by message passing communication methods like Inter Process Communication. a couple usecases are:

1. JsonRPC. vscode extensions use this. each extension runs in a nodejs instance (i think) and communicate with vscode via rpc. this enables them to do long running operations in the background without blocking the main vscode app (or other extensions).
2. declarative UI. addons might for example be long running processes in the background (or even run remotely via network), and they can just provide a json / xml file with all their UI pre-declared so that the main app / game can take care of deserializing them into widgets and calling them. now, they can handle onclick or other events via some sort of async message passing and the addons can communicate to the parent if they need to change their UI (button background color) props or even replace it with a new UI by sending json / xml (something something serverside rendering xD).
3. wasm plugins have been a dream for years at this point. you cannot pass rust data without serializing into / out of wasm yet. they can only deal with primitive types like C ABI or use the serialization mechanism.

eg: Button

{
	"widget_type" : "button",
	"text" : "click me sempai",
	"on_click" : "clicked", // send this string to the addon in the background
	"width" : 64.0, 
	"height" : 32.0
}

yakui itself doesn't need to do much to help with Reflection, but the core types (layouts or size / pos types etc..) being serde friendly would help a lot. especially by avoiding magic types like box / arc / rc where possible.

ofcourse, having some sort of reflection system would make it much easier to programmatically create bindings. but that's a feature request for another day. if we need to create bindings programmatically, rustdoc's json output might even turnout to be a better approach.

reference: wasm is going to use a structural typesystem bytecodealliance/wasmtime#4522 and eventually it might become the defacto reflection system in rust as it defines both the api and the abi.

[mwcampbell]

@LPGhatguy I'm the main developer of AccessKit, and I just came across this issue. I'm glad to read that accessibility is a priority for you.

If you want to see where I'm at with AccessKit, check out my egui branch. I just merged text editing support. So far, AccessKit only has a Windows implementation. I'll be focusing on Mac over the next couple of weeks though.

I've got funding covered for the remainder of this year. After that, I don't expect to be available for funded work on AccessKit until the middle of next year, since I've also got other projects that need my attention.