Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate raylib and others for ui #33301

Open
adeebshihadeh opened this issue Aug 14, 2024 · 7 comments
Open

Investigate raylib and others for ui #33301

adeebshihadeh opened this issue Aug 14, 2024 · 7 comments

Comments

@adeebshihadeh
Copy link
Contributor

adeebshihadeh commented Aug 14, 2024

openpilot's UI used to be written in https://github.com/memononen/nanovg, and in retrospect that it was nicer to work in than . Qt is nice for cabana, but it turns out to not add much value to the openpilot ui and adds a bunch of conceptual and dependency complexity.

Some options

We're looking for:

  • cross platform: Linux, macOS, etc.
  • small, simple. single header is ideal
  • Python bindings? would be great to have all or most of the UI written in Python

The onroad camera view is a good test for trying out these libraries.

@adeebshihadeh
Copy link
Contributor Author

@deanlee want to try to make a small demo in raylib of the onroad camera view?

@deanlee
Copy link
Contributor

deanlee commented Aug 16, 2024

I tried writing a simple demo based on Raylib to display the road camera (without shaders and transforms, just to quickly get a feel for Raylib's basic features).
Raylib can simplify some OpenGL code, and its text rendering performance is also somewhat better (currently, in the drawHud function of the Qt UI, the three drawText function calls consumes about 3% of the CPU).
Raylib might be a good option. The real challenge is the workload of migrating the entire UI to Raylib, including rewriting network management, WiFi manager, and multilingual support, among other features. Additionally, Raylib itself lacks a signal-slot mechanism. All logic is driven by a while loop, and input/output would also need to be restructured, making the workload quite significant. Considering that the current UI is relatively stable, if the goal is just to save space, it might be worth first optimizing other dependencies that can be relatively easily removed before eventually migrating to Raylib, RayUI, or another library.

Although it's possible to use Raylib within a Qt widget to render graphics, this actually increases complexity and maintenance difficulty. There's no good way to gradually transition to Raylib and remove the Qt dependency through multiple PRs. It looks like a complete rewrite would be necessary.

@adeebshihadeh
Copy link
Contributor Author

adeebshihadeh commented Aug 16, 2024

Qt was a big improvement over the old APKs, and I expect this to be a similar improvement. Smaller dependency, simpler library, better docs, etc. I also expect it to help with #30294.

I think we do have to find some way to do this incrementally; like you said, it's too much to do all at once. If you can't figure out a way to do it, I can start it. Want to put up a draft PR with the demo you wrote?

@deanlee
Copy link
Contributor

deanlee commented Aug 16, 2024

I'll commit a draft after adding the shader, transformations, and other features to ensure it matches the CameraView's output.

@AskAlice
Copy link
Contributor

AskAlice commented Aug 18, 2024

fyne is quite nice although it's written in golang which would be a bit of a hurdle, though it wouldn't be the first time the UI were separate from the py.
I think it also might be worth considering from this list https://github.com/sudhakar3697/awesome-electron-alternatives#python

@deanlee
Copy link
Contributor

deanlee commented Aug 19, 2024

PR #33326 provides a draft on using Raylib to display the road camera, with automatic scaling to fit the window while keeping the aspect ratio.

@deanlee
Copy link
Contributor

deanlee commented Aug 25, 2024

First Step: Migrate Onroad Widgets to Raylib
1.Organize the existing code by encapsulating related functions into specific classes to improve clarity and maintainability, The new class will expose only two public functions: updateState() and draw(), facilitating a smoother transition to Raylib.

After step 1, the AnnotatedCameraWidget will function as a container for various components such as HudRenderer, AlertRenderer, DriverMonitorRenderer, ModelRenderer, and others. Its responsibilities will include updating transformations, managing window size, and handling user input. It will not handle state updates or rendering directly; instead, these tasks will be delegated to the drawing functions of its contained classes.

Following Step 1, we can also remove the UIScene struct from uiState to streamline it, allowing uiState to hold only the most essential state information required by the entire application.

  1. Migrate to raylib

Tools:

Misc.

The final directory structure might look like this:

alerts.cc – Handles onroad alerts.
dmon.cc – driver monitor updates and rendering.
hud.cc – HUD updates and rendering.
model.cc – model updates and rendering.
annotated_camera.cc – container for update and rending onroad window.
onroad_home.cc – container for the Raylib window.
raylib_helpers.cc – helper functions for tasks such as managing fonts and drawing text.

Here's a rough draft, written from scratch (interface issues can be addressed later). Although there is a lot of temporary code and commented-out sections with no cleanup yet, approximately 600 lines of onroad code have already been reduced.

Screenshot from 2024-08-26 01-22-01

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants