Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support ui-interaction with video #7353

Closed
jprochazk opened this issue Sep 4, 2024 · 2 comments · Fixed by #7457
Closed

Support ui-interaction with video #7353

jprochazk opened this issue Sep 4, 2024 · 2 comments · Fixed by #7457
Assignees
Labels
ui concerns graphical user interface 🎞️ video

Comments

@jprochazk
Copy link
Member

jprochazk commented Sep 4, 2024

We have video frames available in regular GPU textures, so we should be able to do GPU picking for hover UIs

@emilk emilk added the 🔺 re_renderer affects re_renderer itself label Sep 4, 2024
@Wumpf
Copy link
Member

Wumpf commented Sep 9, 2024

What's the task here exactly? We don't have gpu picking for images either at all, so what do we want to accomplish here?
I.e. what's the ui outcome we're lacking and wanting to achieve here?

@emilk emilk added the 🏎️ Quick Issue Can be fixed in a few hours or less label Sep 9, 2024
@jprochazk jprochazk added ui concerns graphical user interface and removed 🏎️ Quick Issue Can be fixed in a few hours or less 🔺 re_renderer affects re_renderer itself labels Sep 9, 2024
@emilk
Copy link
Member

emilk commented Sep 9, 2024

There is several steps here:

A) support clicking videos to select them
B) showing a magnification of the video on hover
C) actual color picking on hover (read exact RGB value)

Only C requires reading back the texture to CPU, and is also the least important

@jleibs jleibs changed the title Support GPU picking for video Support hover/picking for video Sep 9, 2024
@jleibs jleibs changed the title Support hover/picking for video Support ui-interaction with video Sep 9, 2024
@emilk emilk added this to the 0.19 - Dataframe and web video milestone Sep 10, 2024
Wumpf added a commit that referenced this issue Sep 19, 2024
### What

* Part of #7353
* still missing hover preview, but this PR gets us to the minimum viable
product.

Generalizes the picking-rect mechanism to a large extent to include both
videos and error placeholders.
Picking code was in a quite poor state, so most of the changes here are
all about making it cleaner and easier to follow. Still some left to be
desired but had to stop somewhere ;-)

<img width="1264" alt="image"
src="https://github.com/user-attachments/assets/859aaa69-d367-4428-b7e7-df1e439fb7a9">


Overview over the refactor (it's almost commit by commit, but I went
back & forth on naming a bit and there's _some_ evolution that I didn't
clean up):
* replace `PickableImage` with a more versatile `PickableTexturedRect`
construct
* each visualizer now has a list of `PickableTexturedRect` to dump its
images into directly on the shared `SpatialViewVisualizerData`
   * similar to how ui label collecting works
* this allows all reasoning about pickable rects to be fully generic, no
need to know all visualizers
* the only exception are depth clouds which aren't images from a picking
perspective but get resolved to one
* pull the `PickingContext` creation out of the mighty `picking` method
* renames on `PickingContext` to make it more clear what's going on
* some more meaningful sub routines for the mighty `picking` method
    * it's still large and complex, but a lot better now 


### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7450?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7450?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!
* [x] If have noted any breaking changes to the log API in
`CHANGELOG.md` and the migration guide

- [PR Build Summary](https://build.rerun.io/pr/7450)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

To run all checks from `main`, comment on the PR with `@rerun-bot
full-check`.
@Wumpf Wumpf closed this as completed in 47c6e47 Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ui concerns graphical user interface 🎞️ video
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants