Split ViewCoordinates component

[emilk]

ViewCoordinates currently has many different responsibilities:

• A) Set the world UP axis
• B) Set the pinhole frustum direction (shortcut: log_pinhole(…, camera_xyz=…))
• C) Naming object axes (e.g. in the cars local space, X is forward and Y is Right)
  • we don't show these names in the 3D view yet, but we should:
  • Add XYZ axes and view coordinate labels to everything #1111
• D) 2D view coordinates for deciding how images should be oriented (not yet implemented)
  • view_coordinates for 2D views #1387

---

I think it makes sense to split these out into different components:

• A) 3DCoordinates: Y=Up (partial) or X=East, Y=North, Z=Up (full)
• B) CameraCoordinates: what we currently log using log_pinhole(…, camera_xyz=…)
• C) AxisLabels: XYZ=RFD (local axis labels for vehicles etc)
• D) 2DCoordinates: XY=RD (2D view coordinates)

(not the final names).

Note that all of these (except perhaps AxisLabels?) are blueprint components, i.e. per space-view.
At the same time, it is important to be able to set the default 2DCoordinates and 3DCoordinates for the entire app, so you don't have to specify it per space view.

B) and C) are functionally equivalent, and should maybe be the same component (basically exactly the same as the current ViewCoordinates)

See also

• ViewCoordinates are not working properly for camera frustums #2244

[jleibs]

I've spent some time thinking about this today and wanted to get down a few thoughts.

As a design-principle we've previously suggested "the addition or removal of the view coordinates component should only change the way the Rerun camera looks at the data, but should never change the way the data is mathematically related. Concretely, whether the ray cast from a hovered pixel in an image intersects a point should not change regardless of any application of view coordinates."

On splitting out Camera Coordinates

This is really the main driver of this work as this is the current usage that violates the above principle.

Currently, pinhole camera_xyz plays a sometimes surprising role by effectively inserting something that acts like an extra RigidTransform into the chain. Once we build our transform inspector, I would expect this implicit transform to be described somehow, as it is necessary to fully define the mathematical relationship between pixel coordinates and 3d space.

Splitting out the component subsequently makes sense, but only solves half of the problem -- it still leaves us with the questions of API and representation.

At the moment we allow the user to specify constraints such as "X Right", referring to:

• X axis of the parent frame
• "Right" from the image.

However, continuing to use that terminology leads to a similar problem. If I have said that "Right" from my image aligns with X from my parent frame, but then I log a ViewCoordinates that makes right a different imager-direction, then either I change the transform, or I end up in a state where right isn't right.

The "Correct" thing to do is, as with transform, would be to avoid any assumption of directionality of axes and refer to x/y/z specifically. However, I'm struggling with a compact syntax.

Practically we want something like

camera_z = parent_x, camera_y = -parent_z, camera_z = parent_y

(note that if parent_x is a vec3, then this is essentially a basis-vector representation of an arbitrary 3x3 transform matrix)

This seems like a lot of complexity for a feature that's there primarily only a convenience in a situation where people didn't want to think about transforms in the first place.

This leads me to considering a proposal like "if you use camera_xyz, we just disable view coordinates and warn if you've used it." But if you can't use the two components together, what have we actually gained by splitting it out into a separate component?

On Splitting 2DCoordinates from 3DCoordinates

In the descripion it says B and C are functionally equivalent, but I wonder if you meant A and D.

2D Spatial spaces involving cameras tend to be practically be 2.5D. This extends to almost all real-world 2D rendering with concepts of both handedness and depth showing up in these contexts. It's often necessary to indicate whether Z is forward or backward, and the existing view coordinates let's us handle this already.

Requiring a user to use a different archetype/component to indicate this also seems error prone. It becomes possible to set the wrong component type for the space, or worse to have conflicting 2D and 3D view coordinates on a single space.

Unless there is a compelling reason to have both on the same entity, I would rather just add additional convenience constants to ViewCoordinates to make it easy to specify XY in a 2D context.

Axis Labels

AxisLabels mostly makes sense to me. I do slightly worry about this being error prone in that it makes it possible to name an axis without setting the direction and vice versa. Probably ok if they are part of the same archetype though since they could generally be logged in a single place.

[emilk]

In the descripion it says B and C are functionally equivalent, but I wonder if you meant A and D.

I meant what I wrote.
A) only need to specify one axis (what is up)
B) and C) both need the three axes X=Forward, Y=… etc mapping.
D) needs specify two axas (what is up, what is right).

[jleibs]

B) and C) both need the three axes X=Forward, Y=… etc mapping.

We're definitely thinking of these differently. Let's sync tomorrow.

(B) in my mind requires a representation that is equivalent to a transform since, as noted above, that's what it's doing under the hood to re-point the camera in an arbitrary direction. Also, as noted, I'm not sure how we even talk about right/left/up/down in this context since changing the 2D coordinates would then implicitly change the behavior of the re-projection.

(C) on the other hand requires nothing other than a list of 3 arbitrary strings to label the axes.

[emilk]

Yeah, I think you right - we may need to split ViewCoordinates into foure separate things (with wildly varying urgency)

[jleibs]

Yeah, I think you right - we may need to split ViewCoordinates into foure separate things (with wildly varying urgency)

What do you see driving the urgency for each?

[emilk]

My view on urgency:

Summary

• Keep support for: 3D up-axis, and camera_xyz
• Soon add support for 2DCoordinates (within the next few cycles, maybe this one)
• Wait with adding general axis labels (including east/west/north/south labels for 3D coordinates)

Details

A) 3DCoordinates

Being able to specify the up-axis is important to keep, so whatever refactor we do should maintain that feature.

Being able to specify east/north etc is not urgent (and should maybe we done with C/AxisLabels instead).

B) CameraCoordinates

What we currently log using log_pinhole(…, camera_xyz=…).
Important existing feature that we should keep.

C) AxisLabels: XYZ=RFD (local axis labels for vehicles etc)

New feature; not very urgent

D) 2DCoordinates: XY=RD (2D view coordinates)

New feature we want to introduce, that many people ask about (in particular setting Y=Up for a specific 2D spatial view)

[jleibs]

Of these, 2DCoordinates is the only thing that seems pressing. But given ViewCoordinates is currently working for 3D and Camera use-cases I don't see an urgency in replacing those.

Is there a reason we can't/shouldn't do the easiest thing that works here and just expand ViewCoordinates to support 2D spaces as in:

• Generate new constants for 2D View Coordinates #6178