Viewpoint tracking not working

[brettle]

Describe the Bug
Viewpoint tracking does not appear to be working.

Steps to Reproduce

1. Load viewpoint_test.zip which has Viewpoint.follow set to "robot:camera" and Viewpoint.followType set to "Mounted Shot".
2. (Optional) Select View > Optional Rendering > Show Camera Frustrums

Expected behavior
The view in the 3d window should be from the perspective of the camera.

Screenshots
Instead, the view looks like this:

System

• Operating System: Fedora Workstation 40
• Graphics Card: Intel® UHD Graphics 630 (CFL GT2)

Additional context
Running the simulation doesn't seem to have any effect on the issue.

I'm seeing this in R2023b and the current R2024a master.

[brettle]

Further investigation reveals that the Viewpoint's perspective is updated after each step, but it is updated relative to the perspective that the Viewpoint had when tracking was initiated. As a result, to get the effect I expected (where the Viewpoint has the same position and rotation as the tracked solid), one needs to copy the translation and rotation fields of the tracked Solid into the position and orientation fields of the Viewpoint and then have the simulation running while you update the translation and rotation fields of the tracked Solid. Is that the way it is supposed to work?

[omichel]

Yes, that's the way it is supposed to work. It allows setting the viewpoint with some offset with respect to the position and orientation of the robot, which is often useful.

[brettle]

I can see how that would be useful, but I think that can be achieved while providing a user experience more inline with what I was expecting. Thoughts on the following possible changes?

1. When the user initiates tracking, the Viewpoint should be set to the perspective of the tracked object. The user could still modify the viewpoint from there.
2. Update the viewpoint whenever the tracked object's perspective changes, not just after each timestep. This would allow the user to, for example, manually aim a camera (while seeing what is within its fov) without needing to run the world.

If the current behavior is to be maintained, the associated documentation should probably be made clearer.

[omichel]

I believe there are use cases where the current behavior is useful. For example, if you watch a soccer game, you want to set the viewpoint to some position and orientation and instruct to follow the ball. You don't want to change the viewpoint that you just set. Generally, changing anything set by the user is usually considered as a bad UX. So, I would not apply your first suggestion. The second one probably makes more sense and could be implemented.

[brettle]

I should have been clearer in my description of my first proposal above. I'm proposing that when the user initiates tracking, the Viewpoint should be set as follows:

1. For "Mounted Shot", the Viewpoint should be set to the perspective of the tracked object. This was the case I was focused on and is the behavior I expected to see.
2. For "Pan and Tilt" or "Tracking Shot" (one of which is presumably what would be used in your soccer example), the Viewpoint translation should not be changed, but the orientation should change to center the tracked object in the view. (There is an extra "twist" DOF in what the orientation should be. Presumably it should correspond to the world's up axis being vertical in the view.)

To be clear, the above would only occur when the user initiates tracking. Since the user has requested the tracking, I don't see changing the Viewpoint in response as a bad UX. From a UX perspective, this behavior is the one that provides the "least surprise", imo.

To return to your soccer example, if you want to track the ball but not have it centered in the view, it seems easier to first track the ball and then adjust the Viewpoint than it would be to first set the Viewpoint and then track the ball.

[omichel]

This is difficult for me to evaluate this without being able to actually test it. Maybe you can go ahead with some implementation and we can test it to see if that seems better than the current system.