pixel coordinate to model coordinate (interactive visualization)

[yihsung]

I have a problem in doing interactive visualization (in python). It seems that there is no API can both intercept key-events and mouse-events, so I turn to use open3d.visualization.gui to implement everything manually. For the mouse-event, I got the pixel coordinate from the API, but I have difficulty in turning it into the model coordinate.

I noticed the open3d.visualization.rendering.Camera.unproject method, but it didn't give me reasonable outcomes... What made me more confused was when I changed the viewpoint, the view_matrix was supposed to change as well, for example, if I pulled away the camera, the translation part of the matrix was supposed to change. However, it stayed the same all the time. Could anyone help me with either

1. turning a (e.x, e.y) (pixel coordinate) to model coordinate or
2. explaining how unproject method works or
3. obtaining the right view_matrix of the camera.
   Thanks!

[yihsung]

Another issue is: mat_matrix*model_matrix = identity (always). That shouldn't be true in general. Please correct me or help me find the right matrices.
Thanks!