Pkl output of STAR fitting

[Emfraje]

Hi

I successfully installed the package and ran the code to fit the OSSO model with meshes from the AMASS dataset. This process generated two output files: a .ply file and a .pkl file.
Next, I used the generated .pkl file as input for the SKEL (https://github.com/MarilynKeller/SKEL). In addition to outputting the generated skeleton, I also outputted the reconstructed skin using both

• the SMPLSequence from: https://github.com/MarilynKeller/aitviewer-skel/blob/main/aitviewer/renderables/smpl.py
• the SKELFitter provided in the align_to_smpl.py (https://github.com/MarilynKeller/SKEL/blob/master/examples/align_to_SMPL.py)

Below the results: On the left side, an image of the input data for OSSO/main.py. In the center, the input data combined with the mesh (edges indicated) of STAR. On the right, the STAR registration in combination with reconstructed skin from the pkl file.

From the above, it is clear that the skin meshes deviate. The initial deviation between the input and the STAR registration is expected.
However, the difference between the .ply output from the STAR registration and the reconstructed skin from the .pkl output is more surprising. Would someone have explanation for this? Is there a different rigging tree used for the STAR registration and the way I tried to reconstruct the skin?

Next, I wanted to run code on my own data. I made sure to register the input meshes before running the OSSO/main.py. I tried both .obj and .ply as input. Again, the output of the STAR registration in main.py deviated from the input mesh, which is not surprising.

I again used this .pkl file as input for SKEL. Here is were it seriously goes wrong. I notice I large deviation in the skin mesh. (The largest one is the one created by the pkl file). There seems to be a scaling issue? However, it is not a uniform scaling.

Would you have any guidance on what goes wrong and how to correct this?

Any help would be appreciated

[MarilynKeller]

Hi, I think the issue is here:

I used the generated .pkl file as input for the SKEL

The pkl output by OSSO contains the STAR pose and shape parameters matching your mesh: theta_s and beta_s.
Yet, the SKEL code expects SMPL parameters, which are different from STAR parameters.

I believe the mesh you reconstruct from the pkl is SMPL(theta_s, beta_s) (purple wireframe), which gives a different body shape and slightly different pose than STAR(theta_s, beta_s) (black wireframe), as shown on your first image.

To precise, what the SKEL fitting code does is :

• load theta and beta from the pkl
• generate the body mesh M = SMPL(beta, theta) (I believe this is what you show in purple)
• Optimize SKEL(beta, q), to find the skeleton pose q that fits the best the purple mesh M

So to make this work, you need the SMPL parameters for your mesh (not the STAR parameters). I see 3 options:

1. I can be wrong but your mesh looks like a sequence from the DFAUST dataset, and SMPL fits have been released publicly. If it is the case, let me know where you downloaded the data and maybe I can direct you to fits.
2. Your input mesh seems to be a SMPL fit already. You can modify the SKEL code to
   i) not use a pkl but just use your input mesh (bypass this line) to load the mesh instead)
   ii) optimize both beta and q in SKEL(beta, q) (Add beta to this line and the next ones)
3. Or fit SMPL to your mesh to get SMPL parameters

Also, it is not clear to me at all why you are using OSSO in the process. Was it just to get a .pkl for SKEL?