Color / texture transfert

[pvnieo]

Hi, Thank you for the awesome library!
Actually, I'm not reporting a bug, but I'm asking for help.

I'm working on a shape matching problem, where I have two meshes, and I want to assign to each vertex in the first mesh, a vertex in the second meshes. A way to visualize if the correspondances are correct is to transfert color (or texture) from one shape to the other. (see example below)

However, actually all this operation are done in matlab, and I want to perform this in python (in jupyter notebook using Panel).

I succeed to color the shape, or apply texture to it, but I can't see how to transfer this. (see below)

I had the idea to use the scalars property in add_mesh to transfer color, but when I use this, the color is not uniform on the source shape, which will be difficult to transfer (see below).

Can you suggest a method to do this?
Thank you in advance.

[github-actions]

Hi and welcome! Thanks for posting your first issue in the PyVista project! Someone from @pyvista/developers will chime in before too long. If your question is support related, it may be automatically transferred to https://github.com/pyvista/pyvista-support

[akaszynski]

Is there any way you could upload your dataset here for us to take a look at it?

[pvnieo]

Hi, thank you for your response!

Here is a link to download a small dataset: https://drive.google.com/drive/folders/1ij6NdB1WuCnTXyK1SIolLSYIkfIMd6n2?usp=sharing

This link contains 2 .off files, which are two shapes, that can be easily read using pyvista, a texture image that I want to apply to one shape and transfer it to another, and a file X_Y.mat which is a correspondance from X to Y. this file contains an array that has the shape: N x 1, where N is the number of vertices of X, and the value at row i is the vertex in Y to which the vertex i should correspond. To read the correspondance:

import scipy.io as sio
m1_m2 = sio.loadmat("067_071.mat")["matches_gt"] - 1

The - 1 is because indices starts at 1 (matlab style).

Thank you for your help, and hope this is what you asked for.

[banesullivan]

So uh, I totally cannot follow what you mean by:

a file X_Y.mat which is a correspondance from X to Y. this file contains an array that has the shape: N x 1, where N is the number of vertices of X, and the value at row i is the vertex in Y to which the vertex i should correspond.

Can you clarify what you are trying to accomplish?

---

In the meantime, we can load up all the data by:

import scipy.io as sio
import pyvista as pv
import numpy as np
import imageio

# My build of libpng is broken... should use `pv.read_texture()`
texture = pv.numpy_to_texture(imageio.imread("small dataset/texture2.png"))

ds = sio.loadmat("small dataset/067_071.mat")
gt = (ds["matches_gt"] - 1).ravel()
est = (ds["matches_est"] - 1).ravel()

a = pv.read("small dataset/tr_reg_067.off")
b = pv.read("small dataset/tr_reg_071.off")

From a quick inspection, the shapes of the gt and est arrays match the number of vertices in the a mesh, but not b.

What are we supposed to do with these data?

Also, how did you texture map the meshes?

[pvnieo]

Hi, Thank you for your response.
Yes actually, the two shapes doesn't have the same number of vertices.

We want to color the shape b or apply the texture on it, as I have done in the images before. And now for shape a, for each vertex i, we want to color it using the color of vertex gt[i] in shape b. Same thing goes for texture transfer.

I hope that my explanation is clear.

[pvnieo]

Hi all,

I added to the link above 2 shapes, surreal_0000.off and surreal_0001.off, which have the same number of vertices, and we don't need a map gt like before, because the vertex 1 in shape 1 is mapped to vertex 1 in shape 2, vertex 2 is mapped to vertex 2, and so one.

[banesullivan]

Ah okay, this is awesome! So how are you mapping the texture to the meshes?

If you have a 1-to-1 relationship between the two meshes, then simply copy the texture coordinates across the two meshes and there's your texture transfer, right?

[pvnieo]

Hi, Thank you for your response.
Actually, I don't know how to apply texture to meshes directly, what I'm doing is to plot the image and specify the texture in the pv plotter:

pl.add_mesh(mesh, smooth_shading=True, texture=texture)

This way, I don't know how to transfer the coordinates!

[banesullivan]

But, how did you generate the texture coordinates in the first place to create this figure:

pl.add_mesh(mesh, smooth_shading=True, texture=texture)

will throw an error if you don't have texture coordinates.

---

hopefully, this will add some context.

In PyVista, a mesh can be plotted with scalars (numerical values associated with each node or cell in the mesh) or with textures (an image that is draped over the mesh via a texture coordinate mapping for all the nodes in the mesh).

If your two meshes have a 1-to-1 correspondence, then they can share the same texture coordinates.

The tricky part is creating texture coordinates for a complex mesh like yours to produce something like the image above. To do so, you need to map the coordinates of the texture image (0 to 1 in the U and V directions) to each node in the mesh so that you can properly drape your texture/image over the mesh.

Here's an intuitive graphic I grabbed from a google search:

Unfortunately, texture mapping isn't always a simple task. In PyVista, we have a texture_map_to_plane method to drape the image over the mesh from a plane, but that isn't what you need for these meshes... you need something a bit more sophisticated. This could be straightforward if your meshes had some sort of structure to the points, but I am not seeing anything...

This leads me to wonder how you are generating these meshes and if the tooling/software there is aware of what these texture coordinates should be but isn't saving them in the off format.

[pvnieo]

Hi @banesullivan Thank you for you response and explanation.
Regarding your last question, I'm not generating these meshes, Indeed, these mashes are part of some shape matching datasets, so when one has a new method for shape matching, he can Benchmark on these datasets.
For Your first question, what I have done is the following:

mesh = pv.read("mesh.off")
texture = pv.numpy_to_texture(np.array(Image.open('data/texture.png')))
mesh_ = mesh.texture_map_to_plane()
pl.add_mesh(mesh_, smooth_shading=True, texture=texture)

And the result is the following:

[banesullivan]

Hm, okay. I have a feeling that texture mapping isn't exactly the best solution, but it looks decent enough!

Try this:

import pyvista as pv
import numpy as np
import imageio

a = pv.read("small dataset/surreal_0000.off")
b = pv.read("small dataset/surreal_0001.off")
texture = pv.numpy_to_texture(imageio.imread("small dataset/texture2.png"))

# Generate and transfer texture coordinates
a.texture_map_to_plane(inplace=True)
b.t_coords = a.t_coords

# Display
p = pv.Plotter(notebook=0, shape=(1,2))
p.add_mesh(a, texture=texture)
p.subplot(0,1)
p.add_mesh(b, texture=texture)
p.link_views()
p.show()

[pvnieo]

Wow, cool! Thanks a lot @banesullivan !
I tried to use this on meshes that does't have the same number of vertices by

b.t_coords = a.t_coords[gt]

with b and gt having the same number of rows, and I got the following image which is not great as you expected:

I have a question, is there a similar method to transfer color. For now, I can color gradually an image using the following:

mesh = pv.read("mesh.off")
texture = pv.numpy_to_texture(np.array(Image.open('data/texture.png')))
mesh_ = mesh.texture_map_to_plane()
pl.add_mesh(mesh_, smooth_shading=True, cmap="hot")

which gives this: