Feature request: support rendering of color and N-D features simultaniously

[SuperXia123]

Support a user-defined feature map as optional input, which could be used to encode any wanted feature with flexibility, such as semantics, normals, etc...

The input feature map is a torch tensor with dimension [N, C]，where N is the gaussian number and C is the channel number which depends on the user wanted features. The output feature map is [C, H, W], indicating the alpha-blended gaussian features.

The users the decode the output feature map according to their encoding strategy, while the rasterization engine do not care about the meaning of the feature map.

A similar implementation of this feature could be found here.

[maturk]

Rasterizing of N dimensional features is already supported by the gsplat backend. here

[SuperXia123]

@maturk Thanks for your response!
If I understand correctly, one can not render colors and self-defined feature channels simultaneously during a single function execution.
If that is true, should I perform the rasterization function twice if I want both colors and features to be rasterized?

[maturk]

Correct simultaneous rendering with a single forward pass call is not supported besides for rgb+depth rendering.

I met a similar problem in my dn-splatter project where i wanted to render rgb+d and normals with a single forward pass. Its easier to do it twice, unless performance is necessary. But this approach is not so elegant.

Happy to discuss this problem further, since it might be a useful thing to support better.

[yzslab]

But, I think it is possible.

If you are using rasterization:

1. Manually call the spherical_harmonics to convert your SHs to RGB
2. Concatenate your RGB and features as a new tensor in [N, 3+F], where F is the number of dimension of your features.
3. Call the rasterization with the concatenated tensor and sh_degree=None

Example:

camera = ...

dirs = model.get_means() - camera.camera_center
colors = spherical_harmonics(model.active_sh_degree, dirs, model.get_shs())
colors = torch.clamp_min(colors + 0.5, 0.0)

extra_features = ...

input_features = torch.concat([colors, extra_features], dim=-1)

rendered_colors, rendered_alphas, meta = rasterization(
    ...
    colors=input_features,
    sh_degree=None,
    ...
)

rendered_rgbs = rendered_colors[..., :3]
rendered_features = rendered_colors[..., 3:]

Another option is to re-implement your own rasterization pipeline using fully_fused_projection and rasterize_to_pixels, instead of relying on the provided rasterization interface.

[SuperXia123]

But, I think it is possible.

If you are using rasterization:

1. Manually call the spherical_harmonics to convert your SHs to RGB
2. Concatenate your RGB and features as a new tensor in [N, 3+F], where F is the number of dimension of your features.
3. Call the rasterization with the concatenated tensor and sh_degree=None

Example:

camera = ...

dirs = model.get_means() - camera.camera_center
colors = spherical_harmonics(model.active_sh_degree, dirs, model.get_shs())
colors = torch.clamp_min(colors + 0.5, 0.0)

extra_features = ...

input_features = torch.concat([colors, extra_features], dim=-1)

rendered_colors, rendered_alphas, meta = rasterization(
    ...
    colors=input_features,
    sh_degree=None,
    ...
)

rendered_rgbs = rendered_colors[..., :3]
rendered_features = rendered_colors[..., 3:]

Another option is to re-implement your own rasterization pipeline using fully_fused_projection and rasterize_to_pixels, instead of relying on the provided rasterization interface.

@yzslab I tried this strategy and it worked. Thanks a lot!