Add an ability to cut holes from terrain geometry

[mrDIMAS]

There is no way to cut holes in terrains geometry, so there is no way create cave entrances in the terrain. This may be a real blocker for some games.

It is also important to add proper tooling for this. The editor should have special brush, that allows to cut holes of various shapes.

[b-guild]

Is the goal to make part of the material transparent, or is the goal to actually remove parts of the mesh?

There are costs associated with rendering transparent materials, but aside from that making part of the terrain transparent ought to be technically less difficult.

Removing part of the mesh means the terrain does not need to be rendered as transparent, but it also means we have to store more mesh data as it cannot simply use repeated instances of that one mesh block. There would be limits to the available hole shapes and we would need a data structure to store the holes. We would have to think about how the holes interact with the LOD system. Are the holes always rendered at max LOD? Are there different versions of the holes for different levels of detail?

[mrDIMAS]

I think it should be purely visual, which means that there should be a mask for holes. Also, such holes should be taken into account when making height field collider for terrains.

[b-guild]

Does this mean that the terrain will need to be rendered using forward rendering in order to make the holes transparent?

[mrDIMAS]

No, check terrain default material, it is already rendering with alpha blending into G-Buffer. This is needed to support multiple layers. Holes will act like a simple alpha test and that should be enough.

[b-guild]

If it is that easy, then why does the documentation say that forward rendering is necessary for transparency? The rendering section of the Fyrox book says that forward rendering "should be only used to render translucent objects." The documentation for scene::mesh::RenderPath::Deferred says "Deferred rendering has much better performance than Forward, but it does not support transparent objects and there is no way to change blending."

The documentation for Unity also claims that their deferred shading render path cannot handle transparency and that forward shading is necessary in that case.

I am no expert at shaders, but the GBuffer pass of the terrain shader has depth_write set to true, and that seems like it could cause trouble for transparency, as it should stop anything further away from being rendered regardless of alpha.

It should be possible to prevent pixels from being written using the discard GLSL command, but it seems that including that command anywhere in a fragment shader seriously reduces performance, and so it is recommended to be used only on small meshes.

[mrDIMAS]

Terrain shader uses alpha blending to stack multiple layers on top of each other in consecutive render passes and it works fine, simply because it is done on gbuffer layers. It is different from rendering final frame with transparent objects.

As for performance, afaik discard causes problems only if you're using derivatives in fragment shader, because it ruins normal parallelism. In most of other cases it is fine. If you look closely at standard material, it uses discard as well for alpha testing (alpha < 0.5) and average performance is ok even if there's a full screen quad with mesh-like texture where more than half of the pixels are discarded. Keep in mind, that all these "it is bad for performance" notes were written 10+ years ago and GPUs evolved dramatically since then.

Edit. Also, you can already "cut holes" in terrains, but just on per-layer basis. This task means global mask for holes and takes physics into account.

[b-guild]

When you say "takes physics into account" does that mean that one of the goals is to tell Rapier to let things fall through the holes? Are there any suggestions about how that might be done? It seems that we might have to generate a mesh in the shape of the hole just for the sake of Rapier, unless there is a feature of Rapier that I have missed.

It would be much easier to generate a collider for the terrain if instead of using a mask for the holes we were to just remove particular triangles from the mesh. That would force the holes to have jagged, pixelated, grid-based edges, but I suspect that when holes are used in practice the player would never see the edges of the holes because they would be covered by some model. For example, if we wanted to create a cave in some terrain, we would need a hole in the terrain, but we would also need a model for the interior of the cave, and where the cave meets the terrain the cave's model would cover the edges of the terrain's hole, so it does not matter if the edges are jagged or smooth.

[mrDIMAS]

Rapier has HeightField::set_cell_status method, which allows you to remove a cell from simulation entirely by setting HeightFieldCellStatus::CELL_REMOVED status. In Fyrox, go to make_heightfield method and modify the height map there.

As for jagged edges - that's purely game designer's head ache, they could simply put some rocks at the cave's entrance to hide those jagged edges.

[b-guild]

How should we determine which cells to remove? I have been studying the CDLOD paper and it describes features that could add interesting improvements to the current Fyrox implementation, including a far more memory-efficient quadtree and the ability to stream terrain data.

I suggest that we implement the memory-efficient quadtree and add a hashmap to associate some of the nodes of the tree with SurfaceData that represents holes in the terrain. There would optionally be SurfaceData to represent holes at multiple levels of detail, and when the LOD gets too low the hole can disappear.

I suggest that there should be a way for objects to easily know the LOD of the surrounding terrain so that they can update their rendering to account for this. In particular, if a cave's hole in the terrain disappears below a certain LOD, then the model that represents the cave should respond by covering the opening with blackness so the player cannot see the terrain where the cave should be.

[mrDIMAS]

How should we determine which cells to remove?

By projecting the hole mask onto the height map of course, both are just rectangular images. There could be mismatch in size, but that's trivial to bypass.

As for mesh-based holes - I wouldn't do it, because it is overengineering. Shader based approach is very simple and does the thing.

As for optimization and stuff - I don't mind.

[b-guild]

That makes sense, but it means that the visible hole will not align perfectly with the collider hole, since the shader-based hole will naturally have rounded edges due to interpolating the mask while the collider hole will be turning whole cells off. So long as the edges of the holes are always covered by other objects, that should not be a problem during gameplay, but it might be a problem during editing if we cannot see the actual shape of the collider. The collider may need to be visualized somehow while editing holes.

Edit: On second thought, I've just been assuming that the mask would be interpolated, but it should be possible to write the shader so that the mask is not interpolated, so that the mask effectively turns on or off whole cells in the shader just like Rapier does in the collider. Again, I'm no expert in shaders, so I'm not clear on exactly how that would be implemented, but it seems like it might be the best solution.

[mrDIMAS]

The collider may need to be visualized somehow while editing holes.

It already can be visualized by turning on Show Physics setting in the editor settings.

I'm not clear on exactly how that would be implemented

Just set filtration modes of the holes texture to Nearest and that's pretty much it.