Registers NVIDIA Micromap Extensions

[NBickford-NV]

These extensions allow glTF assets to store and transfer information about micromaps – a compact and performant representation for ray tracing and rasterizing highly detailed geometry and opacity – and micromeshes, which use micromaps.

This merge request adds the following extensions under the NV prefix:

• NV_micromaps
  • Defines what micromaps are in glTF, and adds an array of micromaps, like images, to extensions.NV_micromaps. This creates a common foundation for NV_opacity_micromap, NV_displacement_micromap, and NV_attribute_micromap.
• NV_opacity_micromap
  • Allows primitives to express opacity micromeshes using opacity micromaps.
• NV_displacement_micromap
  • Allows primitives to express displacement micromeshes using displacement micromaps.
• NV_attribute_micromap
  • Further extends micromeshes to represent arbitrary barycentric-sampled data for microvertices or faces of microtriangles. For instance, this can be used to store microvertex normals or colors.
• NV_micromap_tooling
  • Allows tools that use glTF to interchange data about how to generate micromaps. For instance, the remesher in the NVIDIA Displacement Micromap Toolkit uses this extension to pass a subdivision level per mesh triangle that the DMM baker should use. This is because the remesher knows how the base mesh was generated, and can use that to provide higher-quality information to the baker.

In the dependency graph below, an edge from A to B denotes that A depends on B:

graph BT 
    NV_opacity_micromap --> |Requires| NV_micromaps 
    NV_displacement_micromap --> |Requires| NV_micromaps 
    NV_micromap_tooling --> |References definitions from| NV_displacement_micromap 
    NV_attribute_micromap --> |Requires| NV_micromaps 

Loading

This merge request includes specifications, JSON schemas, and examples for these extensions. Implementations are available in the NVIDIA Micro-Mesh SDKs and Simplygon SDK 10.1 (blog post, release notes). Additionally, a fork of cgltf with an implementation of these extensions will be available shortly after this merge request is posted.

Here are a few design notes about these extensions relevant to this merge request:

Micromap formats

The NV_micromaps extension references the BARY file format for barycentric-sampled data, but also allows using custom micromap formats. Referencing types in BARY hopefully makes it easier for users to identify how glTF properties match BARY properties and rendering API inputs.

BARY files use the model/vnd.bary Media Type (https://www.iana.org/assignments/media-types/model/vnd.bary).

NV_opacity_micromap, NV_displacement_micromap, and NV_attribute_micromap allow reusing micromap files across glTF files, and providing or overriding certain properties of micromaps with data from glTF accessors.

Graphics API compatibility

In most cases, opacity and displacement micromesh data can be passed directly to the graphics API, such as by using the Vulkan VK_EXT_opacity_micromap or VK_NV_displacement_micromap extensions. I presented a talk about how to use this with Vulkan and DirectX with NVAPI at GTC 2023 online.

Half floats

Since VK_NV_displacement_micromap can store directions and direction bounds with both half floats and floats, VK_NV_displacement_micromap adds a half float accessor component type corresponding to the OpenGL HALF_FLOAT type. We've intended to word this so that it should not conflict with other specs if they also define half floats.

Why separate extensions instead of one?

Separating NV_opacity_micromap and NV_displacement_micromap allows renderers that only support one or the other to quickly filter files.

OMMs (Opacity Micromaps) and DMMs (Displaced Micromeshes) are hardware-accelerated in NVIDIA Ada Lovelace GPUs. However, arbitrary attribute micromeshes (NV_attribute_micromap) usually require renderers to read from their data buffers in their shader code, so that is separated in case renderers only support OMMs and DMMs.

Finally, NV_micromap_tooling allows files to specify partial micromesh data – for instance, only the per-vertex directions – to transfer this data to tools. Partially specified data does not define a full micromesh (i.e. it would not be valid if NV_displacement_micromap was added without values), so we have an extension dedicated for tools that need to write intermediate glTF files.

Base mesh fallbacks

One neat thing about these extensions is that if a renderer does not support them, it will usually fall back to the base mesh, which is well-defined and visually consistent. NV_displacement_micromap has direction bounds in part to support this: for instance, they make it possible for an artist to specify a base mesh, a high-res reference mesh, and bake a micromesh with displacements both above and below the surface of the base mesh. When a renderer supports NV_displacement_micromap, the file will look like the reference mesh, and when it does not, it will look much like the artist-defined base mesh. (Direction bounds can be baked into vertex positions and directions to save space at the expense of altering the base mesh.)

RFC 2119 definitions

The specifications use MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL as in RFC 2119. I’ve noticed most vendor extensions don’t do this, though; hopefully this is OK, since the glTF 2.0 spec uses RFC 2119.

Thanks!

[NBickford-NV]

model/vnd.bary is now a registered IANA Media Type (https://www.iana.org/assignments/media-types/model/vnd.bary), so I've edited the description to reflect this.

[NBickford-NV]

The Vulkan VK_NV_displacement_micromap extension has been released in Vulkan 1.3.245; I've updated the pull request description to link to it. Thanks!