Ways for using more data types: int64_t, uint64_t, double, half-float...

[cesss]

I really love glTF, and I consider it one of the hugest achievements in CG history. But I'm facing now an obstacle in my use case, and it's that I'd like it to store values in more data types.

I tend to believe that this shouldn't affect JSON sections, as JSON is very permissive in number datatypes, and you could basically throw whatever data type you wish in a JSON value. But it affects buffers and accessors, because the allowed data types are limited by the glTF 2.0 spec.

I'd basically wish to have all the C standard data types at my fingertips: int8_t, uint8_t, int16_t, uint16_t, int32_t, uint32_t, int64_t, uint64_t, float, and double. And considering how useful 16bit half-floats are in OpenGL, I'd like to have half floats too.

From that set, glTF buffers already support 32-bit float, as well as the 8 bit and 16 bit integer variants (32 bit too but only as unsigned, and only for indices IIRC -thus, you cannot define 32bit/channel images).

Now, because I'm the kind of person who likes to start doing things just right now, I'm thinking how I could do this without changing the spec. The obvious choice would be with an extension.

But I'm failing to find a way of writing a clean extension: the spec says that componentType is required in accessors. This means that if I implement my own extComponentType and put it inside the "extensions" of the accessor, I will have duplicated values of the component type. Yes, I can specify that my extComponentType overrides the standard componentType, but it looks not clean at all...

Do you think there could be a cleaner way for writing this extension?

[lexaknyazev]

glTF accessors are used to access mesh, joint, and animation data, so their types are limited to these use cases.

A custom extension may start with overriding the list of allowed componentType enum values by adding entries like GL_HALF_FLOAT, GL_INT64_ARB, etc. However, such accessors won't be usable by mesh.primitive.attributes elements as defined in the table here. To overcome this, another extension similar to KHR_mesh_quantization would be required.

On the other hand, a regular combination of buffer and bufferView is enough to store any arbitrary binary data without referencing it from spec-defined glTF objects.

[javagl]

From a bird's eye perspective, I fully agree. I think that the lack of double and long/int64 is a severe and very unfortunate limitation. It's only a matter of time until these data types will be required (or at least, generally supported) for OpenGL and other Graphics APIs. On top of that, there are many cases where glTF (or "the overall approach of glTF") is supposed to be used to store arbitrary binary data. I think it would be great if it was possible to "carve out" the buffer, bufferView, accessor part of glTF, and call it something like "The JSON Binary Data Layout Specification®™", including the missing component types.

(Of course, people would then ask for support of further data types (like int128). This could, to some extent, anticipated right from the start. But then, people might ask for custom data types, eventually making it necessary to define types explicitly, like it is done in 'Protocol Buffers' and similar approaches. But I think that much (if not all) of this could then be defined on top of the basic definition of the binary data layout. There also are proposals for extensions like EXT_structural_metadata, which faced the same challenges, and which do include very generic type definitions, including things like FLOAT64. But these extensions usually cover a specific use-case, had to work around the lack of support for such data types in glTF, and consequently, there is no small, lean "core" specification for the binary data layout itself)

So right now, the support for additional contentType values (in a form that is closer to 'core' glTF) can only be achieved by dedicated extensions, like lexaknyazev said.

[cesss]

Thanks, I didn't know about EXT_structural_metadata. It implements almost everything I had on my mind, with the exception that I guess it cannot be used for storing vertices coordinates as double, if I understand it correctly.

However, from the comment by @lexaknyazev I just learnt that extensions can modify the list of entries allowed for a given field (such as componentType for example). But I feel confused about that: Aren't parsers supposed to parse successfully any glTF file by just ignoring the extensions the parser doesn't support? That can be done in a clean way by just skipping the "extensions" sections that are not supported, but, if you are parsing a mesh and then you find a VEC3 of FLOAT64, this would be a parse error, wouldn't it? If affirmative, it would break the paradigm of a successful parse by skipping unsupported features. Yes, I know you can mark the extension as required, but... sometimes you might like to get the cameras from a file whose meshes you cannot read, for example.

Another problem: if your extension adds new allowable entries to componentType, how do you choose their number ID so that they don't collide with other extensions, or with future glTF releases? Does the protocol for accepting extensions take care of this?

[lexaknyazev]

Aren't parsers supposed to parse successfully any glTF file by just ignoring the extensions the parser doesn't support?

JSON Schema for accessor.componentType allows any integer as a valid value. Such accessors won't be usable by the existing ecosystem but parsers should be resilient anyway. If an accessor of a new type is used by an existing spec-defined glTF object, e.g., position attribute, the glTF asset should mark the relevant extension as required, thus heavily limiting asset's portability.

An alternative approach would be to extend accessor objects in general. The extension object would contain pointers to float64 data while the original accessor would point to standard float32 version of the same data.

Both options have their pros and cons.

how do you choose their number ID

In case of accessor.componentType, all values match OpenGL, which has enums for float16, float64, int64, and uint64.

[cesss]

Thanks a lot!!! With this I think I have everything that I need for start working. I'm going to close this issue, but if you think it should be kept open, or if you think a new one should be created as a feature request of adding support in accessors for all OpenGL types, just feel free to do so.

[spnda]

@cesss Have you finished writing the extension proposal? I'd be very interested in an extension that allows GL_HALF_FLOAT, GL_DOUBLE, GL_INT, and GL_UNSIGNED_INT64_ARB for components. I'm actually surprised that signed ints are not allowed at all yet.