[models] Instanced Drawing with Custom Shader Corrupts Drawing with Default Shader

[TheTophatDemon]

• I tested it on latest raylib version from master branch
• I checked there is no similar issue already reported
  (#2211 is related but isn't the same sort of issue).
• My code has no errors or misuse of raylib

Issue description

In my original project, I was trying to draw instanced copies of a particular mesh, and then use that same mesh to draw a non-instanced version of it with the default material. However, something about the DrawMeshInstanced(...) call seemed to corrupt the state of the renderer and made the non-instanced version not draw correctly.

This only occurs when the vertex shader of the instanced material does not declare the attributes for vertex normal and color, which can be demonstrated by replacing the example's shader with "based_lighting_instanced.vs" from the raylib examples.

Although this can be seen as a misconfiguration on the user's end, this error does not occur when instancing is not involved. Even so, an error in the instanced vertex shader should not affect rendering using the default shader as well.

Environment

Linux, Pop! OS (Ubuntu & Gnome desktop), OpenGL 4.6.0 NVIDIA 510.73.05, NVIDIA GeForce GTX 1650, NVIDIA proprietary driver version 510.73.05

Issue Screenshot

Result of the code example below:

Result of the code example, with the DrawMeshInstanced(...) line removed.

Expected result (Achieved by replacing "base_instanced_stripped.vs" with "base_lighting_instanced.vs")

Code Example

#include "raylib.h"
#include "raymath.h"

#include <stdlib.h>
#include <math.h>

#define MAX_INSTANCES  10

int main(void)
{
    const int screenWidth = 800;
    const int screenHeight = 450;

    InitWindow(screenWidth, screenHeight, "raylib [shaders] example - mesh instancing");
    SetTargetFPS(60);

    Camera camera = { 0 };
    camera.position = (Vector3){ -25.0f, 25.0f, -25.0f };
    camera.target = (Vector3){ 0.0f, 0.0f, 0.0f };
    camera.up = (Vector3){ 0.0f, 1.0f, 0.0f };
    camera.fovy = 45.0f;
    camera.projection = CAMERA_PERSPECTIVE;
    SetCameraMode(camera, CAMERA_ORBITAL);

    Mesh cube = GenMeshCube(1.0f, 1.0f, 1.0f);

    Matrix *transforms = RL_MALLOC(MAX_INSTANCES*sizeof(Matrix));

    for (int i = 0; i < MAX_INSTANCES; i++) {
        transforms[i] = MatrixTranslate(0.0f, -10.0f, 0.0f);
    }

    //Load a shader that's missing attributes for vertex color and normals.
    Shader shader = LoadShader("resources/shaders/glsl330/base_instanced_stripped.vs",
                               "resources/shaders/glsl330/base.fs");

    shader.locs[SHADER_LOC_MATRIX_MVP] = GetShaderLocation(shader, "mvp");
    shader.locs[SHADER_LOC_MATRIX_MODEL] = GetShaderLocationAttrib(shader, "instanceTransform");
    
    //Instanced material
    Material instancedMaterial = LoadMaterialDefault();
    instancedMaterial.shader = shader;
    instancedMaterial.maps[MATERIAL_MAP_DIFFUSE].color = RED;
    
    //Non instanced material
    Material normalMaterial = LoadMaterialDefault();
    normalMaterial.maps[MATERIAL_MAP_DIFFUSE].color = BLUE;

    while (!WindowShouldClose())
    {
        UpdateCamera(&camera);
        
        BeginDrawing();

            ClearBackground(RAYWHITE);

            BeginMode3D(camera);
            	//The cubes drawn before and after the instanced draw call are not rendered unless the instanced draw call is removed.
                DrawMesh(cube, normalMaterial, MatrixTranslate(-2.0f, 0.0f, 0.0f));
                
                DrawMeshInstanced(cube, instancedMaterial, transforms, MAX_INSTANCES);
                
                DrawMesh(cube, normalMaterial, MatrixTranslate(2.0f, 0.0f, 0.0f));
            EndMode3D();

        EndDrawing();
    }
	
    RL_FREE(transforms);

    CloseWindow(); 

    return 0;
}

Shader code for "base_instanced_stripped.vs":

#version 330

in vec3 vertexPosition;
in vec2 vertexTexCoord;

in mat4 instanceTransform;

uniform mat4 mvp;
uniform mat4 matNormal;

out vec3 fragPosition;
out vec2 fragTexCoord;

void main()
{
    mat4 mvpi = mvp*instanceTransform;

    fragPosition = vec3(mvpi*vec4(vertexPosition, 1.0));
    fragTexCoord = vertexTexCoord;

    gl_Position = mvpi*vec4(vertexPosition, 1.0);
}

Other shaders mentioned are from the raylib examples repository.

[raysan5]

@TheTophatDemon I think the issue could be related to some OpenGL state set by DrawMeshInstanced() that is not reseted after drawing and it "leaks" to the other draws.

[raysan5]

Just tested it with current implementation and it works ok for me.

Here it is the simplified code I tested, I used the shader in raylib.

#include "raylib.h"
#include "raymath.h"

#define RLIGHTS_IMPLEMENTATION
#include "rlights.h"

#include <stdlib.h>
#include <math.h>

#if defined(PLATFORM_DESKTOP)
    #define GLSL_VERSION            330
#else   // PLATFORM_RPI, PLATFORM_ANDROID, PLATFORM_WEB
    #define GLSL_VERSION            100
#endif

#define MAX_INSTANCES  50

//------------------------------------------------------------------------------------
// Program main entry point
//------------------------------------------------------------------------------------
int main(void)
{
    // Initialization
    //--------------------------------------------------------------------------------------
    const int screenWidth = 800;
    const int screenHeight = 450;

    InitWindow(screenWidth, screenHeight, "raylib [shaders] example - mesh instancing");

    // Define the camera to look into our 3d world
    Camera camera = { 0 };
    camera.position = (Vector3){ -125.0f, 125.0f, -125.0f };
    camera.target = (Vector3){ 0.0f, 0.0f, 0.0f };
    camera.up = (Vector3){ 0.0f, 1.0f, 0.0f };
    camera.fovy = 45.0f;
    camera.projection = CAMERA_PERSPECTIVE;

    Mesh cube = GenMeshCube(1.0f, 1.0f, 1.0f);

    Matrix *transforms = RL_MALLOC(MAX_INSTANCES*sizeof(Matrix));   // Pre-multiplied transformations passed to rlgl

    // Scatter random cubes around
    for (int i = 0; i < MAX_INSTANCES; i++)
    {
        Matrix translation = MatrixTranslate((float)GetRandomValue(-50, 50), (float)GetRandomValue(-50, 50), (float)GetRandomValue(-50, 50));
        Vector3 axis = Vector3Normalize((Vector3){ (float)GetRandomValue(0, 360), (float)GetRandomValue(0, 360), (float)GetRandomValue(0, 360) });
        float angle = (float)GetRandomValue(0, 10)*DEG2RAD;
        Matrix rotation = MatrixRotate(axis, angle);
        
        transforms[i] = MatrixMultiply(rotation, translation);
    }

    // Load shader
    Shader shader = LoadShader(TextFormat("resources/shaders/glsl%i/base_lighting_instanced.vs", GLSL_VERSION),
                               TextFormat("resources/shaders/glsl%i/lighting.fs", GLSL_VERSION));

    // Get some shader loactions
    shader.locs[SHADER_LOC_MATRIX_MVP] = GetShaderLocation(shader, "mvp");
    shader.locs[SHADER_LOC_VECTOR_VIEW] = GetShaderLocation(shader, "viewPos");
    shader.locs[SHADER_LOC_MATRIX_MODEL] = GetShaderLocationAttrib(shader, "instanceTransform");

    // Ambient light level
    int ambientLoc = GetShaderLocation(shader, "ambient");
    SetShaderValue(shader, ambientLoc, (float[4]){ 0.2f, 0.2f, 0.2f, 1.0f }, SHADER_UNIFORM_VEC4);

    CreateLight(LIGHT_DIRECTIONAL, (Vector3){ 50.0f, 50.0f, 0.0f }, Vector3Zero(), WHITE, shader);

    // NOTE: We are assigning the intancing shader to material.shader
    // to be used on mesh drawing with DrawMeshInstanced()
    Material material = LoadMaterialDefault();
    material.shader = shader;
    material.maps[MATERIAL_MAP_DIFFUSE].color = RED;

    // Non instanced material
    Material normalMaterial = LoadMaterialDefault();
    normalMaterial.maps[MATERIAL_MAP_DIFFUSE].color = BLUE;

    SetCameraMode(camera, CAMERA_ORBITAL);  // Set an orbital camera mode

    SetTargetFPS(60);                      // Set our game to run at 60 frames-per-second
    //--------------------------------------------------------------------------------------

    // Main game loop
    while (!WindowShouldClose())            // Detect window close button or ESC key
    {
        // Update
        //----------------------------------------------------------------------------------
        UpdateCamera(&camera);

        // Update the light shader with the camera view position
        float cameraPos[3] = { camera.position.x, camera.position.y, camera.position.z };
        SetShaderValue(shader, shader.locs[SHADER_LOC_VECTOR_VIEW], cameraPos, SHADER_UNIFORM_VEC3);
        //----------------------------------------------------------------------------------

        // Draw
        //----------------------------------------------------------------------------------
        BeginDrawing();

            ClearBackground(RAYWHITE);

            BeginMode3D(camera);

                DrawMesh(cube, normalMaterial, MatrixTranslate(-2.0f, 0.0f, 0.0f));

                DrawMeshInstanced(cube, material, transforms, MAX_INSTANCES);

                DrawMesh(cube, normalMaterial, MatrixTranslate(2.0f, 0.0f, 0.0f));

            EndMode3D();

            DrawFPS(10, 10);

        EndDrawing();
        //----------------------------------------------------------------------------------
    }

    // De-Initialization
    //--------------------------------------------------------------------------------------
    RL_FREE(transforms);

    CloseWindow();        // Close window and OpenGL context
    //--------------------------------------------------------------------------------------

    return 0;
}

[raysan5]

By the way, just updated the shaders_mesh_instancing, I simplified it and illustrate the simple mesh drawing before and after the instanced meshes.

[TheTophatDemon]

Thank you for the updated example. It does indeed work as intended on my machine.

However, the issue is that the bug occurs only when the shader does not include the vertex color attribute, whereas your example does include it.

If you run your example as you've posted it, but you remove all mentions of vertex color in 'lighting_instancing.vs' and 'lighting.fs', you may notice that the 'blue cubes' turn pure white and are barely visible, as in this screenshot.

This is despite the fact that a.) The blue cubes aren't even using that shader, and b.) 'lighting.fs' does not use the color attribute in any of its calculations.

[raysan5]

@TheTophatDemon I already found the issue but it could take me a while to fix it.

To DrawMesh(), I provide matDefault = LoadMaterialDefault() (that uses the internal shader used for batching buffer) and it enables vertex color attribute BUT GenMeshCube() does not generate vertex colors, so, when drawing, the color attribute is enabled in shader BUT no vertex color is provided to shader.

Worth mentioning that result is undefined behaviour. So it can randomly work depending on GPU/drivers/API state previous configurations...

I'm thinking about the proper solution to this problem...

[raysan5]

After lot of investigation I don't know yet why this issue happens. Previous statement is true but UploadMesh() already considers that possibility (a vertex attribute shader input not provided by mesh) and setups a default input value for that vertex attribute... but it does not seem to work when just providing the VAO; the attribute needs to be explicitly disabled before drawing.

Anyway, it seems to work as expected now.

[Greu]

Hello, I'm reopening this issue because I was trying to do Mesh instancing with raylib 5.5 (Awesome library btw!) on my desktop and I have the same problem.

I tried the mesh instancing example but I was always getting an empty windows.

If I'm commenting the line 127 DrawMeshInstanced(cube, matInstances, transforms, MAX_INSTANCES); then I can see the two cubes drawn by DrawMesh.

Finally I checked the code you pasted @raysan5 in this thread and I noticed a difference compared to the examples.
The following line is missing in the examples

shader.locs[SHADER_LOC_MATRIX_MODEL] = GetShaderLocationAttrib(shader, "instanceTransform")

When adding this line everything works perfectly.

Hope this help.

Edit: it looks like it's not reopening the issue automatically. I don't know if you'll get notified. If no answer I will create a new issue in a few days for visibility.
Edit2: Issue reopened. Thank you.