A Long Hike In The Dark: Making Debugging OpenGL Less Scary

Greetings traveler. We all know the pain you’re experiencing. You wanted to make something great, you did all the steps, yet the screen is still black– no, white– no, it’s just wrong. Sit, rest, but don’t despair, because this isn’t a wall. It’s a step. It’s not a step we like, and none should be proud of it, but it is a step we all take again and again.

Debugging render-related things is really a long hike in the dark. It can be very time consuming, feel like a puzzle, a riddle, a very unfair Dark Souls game, and a hundred other things. Most of all, it makes you feel dull, and once we solve the problems that torture us so, we forget to document it – the problem is done, let’s forget it as soon as possible!

Who are your best friends on this hike? Is there such a thing? Of course! Although it takes time to meet and greet them, and know how to use them, there are many such friends. Let’s talk about some. As a sidenote, most of what I do these days is Rust-oriented, but for OpenGL, most of it is simply the same as in C. I’ll also be using the gl crate directly, as I’ve had problems with glium, glow and glutin. Not big problems, mind you, but there was some unexpected stuff that made me think going native might remove one level of uncertainty. I’m making a window in sdl2 and then getting a raw gl context and working with it. Here’s that setup, more-or-less:

let sdl = sdl2::init().unwrap();
let video = sdl.video().unwrap();
let window = video
    .window("COZ!", config.size.0, config.size.1)
    .opengl()
    .build()
    .unwrap();
let gl_context = window.gl_create_context().unwrap();
gl::load_with(|s| video.gl_get_proc_address(s) as *const _);

With this, we have everything we need and we’ll use things from the gl:: package directly!

Reactive Debug Callbacks

Before we even start, make sure you have your tools. OpenGL has debugging capabilities, and we need to build on that. If you just use them as are, they print things like this:

[2023-12-12T00:42:32Z DEBUG] DEBUG CALLBACK!
 source = 33350,
 type = 33360,
 id = 131218,
 severity = 37191,
 message = "Program/shader state performance warning: Vertex shader in program 1 is being recompiled based on GL state."

I’m afraid this doesn’t mean much to me, and from what I see I can’t even be sure if this is just a notification, warning, error or… worse? We want something better. However, let’s first see how to get even this! The trick is to use gl::DebugMessageCallback, which is not easy to read itself:

gl::DebugMessageCallback(Option<fn(u32, u32, u32, u32, i32, *const i8, *mut std::ffi::c_void)>, *mut std::ffi::c_void);

If you’re anything like me, you never want to read or write this, especially in Rust (yes it is unsafe), but it is what it is and we have to, so let’s see what this is about. Reading the Khronos docs tells us a bit more: the first argument, inside the option is this function pointer:

pub extern "system" fn debug_callback(
    debug_source: u32,
    debug_type: u32,
    debug_id: u32,
    debug_severity: u32,
    debug_length: i32,
    debug_message: *const i8,
    user_param: *mut std::ffi::c_void,
) {}

The second argument is the actual user_param – we can use this to move something from the outside of the debug callback into it (and will mostly not do so now). We’ll be writing this function. The first alarming thing is that most of the things we receive are u32… Let’s start with the most obvious one: there’s an debug_message. Now, if you’re new to the C/C++ interop world, the *const i8 might scare you, but this is something you can apply a recipe to. Here’s how to turn that into a normal Rust string:

let raw_string_ptr: *const i8 = debug_message;
let c_str = unsafe { CStr::from_ptr(raw_string_ptr) };
let debug_string = c_str.to_str().expect("Failed to convert CStr to str");

For starters, we can just print this and call it a day, but as you saw up there – “Program/shader state performance warning: Vertex shader in program 1 is being recompiled based on GL state.” isn’t really too telling. Warning? Prophecy? We just don’t know.

The other arguments are mostly something called GLenum – an enum that has many different things all patched into the same whole. We need to distinguish between these, and I’m here to tell you I did it so you don’t have to. Let’s extract the behaviour type, for example:

let type_str = match debug_type {
    gl::DEBUG_TYPE_ERROR => "Error",
    gl::DEBUG_TYPE_DEPRECATED_BEHAVIOR => "Deprecated Behavior",
    gl::DEBUG_TYPE_UNDEFINED_BEHAVIOR => "Undefined Behavior",
    gl::DEBUG_TYPE_PORTABILITY => "Portability",
    gl::DEBUG_TYPE_PERFORMANCE => "Performance",
    gl::DEBUG_TYPE_MARKER => "Marker",
    gl::DEBUG_TYPE_PUSH_GROUP => "Push Group",
    gl::DEBUG_TYPE_POP_GROUP => "Pop Group",
    gl::DEBUG_TYPE_OTHER => "Other",
    _ => "Unknown",
};

If you wonder how you find this out – it’s basically this gif.

Ride to Minas Tirith, go through the docs, go through them some more. Now that we have this info, we can (arbitrarily) decide what to do with these message types:

let print_fn = match debug_type {
    gl::DEBUG_TYPE_ERROR => print_err,
    gl::DEBUG_TYPE_UNDEFINED_BEHAVIOR => print_err,
    gl::DEBUG_TYPE_DEPRECATED_BEHAVIOR => print_warn,
    gl::DEBUG_TYPE_PORTABILITY => print_warn,
    gl::DEBUG_TYPE_PERFORMANCE => print_info,
    gl::DEBUG_TYPE_MARKER => print_debug,
    gl::DEBUG_TYPE_PUSH_GROUP => print_debug,
    gl::DEBUG_TYPE_POP_GROUP => print_debug,
    _ => print_info,
};

These print_* functions are very simple and they wrap around the log crate macros but can easily also wrap around a simple println!, here’s one of them as an example:

use log::{error};


fn print_err(s: &str) {
    error!("{}", s) // might have been a println!
}

We can now actually make a bit of a difference and pass the string we retrieved as a message into the print_fn function to make a color difference at least. It now prints errors like this and warnings like this. We now know that the message above is just a notification.

We can do the same for other enums, like source and severity:

let source_str = match debug_source {
    gl::DEBUG_SOURCE_API => "API",
    gl::DEBUG_SOURCE_WINDOW_SYSTEM => "Window System",
    gl::DEBUG_SOURCE_SHADER_COMPILER => "Shader Compiler",
    gl::DEBUG_SOURCE_THIRD_PARTY => "Third Party",
    gl::DEBUG_SOURCE_APPLICATION => "Application",
    _ => "Unknown",
};

let severity_str = match debug_severity {
    gl::DEBUG_SEVERITY_HIGH => "High",
    gl::DEBUG_SEVERITY_MEDIUM => "Medium",
    gl::DEBUG_SEVERITY_LOW => "Low",
    gl::DEBUG_SEVERITY_NOTIFICATION => "Notification",
    _ => "Unknown",
};

Once you’re done with pretty much all the arguments there, you can make a nice format out of them and print_fn them:

print_fn(format!(
    "DEBUG CALLBACK!\n\tsource = {},\n\ttype = {},\n\tid = {},\n\tseverity = {},\n\tmessage = {:?}\n",
    source_str, 
    type_str, 
    debug_id, 
    severity_str, 
    debug_string).as_str());

Once we have this function all set, we need to call it. Go to your main code, just under where we made the OpenGL context and do a simple couple of calls:

unsafe {
    gl::Enable(gl::DEBUG_OUTPUT) };
    gl::DebugMessageCallback(Some(debug_callback), std::ptr::null_mut::<GLvoid>());
}

With this, our first line of defense is up: we have reactive debugging on. Whenever something happens – and I really mean WHENEVER – we’re getting info about it. The good thing? We’re using log and probably env_logger, so we can filter these messages!

Direct Engagement

But sometimes, this just isn’t good enough: sometimes things happen too fast, and the OpenGL reaction isn’t good enough — it comes too late and we can’t really tell where exactly it’s coming from. In this case, we need to bring a knife to the knife fight and have a much more direct way of attacking the problem. Introducing check_gl_error:

pub fn check_gl_error(marker: &str) {
    let error_code = unsafe { gl::GetError() };

    if error_code != gl::NO_ERROR {
        let error_string = match error_code {
            gl::INVALID_ENUM => "Invalid Enum",
            gl::INVALID_VALUE => "Invalid Value",
            gl::INVALID_OPERATION => "Invalid Operation",
            gl::STACK_OVERFLOW => "Stack Overflow",
            gl::STACK_UNDERFLOW => "Stack Underflow",
            gl::OUT_OF_MEMORY => "Out of Memory",
            _ => "Unknown Error",
        };

        error!("OpenGL Error at {}: {}", marker, error_string);
    }
}

The good thing about this one? It’s pretty short and to the point. The bad thing? It’s needed after every call if you get into the weeds.

Not even kidding. Put this everywhere if you’re in a pinch. It will serve as a guide for sure.

check_gl_error gives you a nice way to see where exactly something fails, and leads to building a utility belt of sorts. check_gl_error says that BufferData VBO failed. Let’s now look at the same message from the reactive debug callback thing we set up earlier. Oh! It’s a message about the buffer not being bound. Cool, let’s see why that is. Now we’re talking about maybe 50 lines of code instead of 500.

This is also where the white-and-red Khronos pages become a dear friend. Here’s one about BufferData. Looks frightening at first, but scroll down and to the part marked “Errors“.

It’s usually not more than 10, and you can directly reference the error from the debug callback.

Once you get here, you’ll already feel at home with OpenGL – most of the surprising things are somewhere past you. The fact that you ran faster than the dragon, however, doesn’t mean that the goblins around you aren’t dangerous.

Unsafe, Pointers, Asserts

We’re in unsafe land through-and-through. I like the idea of higher-level abstractions like glow but after trying it, it was grinding against my intuition gained from experience. What I mean here is that I write a piece of code, it looks like it should work, and then it crashes. It wouldn’t crash in C. The order of operations seems right. The bindings seem okay. Writing it in pure gl works. So, no. Thank you, I’ll maybe make a small example and a pull request.

One of the most common problems I’ve run into is becoming relaxed. Ah, I’ll just copy paste this cast into *const u8 here too, it’ll work. It compiles. It doesn’t work.

I found an example online of someone suggesting this is how we generate a vertex array, for example:

let vao: gl::types::GLuint = 0;
gl::GenVertexArrays(1, vao as *mut GLuint);
check_gl_error("GenVertexArrays");

It looks right, if you close off all the screaming Rust senses you might have developed. It feels like it should be the equivalent of the C version, but it’s not. Here’s the correct one:

let mut vao: gl::types::GLuint = 0;
gl::GenVertexArrays(1, &mut vao as *mut GLuint);
check_gl_error("GenVertexArrays");

The differences are really small but they mean everything. If it’s hard to compare, vao needs to be mut, and we need to pass in a &mut vao instead of just vao. Now, it works. There’s no error here, so we need to make sure we are steadfast: we need to assert whatever we believe should hold. As one of my senior friends from Ubisoft used to say: don’t be afraid to crash your programs for good reason.

let mut vao: gl::types::GLuint = 0;
gl::GenVertexArrays(1, &mut vao as *mut GLuint);
check_gl_error("GenVertexArrays");
assert!(vao != 0);

This goes all over the place: vaos, vbos, shaders, shader programs – whatever you see can be asserted, assert it. Not doing so right away is just waiting to bite you later.

Safely Ubiquitous [u8]

If you get a buffer of floats, it’s pretty certain that the function you’re passing it to probably needs a byte array in OpenGL. It might need you to tell it that it’s floats inside as another parameter, but you’ll need a lot of byte arrays. Here’s how to get those using a snappy little function:


pub fn to_u8_buffer<T>(buffer: &[T]) -> Vec<u8> {
    unsafe {
        core::slice::from_raw_parts(
            buffer.as_ptr() as *const u8, 
            std::mem::size_of_val(buffer)).to_owned()
    }
}

I love to load my buffers interlaced completely, so that instead of having a buffer with positions, and then one with normals, and then one with textures, I have one with (position, normal, texture) tuples. The good thing about this approach is that this tuple resembles a struct, which means we can make it into a struct, and then pass a full vector of those struct elements in directly. If we have something like…

#[repr(C)]
#[derive(Debug, Clone, Copy)]
pub struct MeshVertex {
    pub position: [f32; 3],
    pub normal: [f32; 3],
    pub tex_coords: [f32; 2],
}

We can feed this into a data buffer simply by doing a most controversial:

let vertices_u8 = to_u8_buffer::<MeshVertex>(&vertices);
gl::BufferData(
    gl::ARRAY_BUFFER,
    vertices_u8.len() as isize,
    vertices_u8.as_ptr() as *const gl::types::GLvoid,
    gl::STATIC_DRAW,
);

The bad side of this kind of loading, as many of you know, is setting up the attributes for the data.

Attributes All The Way Down

See, every single different data point you want to have about your buffers is called an attribute and attributes need to be enumerated and measured to produce results (and be loaded into a shader). What does this mean for us? Well, we need to call these two functions for each:

gl::EnableVertexAttribArray(_);

gl::VertexAttribPointer(_, _, _, _, _, _);

The underscores are there to show you where you can go wrong. Let’s elaborate. The first place in both is the index of the attribute. In our case, position will be 0, normal will be 1, and texture will be 2. This is quite a normal way to do things but NOTHING is set in stone. You can have whatever wacky setup you want, and that’s both good and bad. For starters, use the defaults, please.

The second and third arguments to VertexAttribPointer are the size and type. For example, 3, gl::FLOAT or maybe 2, gl::UNSIGNED_BYTE. This is telling us what kind of data is going to be separated and sent to the shader. This is where it gets hard to know what we know.

The next argument is a OpenGL boolean, so a gl::TRUE or gl::FALSE value called “normalized”. You can read about it in the docs; I never needed to set it true, so for me, it’s just hardcoded to gl::FALSE. Please tell me why this is wrong, if it ever is.

The last two are called stride and offset, and this is most of your problems. Trust you me, you can lose hours here, because errors will not be obvious, except that they will draw weird if at all. I bet that years of developer time have been sacrificed into these arguments, but they aren’t that hard, as long as you remember what they mean:

Stride measures how much we need to jump to get to the next tuple from this one. Offset measures how far we are inside the tuple. Both of these are ALWAYS IN BYTES. Don’t believe anyone who tells you otherwise.

Let’s see the settings for the structure above, where we have 3 floats for position, 3 floats for normals, and 2 floats for the texture coordinates per tuple. First, let’s convert that to bytes with 4 bytes per float: 12 bytes for position, 12 bytes for normals, 8 bytes for texture coordinates. To get stride, and this will be the FINAL calculation for stride, add all of these together. So, stride is 12 + 12 + 8 = 32. That’s it. This is how much we need to jump from the start of one vertex position, for example, to get to the same place in the next vertex. Stride should be the same across all attributes when we have them interlaced.

Lastly, offset is going to be different for each attribute: it’s how many bytes we walked within the tuple to get to this place. So, let’s see how many it is: for positions, as it’s at the start of the tuple, we walked 0. That’s the offset of the position attribute. Then position took 12 bytes, so normals are offset by 12. After that, texture coordinates start at 8 more, so 20. In code, this is:

gl::VertexAttribPointer(0, 3, gl::FLOAT, gl::FALSE, 32, 0);
gl::VertexAttribPointer(1, 3, gl::FLOAT, gl::FALSE, 32, 12);
gl::VertexAttribPointer(2, 2, gl::FLOAT, gl::FALSE, 32, 20);

Read this carefully. The first argument is an index, it just goes up. The second argument is how many of the next one (floats) we need for this attribute (so 3, 3, 2), and the last two are the sum of all byte sizes, and the relative placement in bytes of the current attribute within the tuple (so 0, 0+12, 0+12+8).

This code is probably best ever written only once, but if you need to change it, I’ll be brave to suggest using build.rs to automatize it.

Dangerous-Code Generation

To be clear, it’s not the code generation that is dangerous. We just want the dangerous code to be generated. If someone were to tell me what attributes I need, I could generate the calls above and let the compiler worry about ranges and offsets and alike. It’s good that Rust lets us do that!

We can start by making a struct that holds our attribute descriptors, something like the following:

#[derive(Serialize, Deserialize)]
struct DataAttribute {
    index: u32,
    size: u32,
}

#[derive(Serialize, Deserialize)]
struct DataConfiguration {
    pub attributes: HashMap<String, DataAttribute>,
}

I always work with floats in shaders (don’t we all?), so for me, only the index and size are important — everything else can be calculated. I’m also using serde here for serialization. Next up, we need the ron crate to make a data packet out of this struct: in a text file, somewhere, have your wanted configuration:

DataConfiguration(
    attributes: {
        "Position": DataAttribute(index: 0, size: 3),
        "Normal": DataAttribute(index: 1, size: 3),
        "TexCoord": DataAttribute(index: 2, size: 2),
    }
}

We then have a function within our build.rs script (if it doesn’t exist, create it in the root of your project — cargo will call it whenever building before the build itself):

fn generate_graphics_configurations(path: &Path) {
    // open folder with all configurations
        // go through each of them and parse them with ron
        // make a string builder or equivalent
        // pack lines of code inside defining the dangerous operations we need
        // save to source folder
}

The actual fun part from here looks like this for me:

builder.append("\t\tunsafe {\n".to_string());
builder.append("\t\t\tgl::BindVertexArray(vao);\n\n".to_string());

let stride: u32 = attributes.iter().map(|a| a.1.size * 4).sum();
let mut offset = 0;

for (name, attrib) in attributes {
    builder.append(format!(
        "\t\t\tgl::EnableVertexAttribArray({}); \/\/ {}\n",
            attrib.index, name.to_uppercase()));

    builder.append(format!(
        "\t\t\tgl::VertexAttribPointer({}, {}, gl::FLOAT, gl::FALSE, {}, {} as *const GLvoid);\n",
            attrib.index,
            attrib.size,
            stride,
            offset
));

    offset += attrib.size * 4;
}

builder.append("\t\t}\n\n".to_string());

To make the build.rs script work, you need a main() in it, so I have something like this, doing a couple of things:


fn main() {
    let binding = std::env::var_os("OUT_DIR").unwrap().into_string().unwrap();
    if let Some(out_dir) = binding.split("target").next() {
        let path = Path::new(&out_dir);


        generate_graphics_configurations(path);
        // ... other generators //
    }

    println!("cargo:rerun-if-changed=build.rs");
    println!("cargo:rerun-if-changed=assets");
}

With this, I don’t really think about a possibility of misalignment anymore, freeing up more time for… solving a blank empty screen. That’s the other half of GL bugs. How do we solve those? Mostly renderdoc, but usually, you messed up your matrices. What do I mean there? Well…

Uniform Matrix Hell

Have you watched the Matrix? Maybe you remember it the same as I: the Matrix is a bad thing. It’s a hell in its own way, one where we’re being utilized for a faceless and literally larger-than-life operation that feeds on us and tells us what to love and what to eat, and how what tastes. I’m bringing this up because it was quite a slap when The Game Awards cohost, Sydnee Goodman, told us that an animated character couldn’t come to the TGA because we’re “still not in the Matrix, sadly”.

Maybe we’re not in the matrix, but uniform matrices sure are in OpenGL. And you need them. If you want a camera in the game, you need them. If you want your meshes to be able to turn, you need them. And they’re a pain to debug, so better NOT debug them at all. Make sure you’re doing it right off the bat. This is my vertex shader, for example:

#version 430 core

layout(location = 0) in vec3 in_position;
layout(location = 1) in vec3 in_normal;
layout(location = 2) in vec2 in_tex_coords;

uniform mat4 MVP; // <---- here we are

out vec3 out_normal;
out vec2 out_tex_coords;

void main() {
    out_normal = in_normal;
    out_tex_coords = in_tex_coords;
    gl_Position = MVP * vec4(in_position, 1.0);
}

To create a matrix like this, you need some library like nalgebra or similar, and it’s not hard per se:

fn create_mvp_matrix(window: &Window, _frame: u64) -> Matrix4<f32> {
    let model = Matrix4::identity();
    let view = Matrix4::look_at_rh(
        &Point3::new(2.0, 2.0, 2.0),
        &Point3::new(0.0, 0.0, 0.0),
        &Vector3::new(0.0, 1.0, 0.0),
    );
    let projection = Matrix4::new_perspective(
        window.size().0 as f32 / window.size().1 as f32,
        45.0,
        1.0,
        100.0,
    );

    projection * view * model
}

This is fine. Now let’s get this to that uniform matrix variable in the shader!

let mvp_matrix = create_mvp_matrix(&window, time.frame_count());
let mvp_matrix_location =
 gl::GetUniformLocation(basic.program, "MVP".as_ptr() as *const i8);
gl::UniformMatrix4fv(
    mvp_matrix_location,
    1, gl::FALSE,
    mvp_matrix.as_ptr() as *const GLfloat,
);

This is the code of a person who skimmed this whole blogpost. Don’t be greedy. Look carefully. This code doesn’t work, but it doesn’t tell you anything. There’s no warning, there’s no traces, there’s — nothing! It seems to be okay, really. The only problem is that when you use it, your graphics, well, doesn’t show up. It literally isn’t there.

There’s a sacred dance attached to this: maybe it’s too big (change view matrix from 2.0 to 1.0). Maybe it’s too small (change view matrix from 1.0 to 1000.0). Maybe I’m in it (change view matrix so that it’s -10 on the Z axis). Maybe it’s the other way (change view matrix so that it’s 10 on the Z axis). There’s a whole other verse after that too, it’s a real headbanger.

In short, there are several things that could have tripped you here. Let’s just mark them for now.

let mvp_matrix = create_mvp_matrix(&window, time.frame_count());

let mvp_matrix_location =
 gl::GetUniformLocation(basic.program, "MVP".as_ptr() as *const i8);
                                                      // ^ are you sure?
// where's the check_gl_error?
gl::UniformMatrix4fv(
    mvp_matrix_location,
    // ^ you never asserted it's not -1
    1, gl::FALSE,
    mvp_matrix.as_ptr() as *const GLfloat,
    // ^ hmmm
);
// where's the check_gl_error?

Back to the old block with this code. The problem here starts from the very beginning of this post: our string can’t just be loaded into a *const i8. Always remember that for this, you need a CString. It’s something very NOT Rust, so it has to be C. Drill this in, it’s good for you.

let name = CString::new("MVP").unwrap();
let mvp_matrix_location =
 gl::GetUniformLocation(basic.program, name.as_ptr());

Next up, read the Khronos docs. Seriously. Location can return -1, and the quickest way to figure it out is to just assert!

assert!(mvp_matrix_location != -1);

I’ll skip putting listings for the check_gl_error but it should be a thing you do WITHOUT thinking. If you call a OpenGL function from the gl:: namespace, just do it. If you’re afraid that things will get slow, remember that it’s better to have slow graphics that works than fast graphics that doesn’t.

The last really important thing is to be sure that your matrix can, when expressed as a pointer, be equivalent to a *const GLfloat. If you’re not sure, check if it can just be turned into a float array, then pass it through as_ptr. If rust_analyzer says that you can shorten it, please do, but don’t just blindly trust pointers.

The same points from before can be applied here: if you have a way to statically assert things about this matrix transformation, do it. If you can automatize this somehow (maybe you need this in a limited number of situations and you can describe them up front), do it carefully.

As an aside, one of the most telling reasons I knew my uniforms were off was actually RenderDoc. When checking it, pick weird values – don’t use a clear view matrix where everything is 1.0 or 0.0: do 30 degrees here, -11 there. Usually, it will throw all these unreadable numbers from the matrix at us, but this time around, if the matrix isn’t being sent — no matter how much you try, you won’t be surprised by a multitude of awful numbers! Remember how strange it feels not to be scared by a matrix. It’s a very primitive part of the brain that controls this, the one we don’t really use anymore: the one that sends chills whenever “the forest got really quiet all of a sudden”. You had a good evolutionary fight up until now, use everything you have!

Conclusion

In the end, we’re really all in this together. If nothing here helped, check the basics: many of the simpler tutorials do things in a certain order, so check the order of creation of things. Check that you use genBuffers and not createBuffers (this got me…). Check that you are passing sizes correctly — read on them in the docs. And when you feel like giving up, come ask. There’s a really cool couple of places on Reddit where people help, but also leave a comment here if you find it and need help. There is no bigger pain than being alone in the dark and knowing that your precious time is ticking. Sometimes, it turns out we just need a rubber ducky. Drop a line, let’s talk!


Posted

in

by

Tags:

Comments

One response to “A Long Hike In The Dark: Making Debugging OpenGL Less Scary”

  1. […] array for interleaved rendering, defining the layout of our attributes (some words on that in my previous blogpost) ourselves and making sure it coordinates nicely with what we read from the source. So, we would […]

Leave a Reply

Your email address will not be published. Required fields are marked *