ndc_to_world() and viewport_to_world() methods on Camera

[asafigan]

What problem does this solve or what need does it fill?

Calculating the inverse of world_to_ndc() and world_to_viewport().

What solution would you like?

Adding ndc_to_world() and viewport_to_world() methods on Camera.

It would also be nice to have matrix versions of these methods that returns the matrix to they you can reuse the matrix when preforming multiple calculations.

What alternative(s) have you considered?

Calculating the matrix yourself.

Additional Context

I am willing to implement this

[alice-i-cecile]

I'm on board. Please feel free to open a PR.

[mockersf]

How can you project a point from ndc/viewport to world? At best it can give a ray

[asafigan]

ndc is a 3d space so you can project a 3d point. world_to_ndc() returns a Vec3 so simply providing the inverse I think makes sense. Plus I only think advanced users would know what ndc is so I think they would know what they are doing.

Viewport is a different issue. For my game, I'm only projecting with orthographic views so the Z doesn't really matter but this is a case by case thing. Maybe there should just be a method to calculate the matrix and have the user decide how they want to use it.

I see that either bevy_math nor glam provide ray types. Is there any plan to add them? When they are added what would the raycast API look like?

[MOZGIII]

Something like this maybe?

https://github.com/aevyrie/bevy_mod_raycast/blob/5949fc7f021e6900c3d14b4e6f22ef4426489441/src/primitives.rs#L192-L214

I'm using 2d game space on 0 Z plane in a 3d game, so getting the world coordinates is easy:

let t = -(Vec3::Y.dot(ray_origin)) / (Vec3::Y.dot(ray_direction));
let mut intersection = (cursor_pos_near + t * ray_direction).xz();
intersection.y *= -1.0;

But there can be all kinds of ways to go from a ray to the world coordinates...

[petereichinger]

Would something like this work?

pub fn ndc_to_world(&self, camera_transform: &GlobalTransform, ndc: Vec3) -> Option<Vec3> {
    let ndc_to_world: Mat4 =
        camera_transform.compute_matrix() * self.computed.projection_matrix.inverse();

    let world_coords = ndc_to_world.project_point3(ndc);

    if !world_coords.is_nan() {
        Some(world_coords)
    } else {
        None
    }
}

** EDIT ** This should be placed in the impl of Camera

[asafigan]

I have a branch with this exact code locally along with other functions. Implementing these functions was pretty straight forward but after implementing them I was wondering how useful they actually were. ndc_to_world seems fine but the other suggestions I made seem pretty niche. Would these be usefully to a lot of people or does it just add functions that won't be used very often?

[MOZGIII]

Well, I'm doing this:

    // Normalized device coordinate cursor position
    // from (-1, -1, -1) to (1, 1, 1).
    let cursor_ndc = (cursor_position / window_size) * 2.0 - Vec2::ONE;
    let cursor_pos_ndc_near: Vec3 = cursor_ndc.extend(-1.0);
    let cursor_pos_ndc_far: Vec3 = cursor_ndc.extend(1.0);

    query.for_each_mut(|(mut game_plane_position, camera, camera_transform)| {
        let camera_position = camera_transform.compute_matrix();
        let projection_matrix = camera.projection_matrix();

        // Use near and far ndc points to generate a ray in world space.
        // This method is more robust than using the location of the camera
        // as the start of the ray, because ortho cameras have a focal point
        // at infinity.
        let ndc_to_world: Mat4 = camera_position * projection_matrix.inverse();
        let cursor_pos_near: Vec3 = ndc_to_world.project_point3(cursor_pos_ndc_near);
        let cursor_pos_far: Vec3 = ndc_to_world.project_point3(cursor_pos_ndc_far);
        let ray_origin = cursor_pos_near;
        let ray_direction = cursor_pos_far - cursor_pos_near;

        tracing::trace!(message = "cursor raycast", ?ray_origin, ?ray_direction);

        // Calcluate the intersection with horizontal plane at zero coordinate
        // height.
        let t = -(Vec3::Y.dot(ray_origin)) / (Vec3::Y.dot(ray_direction));
        let mut intersection = (cursor_pos_near + t * ray_direction).xz();
        intersection.y *= -1.0;

        game_plane_position.coord = intersection;

        tracing::debug!(message = "cursor position", ?intersection);
    })

Here, the Mat4 is reused for multiple invocations, which wouldn't be possible with the function proposed above. So, while I am kind of doing the ndc to world conversion, I'd probably still be unable to use that function.

So, maybe we'd better think more about the API first, and/or maybe implement it as a third-party crate with a useful toolbelt of functions so we try things before touching the core API surface?

[asafigan]

It seems compute_ndc_to_world_matrix and ndc_to_world would be helpful based on my own code and @MOZGIII's code. I also see that viewport to ndc 2d transformation would be useful.

Is the definition of ndc the same across most graphics APIs? If not, it might be good to have more of these functions plus some useful constants. In the future, if the graphics back end changes then less packages have to be updated to conform to the new back end.

[petereichinger]

I modified my code and created a repo with it. It's implemented as an trait for the Camera struct. https://github.com/petereichinger/bevy_trafo

[irate-devil]

I think this can be closed since #6126 landed.