How to extract a frame from HtmlVideoElement ?

[astariul]

I'm trying to use the webcam from the browser and pass the video feed to a Rust function.

As a template, I use the example of Google mediapipe for hand landmarker : https://codepen.io/mediapipe-preview/pen/gOKBGPN

In that example, they have a HtmlVideoElement in JS, then pass it to the code that does the hand detection :

const video = document.getElementById("webcam") as HTMLVideoElement;

[...]

results = handLandmarker.detectForVideo(video, startTimeMs);

I'd like to do exactly the same, in Rust.

I created my Rust function :

use wasm_bindgen::prelude::*;
use web_sys::{console, HtmlVideoElement};

#[wasm_bindgen]
pub fn my_fn(video: JsValue) -> Result<(), JsValue> {
    match video.dyn_into::<HtmlVideoElement>() {
        Ok(video) => {
            console::log_1(&format!("test").into());

            Ok(())
        }
        Err(_) => Err(JsValue::from_str("argument not a HtmlVideoElement")),
    }
}

I can successfully call the Rust function from JS, and it works because I see test printed in the console.

But now for the next step, which is extracting the pixel values for the current frame from the HtmlVideoElement, I have no idea how to do that...

I look through the documentation, but I can't find anything on how to extract the raw data of HtmlVideoElement... I know it's definitely possible, because mediapipe does it (HandLandmarker.detectForVideo())

I just don't know how to do it in Rust.
Any help would be greatly appreciated !

[63b8e7lmk1jv9c0ovus4hqkcasqgp0ed]

I don't think the full implementation is relevant to wasm-bindgen beyond that there's some missing functions on that element. You have some options though.

• You could write bindings for the missing functions on that element
• You could write bindings for the other library that you mentioned
• You could find other JS solutions, and write bindings for them.

[astariul]

Thanks for the answer.

After more research, I could make it work.

The solution is not to use the HtmlVideoElement but the canvas.

So in the JS code, I draw the frame from the HtmlVideoElement into the canvas, and then I retrieve the frame from the canvas. Then I just pass this image to my Rust fucntion:

canvasCtx.drawImage(video, -canvasElement.width, 0, canvasElement.width, canvasElement.height);
const img = canvasCtx.getImageData(0, 0, canvasElement.width, canvasElement.height);
my_fn(img);

The Rust code :

use wasm_bindgen::prelude::*;
use web_sys::{console, ImageData};

#[wasm_bindgen]
pub fn my_fn(frame: ImageData) {
    let pixels = frame.data();
    let size = pixels.len();
    let w = frame.width();
    let h = frame.height();
    console::log_1(&format!("-> {w} {h} = {size}").into());
}