feat: add safari camera encode support

[alcolmenar]

A number of things to happen to fully enable Safari support due its lack of support for Web specific APIs used:

• Determine a method to enable a method generate VideoFrame to pipe through the VideoEncoder
• Remove usage of MediaStreamTrackProcessor in camera_encoder.rs and implement method
• Determine a method encode Audio using the opus codec (Safari doesn't support AudioEncoder) most likely going to need a 3rd party library for this
• Remove usage of MediaStreamTrackerProcessor in microphone_encoder.rs and implement method
• Determine a method to decode the Audio stream in peer_decoder.rs and pipe output through the Web Audio APIs (Safari doesn't support AudioDecoder)
• Remove usage of MediaStreamTrackGenerator and implement above

Fixes #58

[darioalessandro]

looking good!! lets see what we can use for audio https://crates.io/crates/opus-oxide

[alcolmenar]

looking good!! lets see what we can use for audio https://crates.io/crates/opus-oxide

@darioalessandro looks like this package doesn't have any code in it

[darioalessandro]

Goddam! Clickbait

[alcolmenar]

Starting from https://github.com/mmig/opus-encdec, I had decided to try to use ScriptProcessorNodes to obtain Audio samples -> Encode -> Transmit -> Decode -> Audio samples, but it introduced a ton of delay and artifacts. Also ScriptProcessorNodes are deprecated for good reason. I then tried to use AudioWorklets, but the repo only had a Worklet created for the encoding side so I forked the repo here and created one. Also, I had to rebuild the js / wasm files to make sure all of the source files show up in the same file so it loads easier when adding the JS as a module. I added Google's FreeQueue library (a high performance ring buffer), but I had to change a little bit because it was using SharedArrayBuffer's as its backing and they aren't in the GlobalAudioWorkletScope.

Created an AudioWorkletCodec struct that essentially creates and interfaces with the worklet and implemented its usage in the both microphone_encoder and peer_decoder using the respective worklets.

As far as testing, I was able to test video on Safari. However, for Audio, after a ton of debugging and trying to figure out what was going on, Safari disables a bunch of Web API features on newer versions if you aren't connecting to a HTTPS endpoint so I wasn't able test Audio fully. Audio from newer Safari browsers to Chrome works, but not very well. I'm not sure how much of that is due to the non-HTTPS endpoint.

Video from Chrome to Safari iOS works. Safari iOS has HTTPS issues as well. Safari iOS to Chrome Video has issues. It seems like it is due to this.). Suspended AudioContexts were common in Chrome using AudioBufferSources, but switching to AudioWorklets made it less common.

[alcolmenar]

What do you think about adding the forked worklet repo as a submodule to this project?

[darioalessandro]

@alcolmenar you are a rockstar!

I'll review your PR today.

To use https with safari you could use ngrok to expose your localhost in the internet and get valid https certs: https://ngrok.com/

[alcolmenar]

To use https with safari you could use ngrok to expose your localhost in the internet and get valid https certs: https://ngrok.com/

I tried that. Maybe I'm doing something wrong but I figured it wouldn't work since there are a few ports that need to be exposed and a free account only allows 1 at a time. Were you able to get a full test in using it?

[darioalessandro]

@alcolmenar I am looking at this now

[alcolmenar]

@darioalessandro how does it look?

[darioalessandro]

You are a legend my friend!!

POC is fantastic! audio sounds great!!

Video is a bit slow but I am sure that if we work together we can make it fast!!

[darioalessandro]

Hey @alcolmenar ! I noticed that there's compiled wasm both encoder/decoder.js workers? do you have the source code for these and license?

[alcolmenar]

Hey @alcolmenar ! I noticed that there's compiled wasm both encoder/decoder.js workers? do you have the source code for these and license?

I forked the opus-encdec repo here https://github.com/alcolmenar/opus-encdec/tree/audio-worklet and made changes to it to create the decoderWorker Worklet. I think I need to update the license though because I also pulled in a Ring Buffer implementation from Google labs https://github.com/GoogleChromeLabs/web-audio-samples/tree/main/src/audio-worklet/free-queue

The compiled wasm is an emscripten compilation of the c implementation of libopus and speexdsp. The license is in the repo already