Support mobile browsers

[PixelPartner]

No description provided.

[brianchirls]

Yes please. Or better yet, is it possible to feature-detect rather than whitelist/blacklist by user agent?

[brianchirls]

Having read further, I think I understand this better now. It looks like Omnitone uses createMediaElementSource to get its audio data from a video or audio element. This feature is broken in a number of browsers, and it is impossible to feature detect because the method exists on an AudioContext object.

• Seems to be broken in Mobile Safari, at least for iOS 9.3. The bug ticket for it is resolved, but it's not working for me in iOS Simulator.
• This was broken in Chrome/Android for a long time, but it looks like the bug ticket was just resolved a few days ago. The Chrome/Android media pipeline is being rewritten anyway, so keep an eye out for changes there in the next version or two.
• Technically, this works in desktop Safari, but I disabled it on a project with long-ish audio files because it was a bit crashy, and I was getting popping sounds.

I suppose, in theory, it's possible to write a some code that would test this by loading a very small audio file (maybe even a dataURI, though sometimes those cause their own problems) and reading the audio samples from that.

So the next question is: is it possible to create a decoder that loads from an ArrayBuffer rather than a media element, as a fallback?

[hoch]

@brianchirls

You diagnosis is pretty close, but here are the reasons behind this coarse UA gating. I am blocking the entire mobile browser due to the incomplete implementation.

• Safari iOS cannot decode the multichannel audio at all. FOA ambisonics requires a 4-channel audio stream.
• Chrome iOS runs WebKit inside, not Blink rendering engine. You'll have the same issue with Safari iOS.
• Chrome for Android M51 does not support the multichannel audio decoding in MediaElement yet. However, the 4-channel audio stream can be decoded properly with the UnifiedMediaPipeline flag enabled.
• Chrome for Android M53 (which is the dev channel ATM) supports the multichannel audio decoding by default, thus Omnitone can be used there without fiddling flags.

I do not believe that the feature detection or sniffing should be a part of the Omnitone library, so now the demo player is handling that part. I agree that it is not exhaustive and have a plan to fix it in the near future. Your idea on using dataURI sounds neat, so I will try that.

[danrossi]

@hoch. I've tested on my Android 4.3 Samsung S3. This device has given me quite a bit of trouble so not a good performance test.

Firefox stable / nightly

If the feature is enabled at all. Doesn't matter what mode is chosen, I'm suffering severe cpu issue and the audio is stuttering. It is also affecting webgl rendering, it's dropping frames. If I don't enable the feature Firefox will decode the VP9 with vorbis audio. It won't decode natively the AAC audio.

This is a shame because Firefox works better on my device than Chrome does.

I'm unsure if the decoding feature is working when panning the camera around because the performance is not good and audio cutting out.

Chrome.

On Chrome stable/ Dev. I cant get webgl rendering video at all with heaps of problems, it's a write off on my device. Audio plays back fine however when panning the camera there is no audible change. Chrome dev has the pipeline enabled.

If there is a work around for IOS that could be interesting also.

I'm presuming the point of this feature would be for cardboard / gear VR viewing and orientation controls on IOS. I don't have access to gear properly yet to check WebVR. When I do I will test this in the Gear VR browser.

[hoch]

@danrossi

Firefox stable / nightly

I haven't tested FF mobile for this project. So I am not sure what the current status is and appreciate the report you wrote here. Please share your experience and file a bug report to Bugzilla if necessary.

On Chrome stable/ Dev. I cant get webgl rendering video at all with heaps of problems, it's a write off on my device. Audio plays back fine however when panning the camera there is no audible change. Chrome dev has the pipeline enabled.

I don't have an S3, but Chrome Dev channel on Nexus 5 shows decent performance. No glitching and smooth visual. However, somehow Nexus 5X and 6P produce glitches in audio, and I do not think this is an Omnitone or WebAudio problem. I have to diagnose this issue thoroughly and solve before M53 comes to the stable. It might be UMP, Chrome media or Android.

If there is a work around for IOS that could be interesting also.

To that end, iOS must support the multichannel audio decoding in iOS Safari.

I don't have access to gear properly yet to check WebVR. When I do I will test this in the Gear VR browser.

Sounds good. Looking forward to your report!

[danrossi]

Firefox Android stable is ok with the default audio demo. There is no video example.

I'll try and create a very basic fiddler demo with your demo videos then. without webgl going. Just to check it performs ok.

If it still does it with that, what should I file in mozilla bugzilla in regards to audio api ?

I don't have content of my own but I have to figure out from scratch how to produce it hopefully from a surround mix. So author a surround mix to 4 channel ambisonics. Run it through the required tools etc.

Chrome on Android regardless of the webgl rendering I could not hear any change in panning. It was ok on Firefox and Chrome on OSX. I could hear changes in the rotation. Is it definitely working on Chrome Android ?

I know the chromium people only test on Nexus. I know my device is end of life so not a good test even for minimum requirements. I've yet to get access to an S7 permanently, but I can get one temporarily. I don't have access to a Nexus but only want to concentrate on Samsung devices for gear VR.

I know stable Chrome works perfect with webgl video on the S7 almost too perfect compared to the frame dropping on older devices. It's a graphics display issue but Firefox works so no idea why Chrome is different.

Hopefully at least I can get Firefox working with orientation rotation.

As for IOS. two channel audio has to be offered there it seems. Doesn't sound like that will change for a while. I'll do some research there , it can't even play multiple audio streams at the same time. It doesn't even have MediaSource support. Very lame indeed.

I just checked the demo against Safari and it is not working either. Will try and debug that. This multi channel demo does work however and I know Safari OSX supports Dolby Ac3.

https://www2.iis.fraunhofer.de/AAC/multichannel.html

[danrossi]

There is more. Chrome dev won't display the webm only the mp4 file so there is a source issue and possible why I get black frames in webgl. Both versions are dropping frames. So have to re-encode these for testing purposes.

Chrome stable will play back the audio without omnitone enabled and then obviously won't work with it enabled.

Firefox seems to play both the mp4 and webm but stuttering. I read its some kind of throttling because of samplerates different to the system samplerate ? .

I've just fixed an npm dependancy issue for webpack. I've created a barebones video demo with no webgl just to confirm audio works. I'll commit these if that is ok.

I'll re-encode these demos if that is ok and host on my cloudfront. And hopefully they fair better ?

[danrossi]

Ok I've run FFprobe through the demo video. High profiles don't work on mobile at all. The audio bandwidth is 512kb also.

I know Android Chrome complains with Mpeg Dash with high h264 profiles. I'm going to produce a couple of downscaled videos to properly check Android functions for now at least in both mp4 and webm.

Then check Mpeg Dash formats although it seems Youtube has activated audio api functionality too now and it uses mpeg dash of course.

[danrossi]

Firefox stable plays fine with this file. So perhaps its a video performance issue. I'll try and make a 720p and see what happens.

Firefox can play the mp4 also but it's more glitchy. Firefox desktop won't decode h264 so needs to be vp9 / vorbis.

Removed my previous comment. I've been playing around with the flags in Dev quite a bit to get webgl working. A reinstall of Chrome Dev now works with both the 360p webm and mp4 ! No frame dropping anymore ..

Will test rendering through webgl now.

[danrossi]

Sorry to keep spamming you.

Chrome stable was ok with the scaled down webm rendering the video. But after a reinstall, audio is fine but no video again.

Chrome Dev video is rendering finally but audio cuts out and frames are dropping without Omnitone enabled. With Omnitone enabled it cuts out completely due to performance issues.

So lets just say it definitely works with Chrome but only on newer devices.

Firefox works with the scaled down webm, no severe audio stuttering. Its a bit glitchy when rotating the audio but not sure if I can hear a change ?.

I'll keep working on Firefox for now so I can at least test rotation is working with WebVR mode.

[hoch]

@danrossi You can't transcode the video and upload it to a public data storage. Jaunt VR allowed us to use the video only inside of the demo player. (it is clearly stated in the demo gallery - in case you missed it) Sorry, but I had to remove the URL from the comment.

Could you please take it down?

[hoch]

@danrossi Thank you so much for your exhaustive testing on mobile browsers. I will read up your comments and try to summarize/evaluate the situation again.

[danrossi]

@hoch.

I apologise for that I didn't realise there was a licensing issue. It is a very noisy soundtrack anyway, not the best. It does help with catching frame dropping and audio sync issues though.

I have reached out to try and get a better demo to continue testing mobile.

I might try and create a surround "atmos" mix to ambisonics and use an panorama image for a demo video also.

[hoch]

I have tested "Resonance" video on Chrome Android Canary 56 and I confirmed Omnitone/demo player work flawlessly. I think it can graduate the 'debug mode' status.

@AdamSomers I am not sure why "Fuerza Imprevista" cannot be played on Android Chrome. It seems like the codec is not supported. Do you any clue?

[danrossi]

@hoch I believe they have fixed the pipeline issues. WebGL is rendering properly with video textures in Chrome stable now so this would be working too.

[hoch]

@danrossi I don't think that was the issue. The video worked fine even before M53. The audio was glitchy on some phones and my guess was the rendering quantum size.

[danrossi]

@hoch I did get the glitchy frame dropping on the audio but reducing the video resolution fixed that in Firefox, Chrome was ok. You mentioned something about media pipeline before, and that was related to any kind of rendering in Chrome on my device. For 6 months webgl was not rendering video now it is thankfully.

[hoch]

I recently did some research on cross-browser/cross-platform support of Omnitone:
https://github.com/GoogleChrome/omnitone/wiki/Cross-browser-and-Mobile-support

As of April 2017, sadly iOS Safari still does't not support the decoding of multichannel audio stream (> 2ch). Any kind of meaningful spatial audio involves with more than stereo stream, thus iOS Safari cannot deliver it at all.

[danrossi]

An absolute shame. I wonder if a native hook via cordova would be possible but maybe not. The amount of work arounds required for IOS and Safari is incredible. I spent 3 months figuring out the HLS WebGL fix after they refused to even respond to the ticket.

[brianchirls]

Revisiting this old issue now that some time has passed...

I see that createMediaElementSource is no longer required and that Omnitone will now accept an audio node. So presumably, even if the browser doesn't properly decode a four-channel audio file, we could load the channels separately and merge them, right?

According to the Cross browser and Mobile support document, Safari isn't passing unit tests because of a method/function that won't return a Promise. It's an old document, but Safari is still using the old, prefixed, non-conforming Web Audio implementation so it probably isn't any better.

I can recommend standardized-audio-context as a shim. It may not solve every problem, but it should at least provide a conforming API. I'm using it on another project, and it works great. However, I'd advise loading it conditionally in example code rather than compiling it into the published library build to keep the size down.

[hoch]

I see that createMediaElementSource is no longer required and that Omnitone will now accept an audio node. So presumably, even if the browser doesn't properly decode a four-channel audio file, we could load the channels separately and merge them, right?

Yes. That was the intent of the change.

I can recommend standardized-audio-context as a shim. It may not solve every problem, but it should at least provide a conforming API. I'm using it on another project, and it works great. However, I'd advise loading it conditionally in example code rather than compiling it into the published library build to keep the size down.

Yes, it is a great shim. However, I would like to minimize the dependency as much as possible. That library seems to be active and well-managed, so it is trivial to include it as a project needed.

I feel bad that we don't have enough documents on the current state of browser/mobile support. Hope I can come back and revisit the issue again in the near future. With that said, I would appreciate if you can share your findings here!

[brianchirls]

Yes. That was the intent of the change.

Awesome! Exactly what I'd asked for, delivered. I love when that happens.

Yes, it is a great shim. However, I would like to minimize the dependency as much as possible. That library seems to be active and well-managed, so it is trivial to include it as a project needed.

Yeah, that's kinda what I had in mind. I've only needed the shim for a scheduling quirk that didn't have an easy workaround but doesn't seem to apply here. I suppose you could use it for API conformance, but that's a convenience issue and probably not worth the dependency. But if you run into other problems, that shim is probably a good place to start.

re: findings, the examples seem to be working just fine on my old Android phone. I don't have an iPhone, but the 360 video demos don't work so well on desktop Safari. That's likely an issue with WebGL performance, though.

Personally, I'm satisfied with this issue and consider it solved, at least to my needs. Though, it's not a bad idea to keep improving documentation and tests.

Thanks again!

[danrossi]

I have access to an S8 and Ipad with IOS 11 beta. I can revisit and run some tests. I think it's a stretch to try and provide backward compatibility. Trying to test webvr on my old S3 was aweful and buggy in chrome.

Just provide a static compatibility method.