Dynamically Enabling/Disabling Audio Track Download Based on User Interaction in Shaka Player

[Amud]

Question :

I am working with a demuxed CMAF HLS stream that includes separate .cmfv video segments and .cmfa audio segments. I've noticed that about 40% of users initially watch the stream in a muted state. To conserve bandwidth, I am interested in dynamically controlling the download of the audio track based on user interactions.
Is there a way to enable or disable the audio track download dynamically during runtime in Shaka Player? Specifically, if a user starts with the video muted and later unmutes it, can the player be configured to start downloading the audio track from the next segment onwards at that moment? I know that disabling audio at the player's instance creation through manifest configuration is possible, but I am looking for a method to re-enable it dynamically during runtime. Any workaround approch is also welcome

Shaka Player version : v4.8.3 (latest)
Browser : Google chome (124.0.6367.202)
OS : Mac 14.0

[joeyparrish]

Is there a way to enable or disable the audio track download dynamically during runtime in Shaka Player? Specifically, if a user starts with the video muted and later unmutes it, can the player be configured to start downloading the audio track from the next segment onwards at that moment?

Not currently. It is possible in the MediaSource API, but Shaka Player was not designed for this. I would also anticipate that some devices (smart TVs, I'm looking at you) might not support removing and adding SourceBuffers dynamically.

To make this happen, I would guess we would need:

1. An automated test that adds and removes SourceBuffers dynamically, which we could run in the lab to prove which platforms would support this
2. A change to the design of MediaSourceEngine in Shaka Player to allow the audio buffer to be added or removed dynamically

However, I would ask you to consider the encoding ladder for your content and the resolutions/bitrates used by your users who are muting. Are people watching 1080p video at 2Mbps with an unused audio stream consuming only 128kbps? Is it worth it to re-engineer the player to save 5.88% bandwidth? (128 / (2048 + 128))

If it's worth it for you, we would welcome your designs and PRs, and we would be happy to guide you and review your work.

[Amud]

Thank you for your reply.

My use case is more of a social media app featuring a vertical video feed. Our user studies indicate that most viewers prefer watching videos with subtitles in mute state while multitasking, such as during commutes or at work, and only enable sound for particularly interesting content. These users are also conscious of their data usage. Furthermore, minimizing API calls is crucial. For CMAF demuxed audio, preventing audio initialization and segment calls can reduce the time to the first frame.

For Android users, I've implemented a solution using ExoPlayer where I disable audio during player initialization by overriding the track selection implementation. If a user decides to enable audio later, I reload the manifest with an overridden 'TIME-OFFSET' field set to 'PRECISE' = true, ensuring they return to the precise moment post-reload. I am able to handle UI transitions smoothly since I am using a surface view for rendering player frames that retain the previous frame.

I'm curious if there's a way to adapt this approach to Shaka Player for web users, allowing similar functionality and user experience.

[joeyparrish]

Sure. You can modify MediaSourceEngine (and StreamingEngine, which calls into MediaSourceEngine) to use web API methods like addSourceBuffer and removeSourceBuffer to dynamically add and remove audio. I see no reason it shouldn't be technically feasible.

But we don't have that in our design today. You'd have to make some design changes. We would be happy to have your contributions!

The closest you have today is something like:

player.configure('manifest.disableAudio', true);
await player.load(content);

// ... later ...

// Save current position
const currentTime = videoElement.currentTime;
// Unload
await player.unload()
// Enable audio
player.configure('manifest.disableAudio', false);
// Load at saved position
await player.load(content, currentTime);

It's not going to be incredibly smooth. addSourceBuffer/removeSourceBuffer should be much smoother.

[Amud]

The suggested approach is working fine but is not incredibly smooth, as you suggested. If you can point me to the areas of changes in the player, I can give it a try with the addSourceBuffer/removeSourceBuffer approach.
PS: I am new to Shaka Player but have fair experience working with ExoPlayer.

[avelad]

Maybe the preload API can help here? (see: https://shaka-player-demo.appspot.com/docs/api/tutorial-preload.html)

[Amud]

I've tried a solution using the precache API to handle audio tracks based on mute state
The approach preloads the audio track in a dummy player resulting in disk caches of it (due to availability of cache-control header in response), and then configures and reloads the main player to avoid loading the audio track from the network again. The experience is now smoother than before.

`

// Save the current playback time
const currentTime = this.video.currentTime;
try {
    // Configure a dummy player to preload the audio track and restrict track switching capabilities
    const dummyPlayer = new this.shakaScript.Player();
    dummyPlayer.configure('abr.restrictions.maxHeight', 1);
    dummyPlayer.configure('abr.enable', false);
    dummyPlayer.configure('manifest.disableAudio', this.currentMuteState);

    // Preload the audio track
    const loader = await dummyPlayer.preload(this.source, currentTime);
    await loader.waitForFinish();
    if (!loader) return;

    // Unload the main player and reconfigure it based on the mute state and restrict track switching capabilities
    await this.player.unload();
    this.player.configure('abr.restrictions.maxHeight', 1);
    this.player.configure('abr.enable', false);
    this.player.configure('manifest.disableAudio', this.currentMuteState);

    // Reload the main player with the new configuration and resume playback
    await this.player.load(this.source, currentTime);
    await this.video.play();

    // Destroy the dummy player
    loader.destroy();
    dummyPlayer.destroy();

} catch (error) {
    console.error('Failed to update mute state and reload the player:', error);
}

`

Do you have any suggestions or see any potential issues with this approach? Is there anything that can be improved?