Captions/subtitles can be slow to appear in live playlist

[Frenzie]

What version of Hls.js are you using?

v1.0.5

What browser and OS (including versions) are you using?

Linux (Ubuntu 21.04, Debian 10), Windows 10, macOS 10.4.
Firefox 78, 88.
Chrome 91.
Edge 92.
Vivaldi 3.8.

Test stream:

Demo page here.

Configuration:

If a configuration setting exists that positively affects this problem I unfortunately haven't found it. So just the default.

{
  "debug": true,
  "enableWorker": true,
  "lowLatencyMode": true,
  "backBufferLength": 90
}

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client on https://hls-js.netlify.com/demo and not just on my page

• The issue occurs in the latest client on https://hls-js-dev.netlify.com/demo and not just on my page

• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

1. Please provide clear steps to reproduce your problem

Let the video play for 3 minutes without selecting a caption. Then if you select a caption from the menu it can take a very long time (20-30 seconds) before they are displayed.

2. If the bug is intermittent, give a rough frequency

While the issue isn't strictly limited to this scenario, it seems to be a reliable method of reproducing it.

Expected behavior

A near-instantaneous switch to subtitles, within seconds.

Actual behavior

It can take more than 30 seconds.

Console output

hls.js-console.log

[robwalch]

Time to display subtitle cues after selecting a subtitle track initially is the time it takes to load the track playlist, and the first fragment plus any time it takes for the browser to display active cue(s) in the new TextTrack. Measure this in the dev tools to identify which parts add up to your perceived delay, and then provide these details and a sample stream as part of the issue.

[Frenzie]

The sample stream is included above.

[robwalch]

Check parseWebVTT to make sure your subtitle times are being parsed correctly. Your VTT times should be relative to MPEGTS 0. Hls.js uses initPTS as the starting PTS time of the stream, and then plots VTT cues based on this on the content of your WebVTT files.

[robwalch]

Also check the demo "Timeline" tab to make sure that when you enable a subtitle track that HLS.js is aligning your VTT playlist with the live media playlist correctly. If it is not it could be loading the wrong segments.

HLS.js does not align VTT tracks based on PTS. It depends on ProgramDateTime to align subtitle tracks. MediaSequenceNumber is not used because the spec does not require alignment based on SN.

[robwalch]

@Frenzie, I tried your stream in Safari and it seemed to align the VTT playlists when enabled with the main playlist. I assume it's using SN to align. It's hard to tell as it could just be aligning the playlist starts, and then if the updates are off by 1, then you'd see the same 10 delay as described in this issue (or subs 10 seconds early depending on when the playlists requested and the response sent related to them updating).

Can you provide more information about the origin of the stream? Why live VTT is being delivered this way without PDT to align the content?

[robwalch]

While I have opened #3993 that could address this use case, I still question whether it's safe. It seems better for streams without PDT, but if a stream has a default track it would load with the main manifest on start, and then could only be off by 1 target duration - but then if SN are not aligned (as they are not required to be) we could end up moving the subs playlist way off from where it belongs. So, for this reason, I may close or modify #3993 to just align the playlist based on the end or start and make HLS.js load more of the VTT files to make sure we cover loading the VTT file that covers the current play time (this seems is my best guess at what is happening in Safari).

Note that (mis-)alignment in this case does not change where the captions are placed. The timing of the cues is always correct. The issue is when they are loaded, and that can be 1 TARGETDURATION off when the playlist alignment is off by 1.

[robwalch]

So, for this reason, I may close or modify #3993 to just align the playlist based on the end or start and make HLS.js load more of the VTT files to make sure we cover loading the VTT file that covers the current play time

Done! Hope you don't mind all the "rubber-ducking" comments.

[mtoczko]

Hi @robwalch,
I was looking at the stream and noticed this behavior:
after about 2 minutes of playback, the NL subtitles automatically start.

[Frenzie]

@robwalch

Can you provide more information about the origin of the stream? Why live VTT is being delivered this way without PDT to align the content?

This is simply what ffmpeg outputs, even with the program_date_time option. NB As you can see it does work on the audio and video playlists. However, a quickly hacked ffmpeg to add PDT didn't seem to make much if any difference in the observed behavior.

make HLS.js load more of the VTT files to make sure we cover loading the VTT file that covers the current play time (this seems is my best guess at what is happening in Safari).

I agree that sounds like a great option. The subtitle fragment downloading is/was quite stingy compared to audio and video for example and yet it's just a few marginal bytes. The difference already seems to be night and day! :-)

@mtoczko That's new in 1.0.5. It doesn't occur in 1.0.4. Apologies for not noticing since a minute and a half or so is long enough to observe the behavior I reported in the OP, but the longer you wait, the more certain it is that it occurs, so I rounded it up to 3 to make sure it could be reproduced. My testing in 1.0.5 has been more limited.

[robwalch]

after about 2 minutes of playback, the NL subtitles automatically start.

It looks like the changes in v1.0.5 that remove cues from the back-buffer, trigger activation of the tracks:
removeCuesInRange sets track.mode to "hidden" so that it can read and remove cues. This triggers a "change" event on the TextTrack. The subtitle-controller sees this as an external track change and activates the track.

[robwalch]

224bf76 fixes the regression in v1.0.5 that would activate the last subtitle track on buffer flush when all subs are off. There are a couple of inelegant ways you could work around it, but I'll just cut another patch as subtitle cue clearing was an important feature in the last patch and I missed this regression.

[Frenzie]

@robwalch
The patches in the PR seem to break the output from the ffmpeg executable hacked to output a PDT.

Without EXT-X-PROGRAM-DATE:

https://hlslive-origindev01-wout.cdn03.rambla.be/main/bbb/master.m3u8

With EXT-X-PROGRAM-DATE and rounded durations:

https://hlslive-origindev01-wout.cdn03.rambla.be/main/bbb2/master.m3u8

[robwalch]

@Frenzie The PDT in your subtitle playlist is roughly 60 seconds higher than the PDT in your main playlist. So when the two playlists are synced on PDT, the subs are moved a minute in advance of the current time and none of them are in range for loading. It would be better to not add PDT if it's not going to be aligned between playlists. I will not put a fix in for this as it is working as designed.

Before the update, it would have only worked as poorly as the steam without PDT where the playlists would be off by the amount of time that is spent streaming before loading a track for the first time. The reason you still get captions eventually in that scenario is because the last segment is still loaded if no others are in range when the subs playlist is behind the main one.

Your new PDT example hacks newer date times onto the subs, moving it ahead of main.

[robwalch]

@mtoczko and @Frenzie Please take a look at #3993 demo build https://deploy-preview-3993--hls-js-dev.netlify.app/demo/ and let me know if it addresses this issue or if you see any other subtitle related regressions. Thanks!

[Frenzie]

Indeed, you're right! That's well spotted. My bad, it must be accidentally taking the times from the next playlist. I'd be inclined to argue it's a minor regression since it worked previously in spite of that, but no matter.

The original issue I reported as well as the automatic subtitle selection issue are solved with #3993, and I haven't noticed anything else that isn't working as expected.

[Frenzie]

PS The https://hlslive-origindev01-wout.cdn03.rambla.be/main/bbb2/master.m3u8 test stream should be fixed now. It doesn't seem to align correctly on 1.0.5 vanilla, but it works great on #3993.

[robwalch]

Thanks @Frenzie,

There was no alignment based on PDT in v1.0.5 for subtitles. That was an incorrect assumption on my part. Tye code was there for audio tracks only, prior to the new changes.