How to set total (i.e., constant/target/planned) duration of the video that is being streamed (live-like VoD) in the video player? #6073

Andrew15-5 · 2024-01-01T21:16:47Z

Andrew15-5
Jan 1, 2024

I don't know where to ask this, but since browsers can't play HLS videos natively (without JS shenanigans) and I was able to stream a video using this library, I think this is a good place to ask.

Context

I have a video stored somewhere in a (server's) FS for now and in a DB later (PostgreSQL's large objects). This video was uploaded by a user (when I'll set up the DBMS), so it can have almost any format (including almost any codecs for video/audio tracks). The user's video now must be playable in any webview (browser, on any platform), so I have to convert it to something like AVC + AAC which are widely supported codecs (in the web). For this matter, I tried using ffmpeg but it didn't work (only with mpv player; see this), so I tried to find some more sources of how to use ffmpeg for real-time video file conversion (+ streaming). Then I have found this. And then I modified (and fixed) it like so:

FFmpeg command

  ffmpeg -i input.mkv \
    -filter_complex \
    "[0:v]split=3[v1][v2][v3]; \
    [v1]copy[v1out]; [v2]scale=w=1280:h=720[v2out]; [v3]scale=w=640:h=360[v3out]" \
    -map '[v1out]' -c:v:0 libx264 -x264-params 'nal-hrd=cbr:force-cfr=1' -b:v:0 5M -maxrate:v:0 5M -minrate:v:0 5M -bufsize:v:0 10M -preset slow -g 48 -sc_threshold 0 -keyint_min 48 \
    -map '[v2out]' -c:v:1 libx264 -x264-params 'nal-hrd=cbr:force-cfr=1' -b:v:1 3M -maxrate:v:1 3M -minrate:v:1 3M -bufsize:v:1 3M -preset slow -g 48 -sc_threshold 0 -keyint_min 48 \
    -map '[v3out]' -c:v:2 libx264 -x264-params 'nal-hrd=cbr:force-cfr=1' -b:v:2 1M -maxrate:v:2 1M -minrate:v:2 1M -bufsize:v:2 1M -preset slow -g 48 -sc_threshold 0 -keyint_min 48 \
    -map a:0 -c:a:0 aac -b:a:0 96k -ac 2 \
    -map a:0 -c:a:1 aac -b:a:1 96k -ac 2 \
    -map a:0 -c:a:2 aac -b:a:2 48k -ac 2 \
    -f hls \
    -hls_time 2 \
    -hls_list_size 2 \
    -hls_flags independent_segments \
    -hls_segment_type mpegts \
    -hls_segment_filename 'stream_%v/data%02d.ts' \
    -master_pl_name 'master.m3u8' \
    -var_stream_map 'v:0,a:0 v:1,a:1 v:2,a:2' 'stream_%v/stream.m3u8'

(I putted it into .justfile under default recipe.) I'm not fluent in FFmpeg, so I didn't change anything (to minimize the example command). Then I created an index.html using the example JS code from here:

index.html

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>HLS video streaming</title>
  </head>
  <body>
    <script src="https://cdn.jsdelivr.net/npm/hls.js@1"></script>
    <video id="video" width="50%" height="50%" controls></video>
    <script>
      var video = document.getElementById("video");
      if (Hls.isSupported()) {
        var hls = new Hls({
          debug: true,
          // liveDurationInfinity: true,
        });
        hls.loadSource("/master.m3u8");
        hls.attachMedia(video);
        hls.on(Hls.Events.MEDIA_ATTACHED, function () {
          // video.muted = true;
          video.play();
        });
      } else if (video.canPlayType("application/vnd.apple.mpegurl")) {
        video.src = "/master.m3u8";
        video.addEventListener("canplay", function () {
          video.play();
        });
      }
    </script>
  </body>
</html>

Then I used serve npm package CLI to serve the local dir (serve .) on a localhost:3000. Now I run the ffmpeg command (using j for just) in a shell in the PWD where index.html and input.mkv are present. Then I hit the play button on the video in the loaded localhost tab. And after a second or two it starts playing the video that is being converted (created) in real time. So I achieved "video streaming" (I also need to figure out how to enable back subtitle file creation and how to use/show subtitles in the video player).

I've checked https://www.rfc-editor.org/rfc/rfc8216 and https://github.com/video-dev/hls.js/blob/master/docs/API.md for "how to set the total video duration" so that in the video player it will show the actual duration of the whole video (not the sum of the already downloaded segments). But changing different DURATION tags didn't affect anything and in this library's API doc I only found this: https://github.com/video-dev/hls.js/blob/master/docs/API.md#livedurationinfinity. This removes the total duration altogether, so it only shows current time since the first received segment.

One fundamental problem I have is that I chose the "live" ffmpeg command instead of "VoD" variant. This means that if I start loading/fetching video data too late then it will start playback not from the first segment and if I start too late, then it will fail (if I removed the "temporary" files and new ones haven't been created yet). But if I instead pick the "VoD" command, then it will only create the playlist files (M3U8) after all the segments were created. With this approach, I will have everything already converted, and the full duration will be available and shown in the video player (I assume). But the client then has to wait as they first click the play button, then wait for the full conversion and only then the playback can be started. So I need to find a middle ground:

the conversion must start "on-demand" (when loading the page with video or when clicking the play button);
the playback must start immediately (while conversion isn't finished yet);
the total video duration must be known when the video playback is started (can be extracted from the input.mkv);
the playback must start from the first segment.

I really need guidance as this topic is pretty heavy (in "terms" and "knowledge") and I'm not a pro at it. Maybe there is another and better approach to stream videos on demand that I don't know of?

P.S. I'm using Firefox 119. I tried this in private tab, and it did work at least twice. Then I tried it again, and now it shows logs about loading new stuff, but there is nothing playing (it shows gray background with the spinner). Clearing any sort of data didn't help. I then tried normal tab, and suddenly it started working again. Does anyone know how this can be? Update: after I killed ffmpeg it now shows 3 or 4 seconds long (segment) video in both normal and private tab. Very strange indeed.

robwalch · 2024-01-07T18:50:30Z

robwalch
Jan 7, 2024
Maintainer

One fundamental problem I have is that I chose the "live" ffmpeg command instead of "VoD" variant.

Right. This is an on-demand encoding issue, not solved by the client library (hls.js). Duration is based on the HLS Media Playlist and not set arbitrarily by the client.

Only a VOD Playlist will give you a finite duration in HLS.js. I don't know of any open-source solutions to producing a VOD playlist where segments can be encoded on the fly, ideally in parallel and prioritized when requested by the client. This takes some coordination between encoder and file server.

I don't think having a (ffmpeg live) Live->VOD HLS Playlist (where duration is unknown until encoding ended) and telling the client what the expected duration is would help unless you want to constrain playback to the time range that has been encoded. If that is what you want, you’ll need to build that application logic around HLS.js. You would need to get, deliver, and handle that duration value from your server somehow with custom logic. It's not a feature or flag you can turn on with ffmeg or HLS.js as far as I know.

Another option is to set users expectations rather that solve on the fly encoding. Show a notification that video is being encoded when encountering a Live Playlist; The final duration will remain unknown until encoding is complete and they can see the progress and watch from the beginning in real-time.

2 replies

Andrew15-5 Jan 7, 2024
Author

to build that application logic around HLS.js

Does this mean I need to modify this library to fit my needs? Because:

It's not a feature or flag you can turn on with ffmeg or HLS.js as far as I know.

Show a notification that video is being encoded

Hmm, even though this can be an option in theory, in reality it will be inconvenient for the user to not be able to interact with timeline at all, except for pausing the video.

I have checked a bit about Plex and Jellyfin, and I've noticed that they do have the exact same problem: transcode the video on-the-fly and still be able to fully interact with the timeline on the client side. The trick is that they fully control server and client side, so they can do virtually anything to get the desired result.

I've checked Jellyfin's logs, which show that it indeed uses ffmpeg to perform any conversions. Moreover, it also uses HLS with .ts and .m3u8 files. And it starts creating TS files when I start playing the video (MKV + AVC + AC3) and it shows the total duration. And if I jump to uncached timestamp then it will start a new ffmpeg process (different offset and stuff like this, but the logs are very long, and it also contains URL which is less helpful than CLI command). So it is implementable, but not necessarily in (pure) HTML5.

I also searched similar questions, and I now know that it's not great when you have a bunch of ffmpeg processes for different users running simultaneously. The "easy fix" is to store already in the universal format and stream it to any client natively (without transcoding). The downside for this is if file uploading is much quicker than the transcoding time (which probably always be the case, unless the upload speed is very slow), this will mean that the user have to wait for "some time" until they can finally watch what they've just uploaded. I thought of some in-between solution, like lazy transcoding, which will then replace the original video and so only 1 transcoding will happen per video (max). But if a user reloads the page, this will cancel the transcoding, and now you have a lot more problems. Or maybe 2 parallel jobs: transcode fully in the background and transcode on-the-fly if user starts watching before the full background transcoding has finished. This is like the best approach I can come up with.

robwalch Jan 7, 2024
Maintainer

Does this mean I need to modify this library to fit my needs?

Not in this case. HLS.js handles HLS parsing and streaming. Your application can still interact with the HTMLMediaElement and SourceBuffers independently. There are also numerous options for creating custom video controls and UI outside this library.

Hmm, even though this can be an option in theory, in reality it will be inconvenient for the user to not be able to interact with timeline at all, except for pausing the video.

Seeking is allowed within live events (within the segments published to the Playlist).

...

It sounds like you have some good leads and an understanding of the constraints with scaling. Good luck! If you have questions specific to HLS.js, let me know.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to set total (i.e., constant/target/planned) duration of the video that is being streamed (live-like VoD) in the video player? #6073

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

How to set total (i.e., constant/target/planned) duration of the video that is being streamed (live-like VoD) in the video player? #6073

Andrew15-5 Jan 1, 2024

Replies: 1 comment · 2 replies

robwalch Jan 7, 2024 Maintainer

Andrew15-5 Jan 7, 2024 Author

robwalch Jan 7, 2024 Maintainer

Andrew15-5
Jan 1, 2024

Replies: 1 comment 2 replies

robwalch
Jan 7, 2024
Maintainer

Andrew15-5 Jan 7, 2024
Author

robwalch Jan 7, 2024
Maintainer