SRT fails to init on high bitrate?

[Obeyed]

Subject of the issue

Hey datarhei team, great job on the restreamer project!

I saw #695 that found some possible challenges with high bitrates. This seems to have been merged already. I did in fact experience something similar with SRT as well.

Has anyone experienced any issues now with higher bitrates?

I was trying out restreamer yesterday and got some unexpected results. Tried both the latest and dev docker image.

I haven't done proper tests to investigate more. Wanted to ask here first to sanity check first.

Your environment
Was trying restreamer in docker on a AWS m5zn.large machine:

• Ubuntu Server 22.04 LTS
• 2 CPU (Intel Xeon Scalable, up to 4.5 GHz)
• 8 GB RAM
• approx. 25 Gbps network connection

Steps to reproduce

ffmpeg -err_detect ignore_err \
  -re -i input.mp4 \
  -c:v libx264 -c:a copy -b:v ${BITRATE} \
  -f flv ${RTMP_URL}

With BITRATE=2M it seems to work without issues, but if I change to e.g. BITRATE=20M then I see drops in the RTMP connection. It's as if the rtmp server or the stream is restarted for short while before it's up again.

Moreover, I saw the following 404 status codes in the logs with the web UI open. Not sure if it's related.

ts=2024-04-17T13:29:43Z level=WARN component="HTTPS" address=":8080" client="IP_ADDRESS_HIDDEN" latency_ms=0 method="GET" path="/memfs/2e0e4217-f674-4c81-8cad-a25550c69542_output_0_0011.ts?session=NUpdxmT3A7HuitjUzgR2J8" proto="HTTP/2.0" rx_size_bytes=0 status=404 status_text="Not Found" tx_size_bytes=109 user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"

[ioppermann]

I tested it locally by pulling the datarhei/restreamer:latest image and sending a high bitrate RTMP stream to it for at least 5 minutes.

$ ffmpeg -f lavfi -re -i testsrc2=s=1980x1080:r=25 -pix_fmt yuv420p -codec:v libx264 -preset ultrafast -b:v 50000k -f flv rtmp://127.0.0.1/live/stream/streamkey
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 15.0.0 (clang-1500.3.9.4)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/6.1.1_6 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopenvino --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Input #0, lavfi, from 'testsrc2=s=1980x1080:r=25':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, yuv420p, 1980x1080 [SAR 1:1 DAR 11:6], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x1206080a0] using SAR=1/1
[libx264 @ 0x1206080a0] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x1206080a0] profile Constrained Baseline, level 5.0, 4:2:0, 8-bit
[libx264 @ 0x1206080a0] 264 - core 164 r3108 31e19f9 - H.264/MPEG-4 AVC codec - Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=18 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=abr mbtree=0 bitrate=50000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, flv, to 'rtmp://127.0.0.1/live/stream/streamkey':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0: Video: h264 ([7][0][0][0] / 0x0007), yuv420p(progressive), 1980x1080 [SAR 1:1 DAR 11:6], q=2-31, 50000 kb/s, 25 fps, 1k tbn
    Metadata:
      encoder         : Lavc60.31.102 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/50000000 buffer size: 0 vbv_delay: N/A
frame= 7746 fps= 25 q=0.0 size= 1460543kB time=00:05:09.84 bitrate=38616.0kbits/s speed=0.999x

In your setup, how long does it take to drop the connection? It doesn't seem to be an issue with Restreamer.

The 404 status code in the logs ist not related to the RTMP.

[Obeyed]

Thanks for the confirmation and details! Sounds like the issue is not with restreamer then. Will follow up here when I know more.

[Obeyed]

Finally had a chance to investigate this. Was unable to reproduce the original issue with RTMP. Using the latest restreamer docker image. I did find when it came to SRT, the initial ffmpeg inspect fails at high bitrates. I enabled the debug logs and found that the ffmpeg analyse command was failing. I believe this is due to the fact that there are no key frames present in the short duration that is analyzed to allow ffmpeg to decode the frames correctly.

Proposed solution

Increase the ffmpeg -probesize argument to 100.000.000 bytes.

There could be cases that would break with even higher bitrates (compared to what I tested with, 100M). Perhaps the probesize could be configurable in the GUI?

I don't know if this may break some other configurations? What do you think is a good approach?

Reproducable with these commands

The examples below all use the same ffmpeg config to send the SRT stream. Only thing that changes is the bitrate (-b:v).

ffmpeg -f lavfi -re -i testsrc2=s=3860x2160:r=30 -pix_fmt yuv420p \
  -c:v libx264 -preset veryslow -b:v {VARIABLE} \
  -f mpegts pipe:1 | srt-live-transmit file://con "{SRT_LINK}"

where srt-live-transmite is installed from https://github.com/Haivision/srt/releases/tag/v1.5.3

and then I analyse that same stream with this command (taken from the restreamer logs):

ffmpeg -fflags +genpts -thread_queue_size 512 \
  -probesize {PROBESIZE_BYTES} \
  -analyzeduration 5000000 \
  -i "{SRT_LINK}"

Changing the -analyzeduration didn't seem to make much difference. A higher -probesize seemed to do the trick.

Test result examples

These results are the same for all the tests. In the next paragraph I add the tests I ran and how many times they failed to inspect correctly.

Failed result example

Notice, there is no frame dimensions defined.

[h264 @ 0x555791532b00] decode_slice_header error
[h264 @ 0x555791532b00] no frame!
[h264 @ 0x555791532b00] non-existing PPS 0 referenced
    Last message repeated 1 times
[mpegts @ 0x555791505780] decoding for stream 0 failed
[mpegts @ 0x555791505780] Could not find codec parameters for stream 0 (Video: h264 ([27][0][0][0] / 0x001B), none): unspecified size
Consider increasing the value for the 'analyzeduration' (5000000) and 'probesize' (5000000) options
Input #0, mpegts, from 'srt://localhost:6000?mode=caller&transtype=live&latency=20000&streamid=030c8751-0b7f-4c09-b0f6-7c8ce7623cbb.stream,mode:request':
  Duration: N/A, start: 52.166667, bitrate: N/A
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
  Stream #0:0[0x100]: Video: h264 ([27][0][0][0] / 0x001B), none, 30 fps, 30 tbr, 90k tbn, 180k tbc
At least one output file must be specified

As far as I understand, the h264 errors reported (repeated multiple times) indicate that there isn't any key frame present for ffmpeg to decode. Once a key frame is received the errors stop.

Valid result examples

Notice the correct frame dimensions defined.

[h264 @ 0x55ce6d434b00] decode_slice_header error
[h264 @ 0x55ce6d434b00] no frame!
[h264 @ 0x55ce6d434b00] non-existing PPS 0 referenced
    Last message repeated 1 times
[mpegts @ 0x55ce6d407780] Packet corrupt (stream = 0, dts = NOPTS).
[mpegts @ 0x55ce6d407780] Packet corrupt (stream = 0, dts = NOPTS).
Input #0, mpegts, from 'srt://localhost:6000?mode=caller&transtype=live&latency=20000&streamid=030c8751-0b7f-4c09-b0f6-7c8ce7623cbb.stream,mode:request':
  Duration: N/A, start: 140.900000, bitrate: N/A
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
  Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 3860x2160 [SAR 1:1 DAR 193:108], 30 fps, 30 tbr, 90k tbn, 60 tbc

Test Runs with different bitrates

The below are tested with bitrates 10M and 100M.

Note, the below (ex. probesize=100.000.000 bytes) failed more often with the 100M bitrate compared to the 10M bitrate.

• probesize 5.000.000 (in bytes, default) - fails ~9/10 times
• probesize 10.000.000 - fails ~6/10 times
• probesize 20.000.000 - fails ~4/10 times
• probesize 50.000.000 - fails ~3/10 times
• probesize 100.000.000 - fails 0/10 times (consistent never failed for all attempts made)

[Obeyed]

It seems that it is indeed possible to manually configure the probesize etc from the edit wizard when you have an existing stream. I suppose this is the preferred approach and this should just be closed?

[ioppermann]

The issue with SRT and high bitrates is that the probesize value is not enough. When pulling a stream from the SRT server, ffmpeg receives the data in the middle of the stream. It has to read enough bytes to find a key frame. This number of bytes it can read is limited by probesize. With a high bitrate this limit might be exceed before a key frame has been found and this results missing information about the stream, e.g. resolution.

A general higher default value for SRT will probably solve this issue.

For adding a new channel you can skip the wizard and directly start with the advanced settings.