Display Issues with HIKVISION DS-2CD6D24FWD-IZHS in the form of green artifacts

[Seliah]

Hi guys! First of all, thank you so much for maintaining and providing this awesome library.

I am currently encountering a problem with display issues that seem like artifacts in RTSP streaming. I would be very happy to know what you guys think might cause effects like this. Also I would be very interested in the steps I need to take or functions I need to implement to fix this.

Description
Artifacts like in the screenshots below pop up in a high resolution (Full HD) RTSP stream with a specific camera type. This happens after starting the stream and stays for a few seconds. It does not always happen but most of the times. The camera type is the DS-2CD6D24FWD-IZHS by HIKVISION. This is a device with 4 cameras which will be identified as one device with many profiles in ONVIF. You can find it here: https://www.hikvision.com/en/products/IP-Products/Network-Cameras/Panoramic-Series/DS-2CD6D24FWD--I--Z--H--S---NFC-/.

In native players like VLC or ffplay, this problem does not occur - even when opening the stream around 10 times simultaneously.

Reducing the bitrate of the streams fixes the artifacts. Using a minimal bitrate is not optimal in my use case though.

I inserted the debug logs of this library that occur in the described event below. I noticed that the following messages will be repeated as long as the stream stays up:

16:13:42.642 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 7 +2s
16:13:42.643 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 8 +1ms
16:13:42.643 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 6 +0ms

NAL types 6, 7 and 8 are called SEI, SPS and PPS. I was wondering if the correct interpretation of these packets would solve my problem. Do you think that could be a possibility?

To Reproduce
Reproducing is likely very hard as this problem only pops up in the case of this exact camera type. It happens on high bitrate Full HD Streams, when opening them up.

Screenshots
Green overlaying artifacts:

Occasional mild artifacts:

Environment:

• OS: Linux - Pop_OS!
• Browsers: Chrome, Firefox, Opera, Vivaldi
• Version: 12.2.0

Steps I took

• I updated this library to the newest version, tried old versions like 6.0.4 as well
• Minimized the usage of the camera to sort out problems of parallel nature
• I analyzed the logs for errors or warnings

Logs

16:13:42.528 browser.mjs:268 msl:rtsp:outgoing DESCRIBE rtsp://192.168.151.73:554/Streaming/Channels/401?transportmode=unicast&profile=Profile_401 RTSP/1.0
CSeq: 2
Accept: application/sdp

 +0ms
16:13:42.535 browser.mjs:268 msl:rtsp:outgoing DESCRIBE rtsp://192.168.151.73:554/Streaming/Channels/401?transportmode=unicast&profile=Profile_401 RTSP/1.0
CSeq: 2
Accept: application/sdp
Authorization: <removed for security>

 +0ms
16:13:42.543 browser.mjs:268 msl:rtsp:incoming RTSP/1.0 200 OK
CSeq: 2
Content-Type: application/sdp
Content-Base: rtsp://192.168.151.73:554/Streaming/Channels/401/
Content-Length: 693

v=0
o=- 1685981622039637 1685981622039637 IN IP4 192.168.151.36
s=Media Presentation
e=NONE
b=AS:5050
t=0 0
a=control:rtsp://192.168.151.73:554/Streaming/Channels/401/?transportmode=unicast&profile=Profile_401
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:5000
a=recvonly
a=x-dimensions:1920,1080
a=control:rtsp://192.168.151.73:554/Streaming/Channels/401/trackID=1?transportmode=unicast&profile=Profile_401
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=420029; packetization-mode=1; sprop-parameter-sets=Z2QAKqwsaoHgCJ+WbgICAoAAA4QAAK/IAg==,aO48sA==
a=Media_header:MEDIAINFO=494D4B48010200000400000100000000000000000000000000000000000000000000000000000000;
a=appversion:1.0
 +0ms
16:13:42.544 browser.mjs:268 msl:rtsp:outgoing SETUP rtsp://192.168.151.73:554/Streaming/Channels/401/trackID=1?transportmode=unicast&profile=Profile_401 RTSP/1.0
CSeq: 3
Blocksize: 64000
Transport: RTP/AVP/TCP;unicast;interleaved=0-1
Authorization: <removed for security>

 +0ms
16:13:42.545 browser.mjs:268 msl:mp4:isom ftyp: [ftyp] (20)  size = 20 (4)  type = ftyp (4)  major_brand = isom (4)  minor_version = 0 (4)  compatible_brands = mp41 (4) +0ms
16:13:42.545 browser.mjs:268 msl:mp4:isom moov: [moov] (647)  size = 647 (4)  type = moov (4)  [mvhd] (108)    size = 108 (4)    type = mvhd (4)    version = 0 (1)    flags = 0 (3)    creation_time = 3768826422 (4)    modification_time = 3768826422 (4)    timescale = 1000 (4)    duration = 4294967295 (4)    rate = 65536 (4)    volume = 256 (2)    reserved = undefined (10)    matrix = 65536,0,0,0,65536,0,0,0,1073741824 (36)    pre_defined = undefined (24)    next_track_ID = 4294967295 (4)  [trak] (491)    size = 491 (4)    type = trak (4)    [tkhd] (92)      size = 92 (4)      type = tkhd (4)      version = 0 (1)      flags = 3 (3)      creation_time = 3768826422 (4)      modification_time = 3768826422 (4)      track_ID = 1 (4)      reserved = undefined (4)      duration = 0 (4)      reserved2 = undefined (8)      layer = 0 (2)      alternate_group = 0 (2)      volume = 256 (2)      reserved3 = undefined (2)      matrix = 65536,0,0,0,65536,0,0,0,1073741824 (36)      width = 125829120 (4)      height = 70778880 (4)    [mdia] (391)      size = 391 (4)      type = mdia (4)      [mdhd] (32)        size = 32 (4)        type = mdhd (4)        version = 0 (1)        flags = 0 (3)        creation_time = 3768826422 (4)        modification_time = 3768826422 (4)        timescale = 90000 (4)        duration = 4294967295 (4)        language = 0 (2)        pre_defined = 0 (2)      [hdlr] (45)        size = 45 (4)        type = hdlr (4)        version = 0 (1)        flags = 0 (3)        predefined = 0 (4)        handler_type = vide (4)        reserved = undefined (12)        name = VideoHandler� (13)      [minf] (306)        size = 306 (4)        type = minf (4)        [vmhd] (20)          size = 20 (4)          type = vmhd (4)          version = 0 (1)          flags = 1 (3)          graphicsmode = 0 (2)          opcolor = 0,0,0 (6)        [dinf] (36)          size = 36 (4)          type = dinf (4)          [dref] (28)            size = 28 (4)            type = dref (4)            version = 0 (1)            flags = 0 (3)            entry_count = 1 (4)            [url ] (12)              size = 12 (4)              type = url  (4)              version = 0 (1)              flags = 1 (3)        [stbl] (242)          size = 242 (4)          type = stbl (4)          [stsd] (150)            size = 150 (4)            type = stsd (4)            version = 0 (1)            flags = 0 (3)            entry_count = 1 (4)            [avc1] (134)              size = 134 (4)              type = avc1 (4)              reserved = undefined (6)              data_reference_index = 1 (2)              pre_defined = 0 (2)              reserved2 = undefined (2)              pre_defined2 = 0,0,0 (12)              width = 1920 (2)              height = 1080 (2)              horizresolution = 4718592 (4)              vertresolution = 4718592 (4)              reserved3 = 0 (4)              frame_count = 1 (2)              compressorname = �������������������������������� (32)              depth = 24 (2)              pre_defined3 = 65535 (2)              [avcC] (48)                size = 48 (4)                type = avcC (4)                configurationVersion = 1 (1)                AVCProfileIndication = 100 (1)                profile_compatibility = 0 (1)                AVCLevelIndication = 42 (1)                lengthSizeMinusOne = 255 (1)                sequenceParameterSets = [object Object],[object Object],[object Object] (28)                pictureParameterSets = [object Object],[object Object],[object Object] (7)          [stts] (16)            size = 16 (4)            type = stts (4)            version = 0 (1)            flags = 0 (3)            entry_count = 0 (4)          [stsc] (16)            size = 16 (4)            type = stsc (4)            version = 0 (1)            flags = 0 (3)            entry_count = 0 (4)          [stco] (16)            size = 16 (4)            type = stco (4)            version = 0 (1)            flags = 0 (3)            entry_count = 0 (4)          [stsz] (20)            size = 20 (4)            type = stsz (4)            version = 0 (1)            flags = 0 (3)            sample_size = 0 (4)            sample_count = 0 (4)          [stss] (16)            size = 16 (4)            type = stss (4)            version = 0 (1)            flags = 0 (3)            entry_count = 0 (4)  [mvex] (40)    size = 40 (4)    type = mvex (4)    [trex] (32)      size = 32 (4)      type = trex (4)      version = 0 (1)      flags = 0 (3)      track_ID = 1 (4)      default_sample_description_index = 1 (4)      default_sample_duration = 0 (4)      default_sample_size = 0 (4)      default_sample_flags = 0 (4) +0ms
16:13:42.557 browser.mjs:268 msl:rtsp:incoming RTSP/1.0 200 OK
CSeq: 3
Session:        928271617;timeout=60
Transport: RTP/AVP/TCP;unicast;interleaved=0-1;ssrc=10de9f4b;mode="play"
Date:  Mon, Jun 05 2023 16:13:42 GMT

 +0ms
16:13:42.558 browser.mjs:268 msl:rtsp:outgoing PLAY rtsp://192.168.151.73:554/Streaming/Channels/401/?transportmode=unicast&profile=Profile_401 RTSP/1.0
CSeq: 4
Range: npt=0-
Session: 928271617
Authorization: <removed for security>

 +0ms
16:13:42.602 browser.mjs:268 msl:rtsp:incoming RTSP/1.0 200 OK
CSeq: 4
Session:        928271617
RTP-Info: url=rtsp://192.168.151.73:554/Streaming/Channels/401/trackID=1?transportmode=unicast&profile=Profile_401;seq=20197;rtptime=1375439940
Date:  Mon, Jun 05 2023 16:13:42 GMT

 +0ms
16:13:42.642 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 7 +2s
16:13:42.643 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 8 +1ms
16:13:42.643 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 6 +0ms
16:13:42.724 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 7 +81ms
16:13:42.725 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 8 +0ms
16:13:42.725 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 6 +1ms
16:13:42.915 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 7 +190ms
16:13:42.916 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 8 +1ms
16:13:42.917 browser.mjs:268 msl:h264depay H264depayComponent can only extract types 1,5 and 28, got 6 +1ms

[steabert]

The NAL units that are SPS and PPS are probably OK to ignore, we cannot handle these in-band but instead we rely on this information to be sent out-of-band in the sprop-parameter-sets in the SDP (which according to your logs seems to be there).

However, not parsing the SEI NAL unit might be the problem. If you can find out more about what this does you could potentially add support for these?

Note: I changed this from "bug" to "enhancement", since the library is not designed for full H.264 support anyway, but we are happy to accept contributions to increase the level of what can be handled.

[Seliah]

Thank you for this information! We are not sure about adding in additional support yet but we will let you guys know.