RTP h264_packet.go H264Payloader() Removes NAL Units of types 9 and 1

[robotogre]

Your environment.

• Version: github.com/pion/rtp v1.7.13*

Problem: H264Payloader() is 'removing' NAL units of type 9 and type 1.

My Test:

• I created an mpegts file using FFmpeg. h264, baseline, 30 fps, all frames evenly spaced with PTS/DTS delta of 3000. GOP size is two seconds (60 frames). I log the NAL units via FFmpeg/FFplay.
• I stream the mpegts file via FFmpeg into my Go program over SRT. I log the NAL units as they come of out the Haivison SRT library. (They match the NAL units recorded from the mpegts file)
• I send each NAL unit into pion/h264Packetizer.Packetize(). I log the NAL unit types. Some units of type 9 and 1 are missing.

(H264) -> SRT -> (mpegts) -> pion/264Reader.NextNAL() -> pion/h264Packetizer.Packetize() -> [Some frames are missing]

Line 87 "filters out" AUD NAL units. https://github.com/pion/rtp/blob/master/codecs/h264_packet.go
But is also removes types 1 & 9 NAL units.

h264_packet.go

What is lost:
Entire Access Units that contain:
NAL Unit Type 1 (Coded slice of a non-IDR picture).
NAL Unite Type 9 (NAL Unit Delimiter)
I have isolated a single Access Unit to diagnose. (Though these Access Unit drops occur multiple times per stream.) Here is the original Access Unit of the .ts file that is sent via SRT:
[FFprobe & FFMpeg output]

[trace_headers @ 0x55b715bc6480] Packet: 802 bytes, pts 226920, dts 226920, duration 3000.
[trace_headers @ 0x55b715bc6480] Access Unit Delimiter
[trace_headers @ 0x55b715bc6480] 0           forbidden_zero_bit                                          0 = 0
[trace_headers @ 0x55b715bc6480] 1           nal_ref_idc                                                00 = 0
[trace_headers @ 0x55b715bc6480] 3           nal_unit_type                                           01001 = 9
[trace_headers @ 0x55b715bc6480] 8           primary_pic_type                                          111 = 7
[trace_headers @ 0x55b715bc6480] 11          rbsp_stop_one_bit                                           1 = 1
[trace_headers @ 0x55b715bc6480] 12          rbsp_alignment_zero_bit                                     0 = 0
[trace_headers @ 0x55b715bc6480] 13          rbsp_alignment_zero_bit                                     0 = 0
[trace_headers @ 0x55b715bc6480] 14          rbsp_alignment_zero_bit                                     0 = 0
[trace_headers @ 0x55b715bc6480] 15          rbsp_alignment_zero_bit                                     0 = 0
[trace_headers @ 0x55b715bc6480] Slice Header
[trace_headers @ 0x55b715bc6480] 0           forbidden_zero_bit                                          0 = 0
[trace_headers @ 0x55b715bc6480] 1           nal_ref_idc                                                10 = 2
[trace_headers @ 0x55b715bc6480] 3           nal_unit_type                                           00001 = 1
[trace_headers @ 0x55b715bc6480] 8           first_mb_in_slice                                           1 = 0
[trace_headers @ 0x55b715bc6480] 9           slice_type                                              00110 = 5
[trace_headers @ 0x55b715bc6480] 14          pic_parameter_set_id                                        1 = 0
[trace_headers @ 0x55b715bc6480] 15          frame_num                                                1011 = 11
[trace_headers @ 0x55b715bc6480] 19          num_ref_idx_active_override_flag                            0 = 0
[trace_headers @ 0x55b715bc6480] 20          ref_pic_list_modification_flag_l0                           0 = 0
[trace_headers @ 0x55b715bc6480] 21          adaptive_ref_pic_marking_mode_flag                          0 = 0
[trace_headers @ 0x55b715bc6480] 22          slice_qp_delta                                      000010101 = -10
[trace_headers @ 0x55b715bc6480] 31          disable_deblocking_filter_idc                               1 = 0
[trace_headers @ 0x55b715bc6480] 32          slice_alpha_c0_offset_div2                                  1 = 0
[trace_headers @ 0x55b715bc6480] 33          slice_beta_offset_div2                                      1 = 0

[mpegts @ 0x55a96707a5c0] pid=100 pes_code=0x1e0
[h264 @ 0x55a96721f940] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x55a96721f940] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 2

media_type=video
stream_index=0
key_frame=0
pkt_pts=226920
pkt_pts_time=2.521333
pkt_dts=226920
pkt_dts_time=2.521333
best_effort_timestamp=226920
best_effort_timestamp_time=2.521333
pkt_duration=3000
pkt_duration_time=0.033333
pkt_pos=37224
pkt_size=98
width=640
height=272
pix_fmt=yuv420p
sample_aspect_ratio=1:1
pict_type=P
coded_picture_number=33
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
color_range=unknown
color_space=unknown
color_primaries=unknown
color_transfer=unknown
chroma_location=left

Here is what happens when the Pion Packetizer consumes it:

srtConnection-handleH264Data() nal.UnitType: 0 nal.RefIdc: 0 nal.ForbiddenZeroBit: false
srtConnection-handleH264Data() Adding srt NAL unit to rtp packets. len(nal.Data): 6 samples: 3000
Packetizer-Packetize() len(payload): 6
pion.rtp.codec.H264Payloader-Payload()
pion.rtp.codec.H264Payloader-Payload() AUD NAL type found, RETURNING!
Packetizer-Packetize() len(payloads): 0

---

Please note that Pion is parsing the first NAL type as '0' (Incorrect), and the second as AUD (Correct). Both NAL units are discarded, and therefore the entire Access Unit.