Trying to pass rtc media packet to ffmpeg

[Generalomosco]

I've be trying to pass rtc media packet to ffmpeg, so i decide using unix socket but seems the frame is not valid!
I'm passing the frame bytes s.vCallback(s.width,s.height,videoKeyframe, int64(t), sample.Data); via unix socket then use ffmpeg to convert it to mp4 by connecting to the socket but unfortunately the mp4 is always corrupted without displaying anything.

The command line
ffmpeg -vsync cfr -f rawvideo -c:v rawvideo -fflags nobuffer -s 320x180 -pix_fmt yuv420p -i unix:./video.sock -f s16le -ar 48k -ac 1 -fflags nobuffer -i unix:./audio.sock -flags +global_header -acodec libfdk_aac -vsync 1 -vcodec libx264 -r 25 -b:v 4000k -pix_fmt yuv420p -preset slow -qp 0 vid.mp4

Here is the code below which i strap scope from save-to-webm

package streamBuf

import (
	"errors"
	"fmt"
	"github.com/pion/rtp"
	"github.com/pion/rtp/codecs"
	"github.com/pion/webrtc/v2"
	"github.com/pion/webrtc/v2/pkg/media/samplebuilder"
)

var (
	// ErrCodecNotSupported is returned when a rtp packed it pushed with an unsupported codec
	ErrCodecNotSupported = errors.New("codec not supported")
)

type MediaBuf struct {
	id                             string
	typ                             string
	audioBuilder, videoBuilder     *samplebuilder.SampleBuilder
	audioTimestamp, videoTimestamp uint32
  audCallback AudCallback
  vidCallback VidCallback
  width int
  height int
  initVid bool
}
type AudCallback func(bool, int64, []byte)
type VidCallback func(int, int, bool, int64, []byte)
func NewMediaBuf(id, typ string, audCB AudCallback, vidCB VidCallback) *MediaBuf {
	return &MediaBuf{
		id:           id,
		typ:           typ,
		audCallback:           audCB,
		vidCallback:           vidCB,
		audioBuilder: samplebuilder.New(10, &codecs.OpusPacket{}),
		videoBuilder: samplebuilder.New(10, &codecs.VP8Packet{}),
	}
}

func (s *MediaBuf) ID() string {
	return s.id
}
func (s *MediaBuf) PushRTP(pkt *rtp.Packet) error {
	if s.typ == "video" && pkt.PayloadType == webrtc.DefaultPayloadTypeVP8 {
		s.PushVP8(pkt)
	} else if s.typ == "audio" && pkt.PayloadType == webrtc.DefaultPayloadTypeOpus {
		s.PushOpus(pkt)
	}
	return ErrCodecNotSupported
}

func (s *MediaBuf) Stop() {
	s.Close()
}

func (s *MediaBuf) Close() {
	fmt.Printf("Finalizing media...\n")
}
func (s *MediaBuf) PushOpus(rtpPacket *rtp.Packet) {
	s.audioBuilder.Push(rtpPacket)

	for {
		sample, timestamp := s.audioBuilder.PopWithTimestamp()
		if sample == nil {
			return
		}
			if s.audioTimestamp == 0 {
				s.audioTimestamp = timestamp
			}
			t := (timestamp - s.audioTimestamp) / 48
			 s.aCallback(true, int64(t), sample.Data)
		
	}
}
func (s *MediaBuf) PushVP8(rtpPacket *rtp.Packet) {
	s.videoBuilder.Push(rtpPacket)

	for {
		sample, timestamp := s.videoBuilder.PopWithTimestamp()
		if sample == nil {
			return
		}
		// Read VP8 header.
		videoKeyframe := (sample.Data[0]&0x1 == 0)
		if videoKeyframe {
			// Keyframe has frame information.
			raw := uint(sample.Data[6]) | uint(sample.Data[7])<<8 | uint(sample.Data[8])<<16 | uint(sample.Data[9])<<24
			width := int(raw & 0x3FFF)
			height := int((raw >> 16) & 0x3FFF)

			if !s.initVid {
				// Initialize WebM saver using received frame size.
				s.width = width
        s.height = height
        s.initVid = true
			}
		}
		if s.initVid {
			if s.videoTimestamp == 0 {
				s.videoTimestamp = timestamp
			}
			t := (timestamp - s.videoTimestamp) / 90
			s.vCallback(s.width,s.height,videoKeyframe, int64(t), sample.Data);
		}
	}
}
func (s *MediaBuf) vCallback(width, height int, keyframe bool, timestamp int64, b []byte) {
  s.vidCallback(width, height, keyframe, timestamp, b)
}
func (s *MediaBuf) aCallback(keyframe bool, timestamp int64, b []byte) {
  s.audCallback(keyframe, timestamp, b)
}

[at-wat]

In your code, vCallback and aCallback receive VP8 frames with RTP payload descriptor and OPUS frames.

ffmpeg \
  -vsync cfr \
  -f rawvideo \ # (I'm not very sure ffmpeg can read raw VP8 stream as rawvideo *1)
  -c:v rawvideo \ # it specifies input codec as raw, but actually VP8 stream is given
  -fflags nobuffer 
  -s 320x180 \
  -pix_fmt yuv420p \
  -i unix:./video.sock \
  -f s16le \ # it specifies input format as raw signed 16^bit little-endian, but actually OPUS stream is given
  -ar 48k \
  -ac 1 \
  -fflags nobuffer \
  -i unix:./audio.sock \
  -flags +global_header -acodec libfdk_aac -vsync 1 -vcodec libx264 -r 25 -b:v 4000k -pix_fmt yuv420p \
  -preset slow -qp 0 vid.mp4

VP8 RTP descriptor https://tools.ietf.org/html/rfc7741#section-4.1
Some heading bytes in sample.Data are VP8 RTP descriptor, which is not a part of raw VP8 stream.
Payload descriptor is parsed by codecs.VP8Packet depacketizer and not in sample.Data.

*1: In general, raw stream of encoded video is difficult to be handled since it usually doesn't have information of frame boundary. Using RTP is easier.

[Generalomosco]

Thanks for the info, i will follow the process

[at-wat]

@Generalomosco sorry, payload descriptor is already parsed by codecs.VP8Packet depacketizer and not in sample.Data. Fixed above comment.

[Sean-Der]

Hey @Generalomosco it would be really amazing if you were able to share this with others!

If you are able to open a PR to this repo me and @at-wat would be able to help also :) no rush though.

The PR doesn't have to be perfect. I can refactor the code to make it pass our tests, as long as you can get something basic working. Thanks for using Pion, I love seeing cool stuff like this.

[Generalomosco]

Thanks i will by tomorrow