Ffmpeg example?

[ZeoWorks]

Hi, is there any examples as how to pipe ffmpeg to webrtc?

[isqad]

Hello!
https://github.com/pion/example-webrtc-applications/blob/62ca572878fe5f600bfd71da47ed427e838b54b7/twitch/main.go#L142

[ZeoWorks]

Hello! https://github.com/pion/example-webrtc-applications/blob/62ca572878fe5f600bfd71da47ed427e838b54b7/twitch/main.go#L142

Thank you!
Would it be possible to pipe the feed directly to WebRTC for the lowest latency possible without the need of rtmp?
(Even UDP would be great).

I've seen this user somewhat achieved this result, however it doesn't work for hardware encoded h264;
https://github.com/wakabayashik/mpegts-to-webrtc

[chalkytoast]

Hi ZeoWorks, my solution was to use ffmpeg's nut format which behaves nicely with pipes.

My ffmpeg command:
ffmpeg -f v4l2 -input_format h264 -video_size 1280x720 -i /dev/video0 -c:v copy -f nut pipe:1

Reading a frame/sample:

func readNutSample(ndx *Demuxer, buf []byte) (int, error) {
	for {
		e, err := ndx.ReadEvent()
		if err != nil {
			return 0, err
		}
		if e.Type() == FrameEvent {
			n, err := e.(Frame).Data().Read(buf)
			return n, err
		}
	}
}

Writing to track:

        ndx := NewDemuxer(stdout)     // ffmpeg nut demuxer
        buf := make([]byte, 0x100000) // 1 MiB
        buf[3] = 1                    // Annex B prefix

	for {
		n, _ := readNutSample(ndx, buf[4:])
		sample := media.Sample{Data: buf[:n+4], Duration: time.Second}
		track.WriteSample(sample)
	}

To keep it short I didn't include error handling. I'm not sure if setting Duration to time.Second is correct, but you know what they say: "The best way to get a correct answer is to post an incorret one"

golang nut demuxer: https://github.com/retailnext/gonut

[ZeoWorks]

Hi ZeoWorks, my solution was to use ffmpeg's nut format which behaves nicely with pipes.

My ffmpeg command: ffmpeg -f v4l2 -input_format h264 -video_size 1280x720 -i /dev/video0 -c:v copy -f nut pipe:1

Reading a frame/sample:

func readNutSample(ndx *Demuxer, buf []byte) (int, error) {
	for {
		e, err := ndx.ReadEvent()
		if err != nil {
			return 0, err
		}
		if e.Type() == FrameEvent {
			n, err := e.(Frame).Data().Read(buf)
			return n, err
		}
	}
}

Writing to track:

        ndx := NewDemuxer(stdout)     // ffmpeg nut demuxer
        buf := make([]byte, 0x100000) // 1 MiB
        buf[3] = 1                    // Annex B prefix

	for {
		n, _ := readNutSample(ndx, buf[4:])
		sample := media.Sample{Data: buf[:n+4], Duration: time.Second}
		track.WriteSample(sample)
	}

To keep it short I didn't include error handling. I'm not sure if setting Duration to time.Second is correct, but you know what they say: "The best way to get a correct answer is to post an incorret one"

golang nut demuxer: https://github.com/retailnext/gonut

Hi Chalky, thank you for this!
Is it possible to use hardware encoding with ffmpeg prior to pipe instead of -c:v copy? (Example; h264_amf / h264_qsv / h264_nvenc)?

[chalkytoast]

Hi Chalky, thank you for this!
Is it possible to use hardware encoding with ffmpeg prior to pipe instead of -c:v copy? (Example; h264_amf / h264_qsv / h264_nvenc)?

I tested it also with h264_qsv, but not extensively.

[ZeoWorks]

Hi Chalky, thank you for this!
Is it possible to use hardware encoding with ffmpeg prior to pipe instead of -c:v copy? (Example; h264_amf / h264_qsv / h264_nvenc)?

I tested it also with h264_qsv, but not extensively.

Thank you! After piping it, does webrtc re-encode the feed? Or is it just a passthrough of the already encoded feed?

[chalkytoast]

Thank you! After piping it, does webrtc re-encode the feed? Or is it just a passthrough of the already encoded feed?

No re-encoding, CPU had about 3% load.

[ZeoWorks]

Hi again, do you think you could provide a full example of how this is achieved?
It would be much appreciated sir as I'm having some difficulty.

[Sean-Zeo]

Hi again, any news?
Thanks sir.

[Sean-Der]

Hi @ZeoWorks

These all look like good solutions to me :)

rtp-to-webrtc also has some good examples of generating RTP from ffmpeg and handling it in Pion.

Doing it in memory is also possible, but significantly more work. Happy to help if you have any further questions.