Varying frame rate caused video lagging.

[not-an-error]

Your environment.

• Version: v3.1.41
• Browser: Chrome Version 109.0.5414.119 (Official Build) (64-bit)
• Other Information

What did you do?

Hi,

We've been using pion to build a server to browser app. The app runs in a loop that renders an image, converts that to H.264 and streams over to the browser in each iteration. So far the experience has been great!

During the development, we sometimes find that the video lags a lot in the browser. Loooking at webrtc stats, we noticed that [framesReceived - framesDecoded] has built up in these cases.

To have a better understanding of our app's behavior, it renders the image according to some user input. When there is no input for a while, the app just idles and stops sending any frames. Upon the next frame, it will be awaken. So there is a sudden frame rate change in this situation.

Here is the most related issues I could find online: https://bugs.chromium.org/p/chromium/issues/detail?id=1098514. In comment-11 one pion dev suggested to use WriteRTP() instead of WriteSample() to customize the RTP packet's timestamp. My questions are:

1. How exactly would RTP timestamp affect the frame's rendering time on the browser side. That is, what's the mechanism behind this:

If the frame rate suddenly increases from 20 fps to 30 fps and the RTP timestamps are set incorrectly it may happen that the 30 fps content is being played out at 20 fps which will cause the queue to grow.

2. Why would WriteRTP() be useful in this case? FWIW, we are using WriteSample(), but the duration calculation is still based on wall clock.

3. Would it be helpful if we just keep sending frames, instead of having possibly large idle time between two frames?

P.S. I've also read #2041, but that's indeed we are doing right now.

Thanks!

What did you expect?

Frame rate stays steady

What happened?

Frame rate lags (significantly sometimes).

[Sean-Der]

Hey @not-an-error

Really sorry I didn't get to this sooner. I am making a bigger effort to triage Pion WebRTC issues.

You need some way to communicate the connection is still active. I don't know the idiomatic way to do that, would it be possible to send 4x4 black frames (some pre-encoded source) to communicate that connection is still alive?

You could also try adjusting the timestamps. However in the time when the connection is dead Chrome is going to be increasing the jitterbuffer etc..

Would love to know how you ended up resolving it. If you didn't I would love to help you take a stab at it!