Allow frame by frame navigation by buffering on-demand

[jojje]

Problem definition

It seems that in order to perform frame-by-frame navigation with the A and D keys, frames have to be buffered already. When navigating to some point in the clip(s) with the mouse or any of the key-shortcuts, the buffer is empty, so frame-by-frame navigation isn't possible. In order to get the frame navigation to work, the user has to use the following cumbersome process:

1. Find interesting spots in the clips to be analyzed by jumping around with the available seek-options.
2. Once a spot has been found, jump back one keyframe (e.g. - key).
3. Press play until the buffer fills up to 50 frames, and be quick on the trigger to pause the video.
4. Navigate back with the A key until the specific frame of interest is found.

Proposed improvement

Get rid of steps 2-4 by either:

• A) Optimal UX: buffering the surrounding frames automatically. E.g. 25 frames on each side and have the code re-buffer automatically if frame navigation reaches either end of the buffer (buffer position 0 or (left|right)_frames.size()-1)
• B) OK UX: providing two new key shortcuts (e.g. shift-A & shift-D) that the user can trigger to automate step 2-3 above. I.e. seek backward one key-frame and then decode the frames up until the current position. Similarly for forward buffering where the buffering would work exactly like playing for 50 frames, with the exception that the views get rendered from the first item in the buffer. I.e. the same as performing steps 2-4 above, and navigating all the way back to the beginning of the buffer using the A key.

With option A, the user could navigate the entire clip frame-by-frame seamlessly.
With option B, they'd have to buffer manually, but would still allow pretty much uninterrupted workflow with just one extra key to press every now and then.

The reason I even mentioned option B was that after looking at the code, it might be easier to implement. Whatever improvement can be done in terms of frame-by-frame navigation would be greatly appreciated.

PS. fantastic tool. Just discovered it and am kicking myself for not finding it sooner.

[jonfryd]

Hi @jojje,

Thanks for sharing your ideas in such detail.

I'm in the process of finalizing the next version (which should be out this week), and once it's done, I'll take a break from working on the tool for at least three to four months. However, I'll keep your input in the back of my mind and see if there is something we can do in the future perhaps.

If someone wants to work on this, I'm willing to accept PRs, of course.

Cheers!

[jojje]

Looking forward to the new version. Hopefully I can implement this one myself on that code base.

Was staring for 30 minutes at CompareVideo::video function, but failed to find an obvious way to enhance the buffering due to so many different concerns being complected in that one function. Will see how the design has changed in the next version, and hopefully it'll simplify this task.

[jonfryd]

Looking forward to the new version. Hopefully I can implement this one myself on that code base.

Was staring for 30 minutes at CompareVideo::video function, but failed to find an obvious way to enhance the buffering due to so many different concerns being complected in that one function. Will see how the design has changed in the next version, and hopefully it'll simplify this task.

Haha, please don't expect a complete rewrite (even though that would be nice and long overdue). I'm mostly focusing on bug fixes and minor, incremental improvements for the sake of reliability.

Unfortunately, even after my unpushed changes, the entire video() method remains overly complex to comprehend and maintain. I appreciate that you had the courage to look into it, though.

[teijiIshida]

Frame by frame comparison is super useful and not many programs do this. I look forward to this feature whenever it will be implemented.

[Frizlab]

I’m looking forward to it too! It would be a game changer.

[jonfryd]

I've pushed a commit that enables fetching new frames when D is pressed. It's nothing fancy, but it's surely an improvement to the user experience. However, it doesn't resolve the process outlined by @jojje above, as addressing backward navigation—which requires frame-accurate seeks and actual pre-buffering—will be harder to do. Maybe at some later point...