Smooth Video Rendering

[superware]

Hello,

I have a GStreamer pipeline ending with d3d11videosink drawing textures (video frames) into a D3DImageEx back buffer, this is super low CPU since no buffers are being copied. For reference, in WPF, the only way I manage to get smooth playback is by running a new Dispatcher in a different thread and using HostVisual and VisualTarget to connect the two visual trees (details). Plus, no "airspace" issues - can draw elements on top of the video.

GStreamer is very versatile and cross-platform, but it seems to be mainly up to Avalonia to allow rendering a video frame immediately (since the pipeline is syncing playback), anything else causes tearing, skewing and other artifacts. Is there a way to get smooth playback in Avalonia?

[maxkatz6]

Smooth video rendering comes to two possible implementations:

1. Use GPU interop, and render content directly into the texture (d3d, opengl or vulkan interop are supported by Avalonia, depending on platform). You should be able to use glimagesink and d3d11videosink. in Avalonia, see these samples https://github.com/AvaloniaUI/Avalonia/tree/master/samples/GpuInterop

2. On some other platforms you might need to use native video controls, in Avalonia it's possible via NativeControlHost - but airspace issue is there.

[superware]

Let me rephrase, by "comparing to Avalonia which is choppy in general" I meant that the video plays a bit choppy even without interacting with that map control (although it's a bit better in x64 Release, I would even consider "production grade"), but panning the map causes a more noticeable effect. Please play the videos on desktop in full screen to appreciate the differences.

Can you please estimate why that interaction significantly affects video smoothness? Is there a way to somehow boost the video control render priority (or possibly lower other controls priority)? Obviously only the video is sensitive to exact rendering timing.

[maxkatz6]

Can you please estimate why that interaction significantly affects video smoothness

Since video control uses dispatcher for each frame updates, it's very sensitive to any other UI operation. Pretty much my assumption, without running any code.

Is there a way to somehow boost the video control render priority

Ideally - don't touch UI thread or dispatcher at all, when there is no need to. And keep everything on the render thread.
I can't really help with exact code, but Composition APIs and GPU interop is basically what you need. Something like CompositionCustomVisualHandler can also help, since you can trigger next render frame there directly.

[FoggyFinder]

And keep everything on the render thread.

True. Yet I didn't find any sample to get access to render thread outside of Render method

[superware]

I looked into CompositionCustomVisualHandler but failed to fully understand it. There are actually only 3 inputs from GStreamer:

• The surface shared handler
• Resize event (width/height)
• "draw" event (surface updated)

I tried UpdateWithKeyedMutexAsync on the control's Render method (instead of on the UI thread) but then the method is only being called once.

Wish there was a simple example of the most performant way to render a DX surface. And since GStreamer is cross-platform with high composition, this might lead to a comprehensive solution for video playing/streaming in Avalonia.

[superware]

And keep everything on the render thread.

True. Yet I didn't find any sample to get access to render thread outside of Render method

@FoggyFinder Any progress?