How to get decoded frames

[stoefln]

Hi there, thanks for this great project!

I would need the frames decoded in full resolution. I saw that the h264-converter is used to decode the data, and forward it to the mediaSource which is assigned to the video tag.
I am not familiar with video decoding at all, so I don't get where I could get the decoded frames from.
My guess was that it should be somewhere around here, right?
Or is there any other place maybe inside of ws-scrcpy where I could access the decompressed frames?

[drauggres]

Hi.
h264-converter does not decode video, it creates mp4-container (google muxing) from raw h264-frames and feeds them to a mediaSource. After that, decoding and rendering is done internally by your browser, i.e. you can't access decoded frames (except capturing the content of the video-tag with .toDataURL()).

For decoded frames you need to look at other players/decodes:

ws-scrcpy/src/app/player/BroadwayPlayer.ts

Lines 46 to 48 in 5af6272

	this.avc.onPictureDecoded = (buffer: Uint8Array, width: number, height: number) => {
	this.onFrameDecoded(width, height, buffer);
	};

ws-scrcpy/src/app/player/TinyH264Player.ts

Lines 46 to 50 in 5af6272

	switch (message.type) {
	case 'pictureReady':
	const { width, height, data } = message;
	this.onFrameDecoded(width, height, new Uint8Array(data));
	break;

ws-scrcpy/src/app/player/WebCodecsPlayer.ts

Lines 89 to 92 in 5af6272

	return new VideoDecoder({
	output: (frame) => {
	this.onFrameDecoded(0, 0, frame);
	},

Also this could be usefull: Difference between different decoders

[drauggres]

For broadway.js and tinyh264 in the specified places a Uint8Array will contain a decoded frame in YUV-format. It is later translated into RGB and is rendered on a canvas.

For WebCodecs frame is VideoFrame (https://www.w3.org/TR/webcodecs/#videoframe-interface)

[stoefln]

@drauggres thanks a lot for the quick reply!

[ahmedmukhtar1133]

I would like to read the decoded frames with OpenCv, but it seems OpenCv's imdecode function is not able to read the frame (returns empty object).
Can you tell me more about the format of the frame?
If you'd even know how to transform it to "normal" RGB format, that would be amazing!

[ahmedmukhtar1133]

Using frame from onPictureDecoded from BroadWay player.

[drauggres]

If you'd even know how to transform it to "normal" RGB format, that would be amazing!

ws-scrcpy/vendor/h264-live-player/YUVCanvas.ts

Lines 14 to 43 in 5af6272

	public decode(buffer: Uint8Array, width: number, height: number): void {
	if (!buffer) {
	return;
	}
	const lumaSize = width * height;
	const chromaSize = lumaSize >> 2;
	const ybuf = buffer.subarray(0, lumaSize);
	const ubuf = buffer.subarray(lumaSize, lumaSize + chromaSize);
	const vbuf = buffer.subarray(lumaSize + chromaSize, lumaSize + 2 * chromaSize);
	for (let y = 0; y < height; y++) {
	for (let x = 0; x < width; x++) {
	const yIndex = x + y * width;
	const uIndex = ~~(y / 2) * ~~(width / 2) + ~~(x / 2);
	const vIndex = ~~(y / 2) * ~~(width / 2) + ~~(x / 2);
	const R = 1.164 * (ybuf[yIndex] - 16) + 1.596 * (vbuf[vIndex] - 128);
	const G = 1.164 * (ybuf[yIndex] - 16) - 0.813 * (vbuf[vIndex] - 128) - 0.391 * (ubuf[uIndex] - 128);
	const B = 1.164 * (ybuf[yIndex] - 16) + 2.018 * (ubuf[uIndex] - 128);
	const rgbIndex = yIndex * 4;
	this.canvasBuffer.data[rgbIndex + 0] = R;
	this.canvasBuffer.data[rgbIndex + 1] = G;
	this.canvasBuffer.data[rgbIndex + 2] = B;
	this.canvasBuffer.data[rgbIndex + 3] = 0xff;
	}
	}
	this.canvasCtx.putImageData(this.canvasBuffer, 0, 0);

[stoefln]

@drauggres I'd prefer to use OpenCv since it's much faster.
I checked how to convert YUV to RGB and it seems there are 3 different types

• YUV
• YUV420p
• YUV420sp

However, there are even more conversion types, some of them use postfixes such as "_IYUV" or "_NV12"
https://docs.opencv.org/3.4/df/d4e/group__imgproc__c.html

Any idea which one to use for converting?

The conversion itself would be simple:
https://stackoverflow.com/questions/2231518/how-to-read-a-frame-from-yuv-file-in-opencv#:~:text=To%20convert%20to%20RGB%20from,the%20correct%20conversion%20flag%20enum

[drauggres]

Sorry, I am also not an expert in video encoding and don't really know what type is it.