Extract motion vectors

[adrianmay]

Hi there,

I'm looking for a very fast way to extract motion vectors from an h264 stream inside some container. I just need the x,y coordinates of the block and it's dx,dy movement. I'd like to cut out any CPU time not contributing to the motion vectors. The data is coming from ffmpeg (encoding as h264 in hardware) so I can set the container format to anything.

Is this repo what I'm after?

Please excuse my ignorance of h264 but I can't answer this question by looking at the code, nor would I know how to do it if you just said "yes", so some guidance would be very much appreciated.

I know this isn't really an "issue" but github seem to have disabled all other forms of messaging.

TIA, Adrian.

[aizvorski]

Hi Adrian - Thanks for asking. The MVs are contained in the slice data - decoding is not currently supported, as this library mostly works with header data. There is some work-in-progress code, please see #8. If you want to dig into that, that would be great, but it is going to be quite a lot of work - mainly because this code doesn't have working CABAC (yet).

There are some possibly better starting points:

• JM reference software: http://iphome.hhi.de/suehring/tml/download/
• JM reference modified by VQEG to dump motion vectors and more: https://vqeg.github.io/software-tools/encoding/modified-avc-codec/
• FFMPEG decoder (try https://github.com/shengbinmeng/ffmpeg-h264-dec), or possibly a couple of simpler decoders https://github.com/mbebenita/Broadway

Note that in addition to the MVs, you will at a minimum need the macroblock types, partitioning modes and likely the reference frames as well. It is simpler than fully decoding the video (no DCT, no motion compensation) but not hugely simpler.

I'm interested in this as well, please let me know how you solve this.

[lbdroid]

I believe that there would be a huge demand for this feature if it were to be implemented. In particular, I'd like to be able to apply this to motion detection in the field of physical security. Motion detection in open source software currently requires full decompression and frame-by-frame analysis, and the CPU cost is near prohibitive.

If motion vectors could be extracted from the compressed domain, it would allow high resolution and high frame rate motion detection even on very low end hardware.

[aizvorski]

@lbdroid Indeed, that would be useful. I can offer to do this feature as a bounty, as it involves a significant effort - please contact me at aizvorski at gmail if you're interested.

[isra60]

If my goal is simply to detect the presence of motion—without needing to measure the magnitude or direction of that motion—would there be a simpler or faster way to obtain this information from the H.264 bitstream? Maybe it is no neccesary to even decode the mv, right? I'm pointing to detect frozen image on the input of the encoder by analyzing it's output