Point Cloud Streams... OpenNI or something else?

[crosswick]

Hi all,

I've been looking at implementing a recent academic paper that deals with real-time 6DOF pose estimation; so far I've considered using PCL for this and found a relevant example there that uses OpenNI for point cloud streaming from a depth camera.

Is OpenNI a typical way of dealing with point cloud streams? I've tried finding something in the Cilantro framework that's relevant but couldn't find much - admittedly with my pretty entry-level C++ knowledge.

Thanks for any pointers.

[Algomorph]

Hi, @crosswick,

First off, the question you want to answer is whether you really want to stream for this -- if your goal is just to design something that can process frames at frame rate, and you don't need a live demo with a real camera attached to your computer/whatever device you use for processing, you can use prerecorded files in a variety of formats (heck, you could even cache individual image frames from disk). Then you'd just need something that can read the input from your desired format quickly.

If you really do need real-time input, the low-level driver/tool package that you wanna use for streaming depth camera input really depends on the hardware you're using. Now, OpenNI was really popular with Xtion RGB-D cameras and the Kinect (& PrimeSense) cameras that came before them. All things considered, it's fairly "legacy" at this point, and there are modern-day replacements in the OSS domain, such as freenect. There are separate drivers for RealSense, Kinect for Azure (i.e. Kinect for Azure SDK), etc. If you use PCL or OpenCV, both have a variety of interfaces to utilize a bunch of these driver packages, implemented with various levels of quality. One of the packages that is dedicated to providing a uniform access (URI-style) to all possible depth (and other) cameras (including reading files from disk) is Steven Lovegrove's Pangolin. Dev on that is not very active at this point, but Steve keeps it well-maintained. I'd recommend taking a look at that and similar packages.

And C++ knowledge will come with usage, as long as you're not afraid to read and learn best practices as you go, so no worries there :)

P.S. cilantro, if memory serves me right, interfaces with Pangolin instead of reinventing the wheel like PCL and OpenCV (no offence to folks at PCL and OpenCV -- I know their packages have probably been around longer than Pangolin and provide access to some extra formats + hardware, it just doesn't make sense to replicate effort on their part IMHO).

[crosswick]

hi @Algomorph - thanks for the in-depth reply!

I do really want real-time streaming, yes... and not only because I'm just super intrigued by this paper:

Real-Time 6-DOF Pose Estimation of Known Geometries in Point Cloud Data

but also because for my intended use case, I am very much looking forward to being able to use the tracking data in real-time... it will probably benefit the organic nature of the system as a whole.

Thanks for the freenect and Pangolin suggestions, I will check those out. The camera I'm looking at is one from Revopoint, who have started to offer an SDK for their consumer 3D scanners, see this example project. I know they have offered OpenNI drivers for their other cameras in the past, but perhaps that framework is too old-fashioned now...

Lastly: if there's any Cilantro example project similar to this PCL project, I'd be interested to learn: Tracking object in real time

It uses a Particle Filter similar to the method described in the paper, which is why it seemed like a good starting point for me.

[Algomorph]

@crosswick looks like your camera is too new to have interfaces in either libfreenect or Pangolin (feel free to contribute!).
The SDK binaries they use in the example you give can be downloaded from here and then linked against, like in the example. IDK why Revopoint doesn't open-source their stuff: from what I can tell, all features are fairly standard. But you get what they give you. IDK whether cilantro has particle filters at this point or not (@kzampog?), I would look through the examples. If it doesn't, I wouldn't rely on PCL too much -- in my experience, unless something radically changed in the last couple of years, PCL is pretty messy and not-very-well-maintained, so I would look at dedicated particle filter libraries or other better-maintained libraries for that.

[crosswick]

@Algomorph are you sure the RevoPoint 3DViewer example needs any binaries to be linked against? I've been able to compile and run it using only that Git repository, which I don't think contains any binaries.

It feels like Cilantro and Pangolin is the way to go... there's been very little development/maintenance of OpenNI as of late, and the PCL Discord seems pretty much a ghost town...

If I were to put some effort into making an interface in Pangolin, and/or get the real-time particle filter stuff working in Cilantro, what would be a good forum to discuss that, currently? The Issue pages of the respective GitHubs, or is there a relevant Discord or Discourse or something?

[crosswick]

ah, this must be those binaries: https://github.com/Revopoint/3DViewer/tree/master/thirdparty/3DCamera