How does it perform on outdoor datasets (such as kitti and nuscenes)?

[Sugar55888]

No description provided.

[maturk]

Hi @Sugar55888, this repo has not been tested with outdoor datasets. Mainly indoor room datasets. Let me know if you try testing it with kitti or nuescenes. Thanks.

[maturk]

One thing to note with unbounded outdoor scenes is that you might need a background model for distant scenery (common in car/driving datasets where the sky is visible). The dn-splatter codebase could be easily extended to incorporate a learned background model, e.g. with a small view dependent MlP as seen in some other works.

[Sugar55888]

Sorry for the late reply. I recently used the code of street gaussian to test on an outdoor dataset, and the results was not bad. But I haven't tested dn-splatter yet, because I haven't achieved good results for the example dataset. I will use dn-splatter to test the outdoor dataset later and share my results. Thank you for your reply!

[Sugar55888]

@maturk Hello, I recently used dn-splatter's code to experiment with some outdoor scenes. I found that some artifacts of different colors tend to appear during rendering. Maybe I need to add some background models or lidar supervision to improve the performance. Relying only on monocular depth estimation may perform well in indoor scenes, but it may need some improvement in outdoor scenes.

[maturk]

Hi @Sugar55888, sorry I think dn-splatter will not work on outdoor scenes. dn-splatter is designed to work best with sensor depth supervision, e.g. from the mushroom or scannet++ datasets, where there are clear and good estimates for depth for each input rgb image (they can be slightly noisy, that is what the edge aware loss is designed to mitigate) and with normal estimates that are relatively multi-view consistent. I feel like if you try generating monocular estimates (e.g. using omnidata or dsine) on datasets that go out of the domain of their training datasets (which are mainly indoor dataset scans similar to scannet) then the monocular estimates will fail and hence dn-splatter will perform poorly since it is trying to regularize on really bad estimates. In your case, you could try the vanilla splatfacto model, since that is only supervised with a RGB loss.

[Sugar55888]

@maturk Thanks for your reply! Yes, monocular depth estimation and normal estimation do not perform well in outdoor scenes. I used the vanilla splatfacto model for training and rendering, and it renders well, but the output geometry is not as good as dn-splatter. Maybe it is more effective to use depth sensor for supervision in outdoor scenes, like Street Gaussians.

[maturk]

What dataset do you have?

[Sugar55888]

I use the data I collected myself.

[szhang963]

@Sugar55888 hi, did you try depth supervision from lidar points?

[Sugar55888]

@szhang963 I tried using lidar in street gaussian, and it worked well. But when collecting datasets ourself, if it is a street scene, we should pay attention to collecting images from multiple perspectives as much as possible to facilitate better model training.

[szhang963]

@Sugar55888 Hi, thanks for your reply. Could you please tell me how to add lidar depth supervision in the splatfacto for street gaussian?
I use depth supervision (L1 loss) between the predicted depth (from rasterize_gaussians) and GT depth (lidar sensor). However, the depth prediction has been not improved.
Could you give me some guidance?
Thanks.

[altaykacan]

Hey @szhang963

I've been working on a similar problem for a while and thought I could help out. Just two questions:

• What poses are you using? If you are using COLMAP or monocular SLAM poses the poses are estimated with an arbitrary scale so they probably will not match the scale of the lidar depths (which is metric scale). You'd probably need to do some alignment as in here: https://github.com/maturk/dn-splatter/blob/main/dn_splatter/scripts/align_depth.py

• How sparse are the lidar depths? If they are point measurements they might not be enough to regularize the rendering. Also, if the lidar depths are sparse, are you making sure that you are only using the non-zero ground truth depth values to regularize the depth renderings from the Gaussians?

These are critical points that I found to be important, if you give more details on what you're trying, we can help you more. Hopefully these points were helpful :)

[szhang963]

@altaykacan Thanks for your help.
I reconstruct the lidar point cloud and then I get merged lidar points and per-frame poses without using colmap.

• Therefore, the depth map is dense except the sky area.
• I only use the non-zero ground truth depth values to calculate the loss.
• The depth map is z-depth in camera coordinate.

Thanks for your help again.

[cainiaoshidai]

@Sugar55888 Hey, how to use lidar for splat. Used for loss or sparse_pc? Thanks.