Panorama 360

[sctrueew]

Hi,

I have a 360 panorama image. How can I display it in 3D?

[EasternJournalist]

Hi, MoGe is designed to handle natural perspective images. While it’s possible to use MoGe with panorama images, it would require a few modifications (which could be quite tedious). Here's one approach to obtain a panorama depth map:

1. Convert the panorama image to cube maps: Split the panorama into 6 faces and predict 3D point maps on each face separately, and convert to Euclidean distance maps. This will give you a panorama depth(distance) map, though the scales from the 6 faces may not be aligned.

2. Rotate and reprocess: Next, convert the panorama image to another cube map by rotating it 45 degrees. Predict the depth for the new cube map, which will overlap with the first one. This overlap allows you to fuse the two depth maps into a seamless panorama depth map, for example, using poisson blending.

The panorama depth map can be easily lifted to 3D points via $d * \vec r$ where $d$ is the depth(distance) and $\vec r$ is the direction of a pixel.

[sctrueew]

Thank you for your response. I appreciate your help.

I would like to ask if it’s possible to extract points directly from a 360-degree image or environment. Currently, I am using a script with dep-anything to obtain points.

def get_uni_sphere_xyz(H, W):
    j, i = np.meshgrid(np.arange(H), np.arange(W), indexing='ij')
    u = (i + 0.5) / W * 2 * np.pi
    v = ((j + 0.5) / H - 0.5) * np.pi
    z = -np.sin(v)
    c = np.cos(v)
    y = c * np.sin(u)
    x = c * np.cos(u)
    sphere_xyz = np.stack([x, y, z], -1)
    return sphere_xyz

Looking forward to your guidance on this matter.

[EasternJournalist]

The code is for lifting panorama depth map to 3D points. However, the problem here is how to obtain the panorama depth map.
It is not recommanded to use models like DepthAnything and MoGe to directly infer with panorama images and take the depth, as the results will be most likely unsatisfying (distorted and not consistent at boundaries). Only models trained with panorama images allows such direct estimation. May be you can find some here https://github.com/bkhanal-11/awesome-360-depth-estimation?tab=readme-ov-file.

If using generalized monocular depth estimation like DepthAnything and MoGe, it is a better idea to divide a panorama image into several perspective images, infer respectively and finally fuse them together. I am planning to implement one in a separate inference script. Please stay tuned.

[sctrueew]

@EasternJournalist Hi, thank you for sharing your thoughtful approach. it's a clever strategy to tackle the limitations of generalized models with panorama images. Good luck with your implementation.