Flickering in multiview video

[Mirzakhanversion2]

Hello,

Thank you for your wonderful work. The problem is when I am trying to generate a video from novel views , I can see a lot of flickering. Is this behavior expected when generating the novel views. Can you please share your view on it?

Thank you

[liuyuan-pal]

Hi, thanks for trying our codes.
The reason for causing the problem is two-fold.
First, 16 views are not enough to train a NeRF with stable renderings, especially for hash-grid-based NeRF. Using the MLP-based NeuS could relieve this problem due to its smoothness constraints in the MLP network.
Second, the generated views are not strictly multiview consistent and some inconsistency still remains. In this case, you may generate multiple times and select the one with the strongest multiview consistency to improve the quality.

[Mirzakhanversion2]

Thank you for your answer. I am actually not doing 3D reconstruction but generating more than 16 views with varying azimuth and elevation. Though it is generating the object, but i can see the flickering between the frames. You can see in the attached video. Can we expect this to be happened if we generate more than 16 views or is it just a mistake from my side?video

[liuyuan-pal]

SyncDreamer is designed to generate 16 views of fixed viewpoints and cannot generate other views. I'm not sure about how to generate images from other viewpoints but it indeed performs much worse if you change these viewpoints.