CorrespondentDream: Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences

We optimize NeRF alternately using the SDS loss and cross-view correspondence loss. To compute the cross-view correspondence loss, we render two adjacent view sets from NeRF with identical noise, inputting them into a frozen pre-trained multi-view diffusion model. We then extract multi-layer features from the diffusion U-Net’s upsampling layers to establish diffusion correspondences between each view pair. Utilizing ground-truth camera parameters and NeRF-rendered depth, we reproject pixels to obtain NeRF correspondences. By minimizing the discrepancy between diffusion correspondences (the pseudo ground-truth) and NeRF correspondences, we correct NeRF’s 3D infidelities in the depths.