Self-Supervised Multi-View 3D Gaze Target Estimation via Probabilistic Ray Marching

RESUMO

We present a self-supervised approach, Self-MVGTE, for estimating 3D gaze targets from multiple camera views. Unlike existing methods that independently estimate 2D gaze targets per camera view, Self-MVGTE predicts gaze targets directly in 3D space for the first time. Moreover, it does not require any ground-truth annotations from the target scene and uses only the multi-view input images from a calibrated camera setup, pseudo 2D gaze target labels from a monocular gaze target estimation model, and 3D gaze vectors from a monocular 3D gaze estimation model. A key challenge is that these pseudo labels are inherently noisy and multi-view inconsistent. To address this, we propose a probabilistic ray marching framework, which models the uncertainty of these pseudo labels and exploits 3D gaze vectors as geometric priors. Specifically, these gaze vectors are first integrated into the monocular gaze target estimation model to improve its generalization to unseen scenes, producing higher-quality pseudo labels. Then, for 3D gaze target estimation, we construct a 3D gaze cone by casting a bundle of rays from the eye position around the gaze vector to strictly constrain the solution space. Within this cone, we propose a depth-guided feature sampling strategy using off-the-shelf DINOv2 and Depth-Anything-3 models, and estimate a spatial likelihood distribution of the gaze target. Finally, we convert the pseudo gaze target labels into a target distribution and softly optimize the network. Extensive experiments on the MVGT dataset show that Self-MVGTE achieves state-of-the-art performance, surpassing existing fully-supervised baselines.