Seeing Eye to AI: Comparing Human Gaze and Model Attention in Video Memorability

Prajneya Kumar^*, Eshika Khandelwal^*, Makarand Tapaswi, and Vishnu Sreekumar

*Equal first contribution. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 2082–2091.

Paper PDF Supplementary material arXiv Project page Code Talk video

Research question

When a machine learning model is trained to predict how memorable a video is, does it learn to attend to the same visual information as humans?

Approach

We developed a CNN–Transformer architecture for video memorability prediction. Each frame passes through an image backbone, such as a CLIP-pretrained ResNet-50; the resulting spatio-temporal features go to a Transformer encoder, and an MLP on the CLS token predicts the memorability score. The CLS token’s self-attention weights give a readout of where and when the model attends.

Model diagram: T video frames pass through a backbone; the frame features, with position embeddings and a CLS token, enter a Transformer encoder; an MLP head on the CLS output predicts memorability. — Model overview. Frame features from an image backbone, with position embeddings and a CLS token, are processed by a Transformer encoder; the CLS output predicts memorability, and its self-attention scores are used for spatio-temporal analysis.

We compared the model’s attention with human gaze recorded during a video recognition-memory task. Twenty participants watched 140 unique videos from each of Memento10k and VideoMem while binocular gaze was recorded with an SR Research EyeLink 1000 Plus at 500 Hz, and pressed a key when they recognized a repeated video. We compared fixation density maps and model attention maps with AUC-Judd, NSS, CC, and KL divergence, using split-half human agreement as a ceiling and shuffled videos as a chance baseline. Panoptic segmentation let us ask how attention was distributed across semantic categories.

Key findings

A model trained only to predict a memorability score showed spatial attention patterns similar to human gaze, especially for more memorable videos.
The model gave more attention to the initial frames of a video, with no temporal bias built into the architecture. Human fixations were also more consistent across people in early frames than in later ones.
Both the model and human observers gave a larger share of attention to things (well-defined objects such as people or cars) and a smaller share to stuff (background regions such as sky or road) than their occurrence in the frames would predict.

Four line plots of average similarity between model attention and human gaze (AUC-Judd, NSS, CC, and KLD) across four ground-truth memorability bins. — Average similarity between model attention and human gaze across ground-truth memorability bins (n = videos per bin). Higher AUC-Judd, NSS, and CC, and lower KLD, mean closer alignment.

Three panels: temporal attention across frames in normal order, temporal attention with frames reversed, and average optical flow magnitude per frame. — Left: distribution of temporal attention across frames in normal order. Middle: the same when frames are reversed. Right: average optical flow magnitude per frame, which peaks mid-video, so motion does not explain the early-frame emphasis.

Bar charts comparing normalized pixel counts, model attention-weighted counts, and human gaze-weighted counts for stuff and things classes. — Model attention-weighted counts (light blue) and human gaze-weighted counts (orange) relative to normalized pixel counts (dark blue), for 20 stuff and 20 thing classes.

Why it matters

Predicting human memory is not necessarily equivalent to explaining it. By comparing model attention with behavioral measurements, this work asks whether a predictive model recovers meaningful regularities in human visual processing, rather than only reporting prediction accuracy.

The paper also notes limits: existing video memorability datasets have data-leakage issues, and larger, more diverse data are needed to test how well these patterns generalize.

Citation

Kumar, P.*, Khandelwal, E.*, Tapaswi, M., & Sreekumar, V. (2025). Seeing Eye to AI: Comparing Human Gaze and Model Attention in Video Memorability. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2082–2091.

@InProceedings{Kumar_2025_WACV,
    author    = {Kumar, Prajneya and Khandelwal, Eshika and Tapaswi, Makarand and Sreekumar, Vishnu},
    title     = {Seeing Eye to AI: Comparing Human Gaze and Model Attention in Video Memorability},
    booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
    month     = {February},
    year      = {2025},
    pages     = {2082-2091}
}

Figures are reproduced from the paper and project page. Questions about the work: prajneya<at>gmail<dot>com