Watch and Crack: Password Inference from Smart-Glasses Video

RESUMO

Typing passwords on smartphones in public places exposes users to video-based side-channel attacks, in which an adversary records the typing session and reconstructs the entered password by analyzing finger movements. Prior video-based keystroke inference attacks have targeted free-form text on tablets, numeric PINs, pattern locks, and passwords under unrealistically simplified conditions. We present the first general video-based keystroke inference pipeline that recovers rule-based alphanumeric passwords from smartphone QWERTY keyboards, covering all four keyboard layouts and requiring no assumptions about the victim or their device. Using the built-in camera of the popular Meta Ray-Ban smart glasses, our pipeline tracks the device and typing fingertips at sub-pixel resolution, predicts keystrokes via a self-supervised ensemble of neural networks, dynamically estimates the on-screen keyboard layout, and computes per-key probability distributions. These video-derived probabilities are combined with prior password-typing distributions based on 2.19 billion leaked passwords to estimate the rank, and therefore the crack time, of the observed password. In a user study using passwords that adhere to a typical enterprise password policy, human-chosen passwords of up to 16 characters can be cracked in hours to days, and password-manager-chosen random passwords of up to 13 characters in under an hour on a modern GPU, reducing password entropy by up to 60 bits relative to the prior baseline. For human-chosen passwords, combining video evidence with prior statistics yields an additional statistically significant reduction of approximately 20 bits compared to either source alone. The attack succeeds at distances up to 1.8 m, across diverse attacker-victim postures and viewing angles, and generalizes across three popular smartphones.