The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability

RESUMO

Properties safety monitors certify, like cross-tenant noninterference and sandbagging detection, are 2-safety hyperproperties witnessed only by two executions, making single-trace monitoring fundamentally limited. Nine LLM monitors are optimal when no leak signal exists but capture little signal as detectability grows; naming what to check closes 61% of the shortfall, showing the gap is mostly information and procedure, not capability. Two frontier LLM judges certified an earlier version of the authors' own benchmark as sound despite a directional bias invalidating three findings.