Safety Monitoring

Real-time monitoring blocks unsafe model actions and flags boundary violations for human review during product use, internal agents, training, and evaluations.

Business impact

• boundary violations — Google reports that the monitor blocks severe boundary violations before they happen.
• flagged serious cases — Google flags serious internal cases for human responders.
• unauthorized actions — Anthropic deployed a classifier to identify attempts to probe or escape a testing environment or unexpectedly obtain internet access.

Data requirements

• agent actions (text) — Google’s monitor reads the agent’s actions in real time.
• scratchpads (text) — Google’s monitor reads the model’s internal scratchpad or step-by-step chain of thought.
• evaluation transcripts (text) — Anthropic ran automated monitors over transcripts from internal evaluations.
• tool-call activity (text) — Anthropic blocks actions before the tool call is run.

AI methods and techniques

• classification — Anthropic uses a classifier to identify probing, escape attempts, and unexpected internet access in real time.
• monitoring — Google runs an automated misalignment monitor over actions, scratchpads, and evaluation activity.
• real-time detection — Both implementations watch behavior as it happens so they can intervene before an unsafe action completes.

Industries

Real-world evidence

2 documented case studies on record.

Companies using this: Google, Anthropic.