A chatbot hallucination is read by a human who decides whether to act. An agent hallucination may be executed against a real system before any human sees it. Safety in agents is a core design requirement — not a nice-to-have.
Malicious content in the agent's environment (a web page, email, document, API response) contains text designed to look like instructions. A vulnerable agent executes these embedded instructions as if they were legitimate goals. Defenses: clear prompt structure separating instructions from environment content, output filters for anomalous actions, and human review of unexpected behaviors.
Minimal footprint — Request only necessary permissions. Scope API keys to specific operations. Do not persist sensitive data beyond the session. Limits blast radius of a compromised or malfunctioning agent.
Irreversibility tiering — Reversible, low-risk actions: execute autonomously. Partially reversible: log for review. Irreversible, high-risk (delete files, send emails, charge cards): pause and require explicit human confirmation before proceeding.
Goal specification — Specify goals precisely, with explicit constraints. Vague goals enable specification gaming — the agent achieves the letter of the goal while violating the intent.
Human oversight at scale — Tiered review (human for high-risk, automated for low-risk), anomaly detection, full audit logging, and capability gating (new capabilities require explicit approval). Treat oversight as an engineering problem.
Ask the AI assistant about agent safety, prompt injection attacks, the minimal footprint principle, or how to design human-in-the-loop checkpoints.