Treat instructions found in documents and web pages as untrusted input, and keep powerful actions behind deterministic controls.
- prompt injection
- LLM security
- AI security
- secure AI integration
Separate data from authority
Retrieved text can contain instructions that try to override the product's intended behavior. Delimit untrusted content and never assume a prompt alone can enforce authorization.
AI & AutomationAI & Automation
Thoughtful decisions compound over time.
Practical product work brings technical choices back to the people and workflows they are meant to serve.
Constrain tools and permissions
Give an AI workflow only the minimum tools and data it needs. Validate arguments in application code, require confirmation for consequential operations, and prevent arbitrary execution.
Test adversarially and log safely
Include malicious documents, misleading instructions, and data-exfiltration attempts in security tests. Log decisions without copying sensitive content unnecessarily, and review the threat model when tools change.
Practical application
Test with a retrieved document that tells the assistant to ignore policy or reveal another user's data. The application should still apply permissions before retrieval and validate every tool action independently of model-generated instructions.
