Promptfoo: LLM evals & red teaming

promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps. Stop using trial-and-error... start shipping secure, reliable agents

Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed. Read the company update.

npm install -g promptfoo
promptfoo init --example getting-started

Also available via brew install promptfoo and pip install promptfoo. Or use npx promptfoo@latest to run any command without installing.

Most LLM providers require an API key. Set yours as an environment variable:

export OPENAI_API_KEY=sk-abc123

Run an eval and view results:

cd getting-started
promptfoo eval
promptfoo view

See Getting Started (evals) or Red Teaming (vulnerability scanning) for more!

Test your prompts and models with automated evaluations
Secure your LLM apps with red teaming and vulnerability scanning
Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
Automate checks in CI/CD
Review pull requests for LLM-related security and compliance issues with code scanning
Share results with your team

Here's what it looks like in action:

It works on the command line too:

We welcome contributions! Check out our contributing guide to get started.

Join our Discord community for help and discussion.

promptfoo