Promptfoo: Make AI Quality a Repeatable Check
A CLI and library for evaluating and red-teaming AI applications, with declarative test cases and CI integration. Start with failures that matter to your product.
I want an AI change to come with a repeatable answer to a simple question: what got better, and what broke? Promptfoo is useful infrastructure for making that answer part of development.
The MIT-licensed CLI and library evaluate prompts, models, agents, and RAG applications using configured test cases and assertions. They support comparisons across providers, automated red-team testing, and CI integration. A prompt or model change can run against the same cases instead of depending on whichever examples someone remembers to try.
I would begin with a small set of real product failures: a wrong answer, an unsupported claim, a bad tool decision, or an instruction the application should resist. Define how each result will be judged, run a baseline, and review the failures when the system changes. Model-graded assertions still need calibration; a passing evaluation only says something about the cases and checks you chose.
Plan the data flow explicitly. Running the CLI locally can still involve external model providers, hosted result sharing, or Promptfoo’s remote red-team generation service. Basic usage telemetry is enabled by default and has a documented opt-out. The remote-generation documentation describes alternatives when that service cannot be used. I would establish those boundaries before putting sensitive examples into an evaluation suite.
The payoff is a shared standard for deciding whether a change is ready to ship, with a failure set that improves as the team learns.
