Checkpoint

Teams shipping LLM agents have unit tests for their code and almost nothing for the agent. The failures that matter are not exceptions. They are a tool called with the wrong argument, a policy boundary quietly crossed, a multi-turn conversation that drifts. None of that shows up in a green build.

I co-founded Checkpoint with Ayushman Gupta and Aaditya Gaur and led engineering as CTO. The bet was that agents need a pre-production failure surface: somewhere to break the loop on purpose, before a user does it by accident. I'm at Recurly now and run it at lower intensity, but the product is live in private beta and the thesis hasn't changed.

Engineers submit an agent config: prompts, tools, schemas. Checkpoint generates adversarial multi-turn test suites across five categories, runs them against stateful mocked tools in a sandbox, and scores the transcripts with an LLM judge working from a structured rubric rather than string matching.

1. 01

   Test generation across happy paths, edge cases, adversarial prompts, policy boundaries, and ambiguous inputs.

2. 02

   Synthetic environments with mocked tool calls that hold state across turns, so multi-step failures are reproducible.

3. 03

   LLM-judged scoring against structured rubrics, which catches semantic regressions that assertion-based tests miss.

• Adversarial generation is the easy half. Deciding what counts as a failure, and making that judgment reproducible, is the actual product.

• Stateful tool mocking mattered more than model choice. Most interesting agent bugs only appear on turn three or later.

• A strong application signal is not a business. Ranking well told us the problem was legible; it didn't tell us anyone would pay yet.