How Braintrust uses AI agents, evals, and CI to ship better software | Ankur Goyal

06/15/26
Source: Lenny’s Newsletter by Claire Vo. Read the original article

TL;DR Summary of How AI Agents and Evals Revolutionize Engineering Workflows with Braintrust CEO Ankur Goyal

AI agents can now handle complex technical architecture and infrastructure tasks beyond human capability, enabling exhaustive benchmarking and optimization. The “agent line” framework helps teams delegate decisions effectively to AI, improving engineering velocity. Additionally, evals serve as dynamic, scalable tools for measuring AI product quality and encoding design preferences, transforming how teams maintain rigorous standards without manual intervention.

Optimixed’s Overview: Unlocking Engineering Efficiency with AI Agents and Evals in Modern Software Development

Revolutionizing Engineering with AI-Driven Benchmarking and Delegation

In a deep technical discussion with Ankur Goyal, founder and CEO of Braintrust, we explore how coding agents are transforming engineering workflows in top-tier tech companies like Notion, Stripe, and Zapier. These AI agents are capable of tackling highly complex tasks such as benchmarking database indexes and optimizing query execution—processes previously too exhaustive for manual human effort.

Key insights include:

  • Automated Benchmarking: Using OpenAI’s Codex, agents can perform week-long benchmark experiments tirelessly, providing detailed insights into performance tuning and infrastructure improvements.
  • Agent Line Framework: A practical approach to deciding which decisions and interactions can be delegated to AI agents, ensuring human engineers focus on high-impact work.
  • Concurrent Agents Workflow: Running multiple agents simultaneously in foreground and background to maximize throughput and accelerate problem-solving.

Demystifying Evals as the New Standard for AI Product Quality

Evals are positioned as the modern equivalent of Product Requirement Documents (PRDs), encoding “what good looks like” so AI models can autonomously determine “how” to meet quality standards. Ankur demonstrates how to build scoring functions that capture subjective qualities such as designer taste, enabling quality to scale beyond individual human reviewers.

  • Live Demo: Creating evals for documentation accuracy and user experience to replace unreliable “vibe checks.”
  • Scaling Quality: Transforming subjective design preferences into repeatable, automated evaluations.
  • CI/CD Integration: Emphasizing the critical role of continuous integration pipelines in boosting engineering velocity for AI-accelerated teams.

Practical Strategies for AI-Enhanced Engineering Leadership

The conversation also covers how senior engineers and technical leaders can leverage AI to improve throughput without sacrificing rigor. Ankur shares prompting strategies for when agents fail, and discusses the importance of investing in infrastructure to support AI-driven development. The episode offers actionable frameworks and insights designed specifically for senior engineering roles aiming to harness AI’s full potential.