Mutagent

Your agent development system

The agents that build, test, evaluate, and improve your agents.

Get access Book a meeting
Design partner session
Bene · Dorian · Burak  |  2026

Building an agent used to take a quarter.
Now it takes a weekend.

Claude Code, Cursor, and the coding agents collapsed idea to prototype.

TIME TO BUILD AN AGENT 2023 a quarter 2024 a few weeks 2026 a weekend

Anyone on your team can ship an agent now.

Building one is easy. Making it production-grade is not.

Production-grade comes down to three numbers. Every agent in production lives or dies on the three C's.

01
Credibility
Does the output actually answer the user correctly? Accurate, grounded, and on-policy.
02
Containment
How much the agent handles end to end, without a human. Low containment means it can't run on its own.
03
Cost
What each run costs in tokens and compute. Waste hides and compounds at scale.

The value is buried in your data.
Our agents help you lift it.

A diagnostics and evaluation system built from your own traces turns improvement into a loop that runs on its own, moving credibility, containment, and cost.

By hand, the lifecycle is one slow loop — and you are inside it.

Every improvement is a hand-run experiment — change something, generate outputs, and a human evaluates whether it helped. Each experiment → evaluate cycle takes weeks, and the corrections never compound.

implement change generate samples tweak by handprompt · tools · logic ⏱ THE BOTTLENECK 👤 manually evaluatea human hand-reads a sample weeks per cycle one slow cycle · nothing compounds
Human-gated
Every change is judged by an engineer reading outputs — slow, subjective, hard to reproduce. The loop only moves as fast as human hours.
Slow even with signal
Even with traces and evals, the loop still runs at review speed — root-cause → fix → confirmed result is a human chain that takes weeks.
Can't scale
Every new capability needs the same manual cycle — so improvement is gated on human hours, not the agent.
THE SHIFT Stop being the loop — start operating it. Humans set direction; agents run the cycles.

The AI Agent Factory

One orchestrator runs every stage end-to-end: work enters on the left, code PRs and agent / skill updates come out on the right.

SOURCES COLD START New feature or intent EXISTING FEATURE Bug, incident or enhancement bug / incident → *diagnose · enhancement → *spec / *optimize MUTAGENT ORCHESTRATOR — end-to-end ADLC automation *specDefine +Design *buildBuild *evalEvaluate *shipRelease *monitorMonitor *diagnoseDiagnose *optimizeOptimize deploy LIVE · PRODUCTION Code PR Agent update Skill update live events / traces re-evaluate the variant → *eval
Design
Get the architecture and the problem breakdown right. Every later stage pays for what Design gets wrong.
Evaluate
Build the evaluation system. Without it, nothing downstream can run on its own.
Diagnose
Turn production failures into root causes. This is where debugging stops being manual.

The whole lifecycle, running agentic.

1
Define + Design
PM defines why and what good looks like. AI Engineer turns it into the Spec.
2
Build
The Coding Agent generates the agent against the Spec.
3
Create Eval System
Historic data and traces fill the dataset and the criteria.
4
Optimization loop
Run the dataset, evaluate, optimize. Every cycle ships a better version.
5
Deploy
Go live with a feedback mechanism in place.
6
Monitor
Live evals run on production traces. Failures surface as they happen.
7
Optimize
Each failure feeds the eval system and triggers the offline loop. A new version redeploys.
BUSINESS CONTEXT SPEC v1 AGENT v1 — accuracy — cost / run EVAL SYSTEM dataset eval-criteria HISTORIC DATA Evaluate Optimize Deploy Monitor live evals Diagnose live data 0 / 100 OFFLINE ONLINE Build

It runs in your environment — cloud and local.

Mutagent installs as agents inside the coding agent you already use. The agents run locally — your traces and code never leave your machine — while reading from and writing to your own platforms.

YOUR ENVIRONMENTyour platforms · cloud + local
SOURCES
evidence in
LangfuseOpenTelemetryDatadogLangSmithBraintrustSigNoz
runs inside your coding agent● LOCAL — data never leaves
Orchestratorconducts the lifecycle — one agent per stage
*specAgentSpecRESEARCH PREVIEW
*buildBuilderRESEARCH PREVIEW
*evalEvaluatorRESEARCH PREVIEW
*shipagent · WIP
*monitorMonitorRESEARCH PREVIEW
*diagnoseDiagnosticsRESEARCH PREVIEW
*optimizeOptimizeRESEARCH PREVIEW
6 agents in research preview
TARGETS
where it writes
code
GitHub
coding agents · .md
Claude CodeCodexCursor
frameworks
MastraDeepAgents
cloud-managed
📎ClaudeVercel
↻generates
new traces
Targets run in production and generate the next round of traces — monitor scans them and the orchestrator runs the loop again.
TODAY · RUNS LOCALLY · EMBEDDED IN HARNESS
The agents run inside your coding agent — your data never leaves your machine.
FUTURE · CLOUD PLATFORM
A hosted Mutagent platform or Managed Agents-as-a-Service — self-host or fully managed, when you want it.

Move credibility, containment, and cost. We improve your system together.

What you get
Customised agents running in your stack, tuned to your domain
An eval framework and dataset built from your own traces
The improve loop live on your production traffic
Your team trained and owning the system by week 12
How it works
01Agent Lifecycle Assessment · half a day. Map your lifecycle, find the gaps, lock the goal and success metric.
02Co-develop · 12 weeks, weekly with the founders. We build the missing agents into your stack; immediate value for your team.
03Handover. Your team owns and runs the system; we stay on call.
Highly customised to your use case. We take a small number of partners, only a few seats.

Diagnostics, on a real email agent.

One internal generation loop drove 89% of traces and $156.74 of cost in a 24-hour window. Found, traced to the exact step, and ranked, automatically.

Mutagent Diagnostics Agent analysing an email agent over a 24-hour window
See the full diagnosis

Built by operators.

Bene
BeneCEO
PhD, 6 patents. Staff TPM at Cruise. CPO at Beam AI.
Burak
BurakCTO
Designed Beam AI's agent platform. His autonomous coding system built Mutagent's MVP in 4 weeks.
Dorian
DorianCPO
Led 13 forward-deployed engineers and the agent lifecycle at Beam AI.

One engine. A team of agents.

IncidentNew FeatureQuestionCI/CDDiagnosticsEvaluatorMutationExperimentBuildTelemetryKnowledge / ContextEvalsAgentsRoot cause: missing tool descriptionAGENT DNA4 / 64 commits

See it work on one
of your own agents.

A 30-minute working session. We run discovery on a real agent of yours and you see the first analysis live. No setup on your side, no commitment.

Pick a time  ·  bene@mutagent.io
Phase 1 · Conceptualize

Define why, design how — into one spec.

Capture the intent and, critically, what “good” means; then shape the agent that delivers it. The signed spec is what every later stage runs against.

Define
Why we're building it, the intent, and the bar for “good” — the acceptance criteria the agent will be judged on.
Design
The agent's shape — the routines, tools, and decision logic that turn that intent into behaviour.
business context intent / goal constraints SPECv1 what “good” means signed ✓
THE CONTRACT The signed spec is the single source of truth every later phase runs against — Build writes to it, Evaluate grades against its “good,” and Optimize has to beat it.
Phase 2 · Build

Your coding agent generates the agent.

The signed spec drives a coding agent — Claude Code, Codex, Cursor, Pi, Hermes, whichever you run — that writes the agent itself. The result is portable: the same agent runs on any harness.

SPEC
v1
→
YOUR CODING AGENT
Claude CodeCodexCursorPiHermes
writes the agent against the spec
→
AGENT v1
RUNS ON
THE SAME AGENT RUNS ON ANY HARNESS
local · in your coding agent
Claude CodeCodexCursorPiHermes
cloud · managed
VercelMastraClaude AgentsDeepAgents
Phase 3 · Evaluate

Make “good” measurable.

Derive a dataset of cases and a set of criteria — each a binary check, so a fail points straight at the broken dimension. Roll them into one 0–100 success rate the loop can chase.

EVALS · the criteria
Guided — cold start
Define criteria with the team from the spec and a few examples — no data needed yet.
Discovered — from traces
Derive criteria from production traces — successes say what to keep, failures what to check for.
DATASETS · the cases
Synthesize — from ground truth
Generate cases from the spec, historical exports, or known-good examples — before you have traffic.
Distill — from traces
Sample real production runs into a representative test set — the cases that actually happen.
EVAL SYSTEM
dataset · pass · fail per case
criteria — each a binary check
Cites its sourcePASS
No fabricated factsPASS
Correct tool usedFAIL
Valid output formatPASS
Policy respectedPASS
success rate
0/ 100
Phase 3 · Evaluate · what you grade

An agent is a whole run — so you grade every layer.

Most agent failures never show up in the final answer. They hide in the path it took, the tools it called, the context it had, and the harness it ran on. A good agent eval taps each layer.

TRAJECTORY — was the whole path sound? ✓ context tool call … tool call final output ✓context ✓tool output ✓tool output ✓output HARNESS — the engine the run executes on ✓ ⚙ tools ▤ skills ⚓ hooks ☷ sub-agents
Trajectory
the path it took
Did it reach the goal by a sound route — no needless detours, loops, or skipped steps? A right answer down a broken path still fails.
Tool calls & outputs
right tool, used right
The right tool with the right arguments — and the result actually read and used correctly, not ignored or misquoted.
Context
input + runtime context
Did it have — or fetch — the right information, and keep the window clean? Most “hallucinations” are really missing context.
Harness
the engine & runtime
The framework and runtime the agent runs on — does the orchestration execute the steps, tools, and state correctly?
Phase 3 · Evaluate · the craft

What makes an eval worth trusting.

A score only drives the loop if you can trust it. Good evals share four properties — and there's a craft to building them.

THE ONE RULE The agent being tested never grades itself — the evaluator is independent. Otherwise it just rubber-stamps its own work.
FOUR PROPERTIES OF A GOOD EVAL
✓Falsifiable
Can it be wrong?
A concrete pass/fail an answer could actually fail — not a vibe.
✓Reproducible
Low variance · deterministic judge
Same input → same verdict. Fix the judge (model, prompt, seed) so the score measures the agent, not grader noise.
✓Valid
Does passing matter?
Tied to a real outcome users feel — not a proxy that just looks nice.
✓Actionable
A failure localizes the cause
When it fails, it names the broken dimension — so the next cycle knows exactly what to change.
HOW YOU GET THERE — DEFINING GOOD EVALS
Lift from real cases
Criteria pulled from real runs + the spec — not guessed.
→
One clear bar per case
A concrete “must be true” — the thing that makes it falsifiable.
→
Calibrate to experts
Tune the judge until it matches expert labels.
→
Weight by impact
Failures that change the outcome outweigh cosmetic ones.
ACTIONABLE vs NOT ACTIONABLE
×NOT ACTIONABLE · a number
“Helpfulness: 7.3 / 10”
A generic 1–10 judge. The number drifts run to run, and when it moves you learn nothing about what to change.
vs
✓ACTIONABLE · a check
“Cites a real source per claim?” ✗
A concrete pass/fail. When it fails, the failure names the exact gap — so the next cycle knows what to fix.
TAKEAWAY An eval is a tool, not a report card — pick the one that tells you what to fix, not the one that hands you a number.
Phase 4 · Diagnose

Failures cluster into root causes — and become new evals.

Live, the agent is watched. Failing traces are grouped by root cause, each cause becomes a finding, and each finding is fed back as a new eval — so the system keeps growing itself.

live production traces cluster root causes missing tool / context7 wrong escalation4 format / schema3 findings → new evals + eval: assert tool present + eval: escalation rule + eval: schema check EVAL SYSTEMgrows every cycle absorbed
Phase 4 · Diagnose · at scale

From a flood of failures to a short list of causes.

You can't read a million failing traces. Diagnosis clusters them by similarity and categorizes each group — so you fix the biggest cause first, not the loudest symptom.

~millions of failing traces
→
cluster ·
categorize
a few ranked causes
Missing context
1.2k
Wrong tool use
880
Off-path / looping
610
Unsupported claim
430
Format / policy miss
210
▦Cluster
Group the failing traces that look alike into one bucket — no human reads them one by one. 1,240 traces → one cluster
→
▤Categorize
Label each cluster by the kind of failure, so every bucket maps to a fixable cause. missing context · wrong tool · off-path
→
★Rank
Order the causes by impact — fix the biggest one first, not the loudest symptom. 1.2k cause → top of the queue
Phase 5 · Optimize

Test variants against the evals — only a winner ships.

Eval-driven development: keep looping build → eval → optimize until a candidate clears the bar and beats the live version. That's the gate that makes every cycle compound.

*build *eval SUCCESS GATE clears the bar? score ≥ bar · beats live PASS ✓ FAIL × *shipv+1 *optimize pass fail iterate
Only a win shipsA variant must clear the eval bar and beat the live version — no regression ever reaches production.
Every pass raises the barThe shipped v+1 becomes the new baseline the next candidate has to beat — the bar only goes up.
The loop compoundsbuild → eval → optimize runs on its own, so gains stack instead of resetting every cycle.
Appendix · Diagnostics example

The full diagnosis.

Diagnostics Agent: problem, evidence table, and why-chain for the email agent
Evidence and why-chain
Diagnostics Agent: signal census, latency heatmap, and ranked findings for the email agent
Findings and remedies
← back to evidence
Lifecycle  stage 1 to 3  Define · Design · Build

Most of the cost is decided before any code runs.

Challenge
Solution
Architecture
Three hard things, rushed: picking the right architecture, breaking the problem down, and aggregating the information the system needs. The cost shows up in every later trace.
The Spec decides all three deliberately, a framework-agnostic contract written before any code.
The handoff
Agents are never fully specced. Relevant information is not aligned, and the PM-to-engineer handoff is not orchestrated, so the agent gets re-defined loop after loop.
The Spec is the orchestrated handoff: one contract the PM and the AI engineer both sign off, knowledge gathered and aligned.
Existing knowledge
The information you already have, historic cases, documents, internal knowledge, sits unused. Defining the agent starts from a blank page when it could start from what exists.
Mutagent uses your historic cases to extract the logic, and aggregates your documents and knowledge automatically, so the spec is built from what you already know.
One sign-off gate. The right architecture, the right breakdown, the right information, before any code.
Lifecycle  stage 4  Evaluate

An evaluation system is more than a metric.

Challenge
Solution
Judge drift
An LLM-as-a-judge drifts. Run it twice, get two answers. There is no repeatable, reliable score to trust.
A calibrated evaluator, validated against a labelled holdout, so the score is stable and trustworthy.
Generic evals
Generic evals grade generic quality. They do not know your agent's real job, your domain, or what "correct" means.
Targeted evals, written against your actual output and input schemas. Per-key rubrics, four dimensions.
Inputs
Inputs are often incomplete and never measured. A missing input becomes a wrong output, and the eval blames the agent for it.
Mutagent measures input failures too, so a bad output is traced back to a bad input, not mis-attributed to the agent.
The dataset
You have historic data and production traces, but no dataset. Turning them into labelled test cases is manual and slow.
Mutagent turns your historic data and traces into a dataset, expanded with synthetic cases and expert-labelled.
This is the leverage point. An evaluation system built for your agent makes everything after it automatic.
Lifecycle  stage 5 to 6  Deploy · Monitor

Ship the MVP, then close the loop.

Challenge
Solution
Inner loop
Shipping is not the end. The inner loop, fast iteration on your own data, only closes if you can measure performance cleanly.
A feedback interface, a feedback API and a human-in-the-loop review, so every run is scored and feeds the next optimization.
Outer loop
The outer loop ties production back to value. Customer-success metrics, engagement, downstream extraction errors, none of it is associated with the agent that caused it.
Wire the outer loop: associate business outcomes and downstream failures back to agent performance, and feed them in as new cases.
Monitor
Teams track the wrong things, generic evals and dashboards, so a real problem never becomes visible.
Monitor against your calibrated eval suite, tied to the success metric. A problem shows up the moment it appears.
The MVP ships the moment it clears the quality bar, regression-gated. Then the loop runs.
Lifecycle  stage 7 to 8  Diagnose · Optimize

Diagnose the failure. Optimize the fix.

Challenge
Solution
Diagnose
Thousands of traces, and no idea how to analyze them to find the problem.
Intelligent filtering and querying across all traces. Tiered checks, cheap code-level first, deep LLM reads only where needed.
Persistence
Diagnosis is not persistent. Every investigation starts from zero and gets redone.
Each diagnosis extracts data back into monitoring and grows the evaluation dataset. The failure index compounds, company-specific.
Optimize
A fix needs a clean regression set, and the optimization loop must close cleanly, or you cannot trust the improvement.
A clean regression set and a closed optimization loop: every fix validated, regression-gated, before it ships.
Every cycle makes the next diagnosis faster, and the eval system stronger.

See it run.

The Mutagent optimizer. This is the actual product, optimizing an invoice-extraction agent.
Mutagent 01 / 19