Don't Fear the Model Router

Route to the best model with confidence.

Don’t Marry Your Model made the easy argument: stop sending every task to the same model because it won the last bake-off.

Use a cheap model for cheap work. Use a stronger model where the work is actually hard. Keep the routing layer loose enough that swapping providers does not turn your codebase into a shrine.

That was right.

It was also incomplete.

The moment you add a router, you have a new system behavior to test. The question stops being “which model is best?” and becomes “did the system choose the right route, use the right tools, keep the right evidence, and stop at the right time?”

If you do not measure that, your model router is vibes with a dispatch table.

The router is not the answer. The router is a hypothesis about how your system should behave.

Mastra has the surfaces to turn that hypothesis into something testable: scorers, runEvals, datasets, and experiments. The names sound like evaluation infrastructure, and they are. The real value is simpler: they make agent behavior visible enough to argue with.

What are we testing?

The router from the earlier post has three specialist routes:

That table is a start. It is not an eval.

An eval needs examples and scorers:

The important move is to test behavior, not just prose quality.

A model can write a beautiful answer after choosing the wrong specialist. A security agent can produce a plausible report without preserving evidence. A support agent can sound empathetic while skipping the refund policy check. The paragraph is the visible part. The trajectory is where the bugs live.

For a router, I start with four axes:

That last row matters. “Other” is where the production scar tissue lives.

Make the router decision scorable

If the router only produces a final answer, you are guessing about the decision. You can score the output, but you cannot tell whether the route was right.

So give the routing step a small structured contract:

type RouterDecision = {
  route: "code" | "long-context" | "general";
  confidence: number;
  reason: string;
};

Users never need to see this JSON. It can be an internal step, a workflow handoff, or a trace span. The scorer only needs access to it.

Here is a deliberately small Mastra agent that does nothing but choose a route:

import { Agent } from "@mastra/core/agent";

export const routerDecisionAgent = new Agent({
  id: "router-decision-agent",
  name: "Router Decision Agent",
  instructions: `Choose the best specialist route for the user request.

Return ONLY JSON:
{
  "route": "code" | "long-context" | "general",
  "confidence": number,
  "reason": string
}

Routing rules:
- code: implementation, refactoring, debugging, code review, APIs, tests
- long-context: large documents, transcripts, policy synthesis, many files
- general: classification, formatting, extraction, simple Q&A

Do not answer the user request. Only choose the route.`,
  model: process.env.ROUTER_MODEL ?? "openai/gpt-5-mini",
});

Yes, this is a little artificial. Good. Evals reward boring seams.

With the decision explicit, you can test the route before the downstream specialist runs. Router failures stop hiding behind failures in the selected model, its prompt, its tools, or the final-answer scorer.

Write a scorer that catches the boring failure

Mastra’s createScorer accepts plain JavaScript functions, LLM judge prompts, or both. Start with functions whenever the failure is deterministic. They are cheaper, faster, and less mysterious.

Route accuracy does not need a judge model. It needs to parse JSON and compare one field.

import { createScorer } from "@mastra/core/evals";

type Route = "code" | "long-context" | "general";
type RouteGroundTruth = {
  route: Route;
  mustMention?: string[];
};

function textFromAgentOutput(output: Array<{ content?: unknown }>) {
  const content = output[0]?.content;
  return typeof content === "string" ? content : JSON.stringify(content ?? "");
}

function parseDecision(output: Array<{ content?: unknown }>) {
  try {
    return JSON.parse(textFromAgentOutput(output)) as {
      route?: string;
      confidence?: number;
      reason?: string;
    };
  } catch {
    return {};
  }
}

export const validRouterJsonScorer = createScorer({
  id: "valid-router-json",
  description: "Checks that the router emits a valid decision object.",
  type: "agent",
})
  .generateScore(({ run }) => {
    const decision = parseDecision(run.output);
    const validRoute = ["code", "long-context", "general"].includes(
      decision.route ?? "",
    );
    const validConfidence =
      typeof decision.confidence === "number" &&
      decision.confidence >= 0 &&
      decision.confidence <= 1;

    return validRoute && validConfidence && decision.reason ? 1 : 0;
  })
  .generateReason(({ score }) =>
    score === 1 ? "Valid router decision." : "Router output was not valid JSON.",
  );

export const routeAccuracyScorer = createScorer({
  id: "route-accuracy",
  description: "Checks whether the selected route matches ground truth.",
  type: "agent",
})
  .generateScore(({ run }) => {
    const expected = run.groundTruth as RouteGroundTruth;
    const decision = parseDecision(run.output);
    return decision.route === expected.route ? 1 : 0;
  })
  .generateReason(({ run, score }) => {
    const expected = run.groundTruth as RouteGroundTruth;
    const decision = parseDecision(run.output);

    return score === 1
      ? `Selected expected route: ${expected.route}.`
      : `Expected ${expected.route}, got ${decision.route ?? "nothing"}.`;
  });

That scorer is not glamorous. That is the point.

If the router cannot consistently produce valid JSON and pick the obvious specialist on a tiny test set, there is no reason to trust it with production traffic. You do not need a philosopher-model grading ontology. You need a smoke alarm with a battery in it.

Run the small eval loop first

runEvals is the fast loop. Give it a target, test cases, scorers, and a concurrency limit. It runs the target against the data and returns aggregate scores.

import { runEvals } from "@mastra/core/evals";
import { routerDecisionAgent } from "../agents/router-decision-agent";
import {
  routeAccuracyScorer,
  validRouterJsonScorer,
} from "../scorers/route-accuracy";

const routingCases = [
  {
    input: "Refactor this React component to remove duplicated state.",
    groundTruth: { route: "code" },
  },
  {
    input: "Summarize these 14 interview transcripts and find recurring objections.",
    groundTruth: { route: "long-context" },
  },
  {
    input: "Classify this ticket as billing, technical, account, or other.",
    groundTruth: { route: "general" },
  },
  {
    input: "Debug a failing Playwright test that only breaks in CI.",
    groundTruth: { route: "code" },
  },
  {
    input: "Extract the renewal date and contract value from this short paragraph.",
    groundTruth: { route: "general" },
  },
];

const result = await runEvals({
  target: routerDecisionAgent,
  data: routingCases,
  scorers: [validRouterJsonScorer, routeAccuracyScorer],
  targetOptions: {
    modelSettings: { temperature: 0 },
  },
  concurrency: 3,
});

console.log(result.scores);
console.log(result.summary.totalItems);

if (result.scores["valid-router-json"] < 1) {
  throw new Error("Router emitted invalid decision JSON.");
}

if (result.scores["route-accuracy"] < 0.9) {
  throw new Error("Router route accuracy fell below 90%.");
}

This is the loop you run while changing the prompt, adding a route, or trying a cheaper router model.

It is not enough for a mature system. It is enough to prevent the most embarrassing regression: “we changed the router prompt and it started sending classification tasks to the premium code model.”

Keep the axes separate. Route accuracy and final answer quality are different scores. JSON validity, allowed tools, and traceability get their own checks. Do not roll them into one “quality” number. Averages are where useful failures go to retire.

Add an LLM judge only where it earns its keep

Some routing is legitimately ambiguous:

Read these logs and tell me why the deploy failed.

Is that code because it is debugging? long-context because of the logs? general because the user asked for a summary? The right route depends on the tools available and what your product promises.

This is where an LLM judge helps, but only with a tight rubric. Mastra scorers can mix function steps and prompt-object steps. Use functions for structure, then a judge for the part that actually needs judgment.

import { createScorer } from "@mastra/core/evals";
import { z } from "zod";

export const routeReasonablenessScorer = createScorer({
  id: "route-reasonableness",
  description: "Judges whether the route explanation matches the request.",
  type: "agent",
  judge: {
    model: process.env.JUDGE_MODEL ?? "openai/gpt-5-mini",
    instructions: "You are a strict evaluator for model-routing decisions.",
  },
})
  .analyze({
    description: "Evaluate the router's decision rationale.",
    outputSchema: z.object({
      score: z.number().min(0).max(1),
      rationale: z.string(),
    }),
    createPrompt: ({ run }) => `
User request:
${JSON.stringify(run.input)}

Router output:
${JSON.stringify(run.output)}

Score from 0 to 1.

1.0 = route is clearly appropriate and the reason cites the right task signals
0.5 = route is defensible but underspecified or ambiguous
0.0 = route is wrong, unsupported, or the reason is unrelated

Return JSON with { "score": number, "rationale": string }.
`,
  })
  .generateScore(({ results }) => results.analyzeStepResult.score)
  .generateReason(({ results }) => results.analyzeStepResult.rationale);

This scorer costs money because it calls a judge model. That is fine when the judgment is worth it.

Do not use it to check whether JSON parses.

Promote good cases into a dataset

Hard-coded eval arrays are fine at the beginning. Eventually your examples become product assets: the failed customer ticket, the weird support conversation, the prompt injection attempt, the request that routed correctly until last Thursday.

Those belong in a dataset.

Mastra datasets are versioned collections of test cases. Every mutation creates a new version, so you can rerun an experiment against the exact case set that existed when you made a model decision.

Datasets need persistence, so configure storage first:

import { Mastra } from "@mastra/core";
import { LibSQLStore } from "@mastra/libsql";
import { routerDecisionAgent } from "./agents/router-decision-agent";
import {
  routeAccuracyScorer,
  validRouterJsonScorer,
} from "./scorers/route-accuracy";

export const mastra = new Mastra({
  storage: new LibSQLStore({
    id: "router-evals",
    url: "file:./mastra.db",
  }),
  agents: {
    routerDecisionAgent,
  },
  scorers: {
    validRouterJson: validRouterJsonScorer,
    routeAccuracy: routeAccuracyScorer,
  },
});

Then create the dataset and add cases:

import { z } from "zod";
import { mastra } from "../index";

const dataset = await mastra.datasets.create({
  name: "router-decisions-v1",
  description: "Representative model-router decisions for CI and experiments.",
  inputSchema: z.string(),
  groundTruthSchema: z.object({
    route: z.enum(["code", "long-context", "general"]),
    source: z.string().optional(),
  }),
});

await dataset.addItems({
  items: [
    {
      input: "Refactor this React component to remove duplicated state.",
      groundTruth: { route: "code", source: "synthetic:happy-path" },
    },
    {
      input: "Summarize these 14 interview transcripts and find recurring objections.",
      groundTruth: { route: "long-context", source: "synthetic:happy-path" },
    },
    {
      input: "Classify this ticket as billing, technical, account, or other.",
      groundTruth: { route: "general", source: "synthetic:happy-path" },
    },
  ],
});

Once you have a dataset, eval cases stop being throwaway script data. They have IDs, versions, history, and experiment results.

That is when evals stop feeling like “test files for prompts” and start feeling like product memory.

Run experiments against the router

With the dataset in place, dataset.startExperiment() runs it against a registered agent, workflow, or scorer.

import { mastra } from "../index";

const dataset = await mastra.datasets.get({ id: process.env.ROUTER_DATASET_ID! });

const summary = await dataset.startExperiment({
  name: "router-gpt-5-mini-baseline",
  description: "Baseline router decision run before adding security route.",
  targetType: "agent",
  targetId: "router-decision-agent",
  scorers: ["validRouterJson", "routeAccuracy"],
  metadata: {
    routerModel: process.env.ROUTER_MODEL ?? "openai/gpt-5-mini",
    promptVersion: "router-2026-07-03",
  },
  maxConcurrency: 5,
  itemTimeout: 30_000,
  maxRetries: 1,
});

console.log(`${summary.succeededCount}/${summary.totalItems} items succeeded`);

for (const item of summary.results) {
  const scores = Object.fromEntries(
    item.scores.map((score) => [score.scorerId, score.score]),
  );

  console.log(item.itemId, item.output, scores);
}

Now the conversation changes.

Instead of “the new router seems better,” you can say:

• The old router scored 0.94 on route accuracy.
• The new router scored 0.98.
• It improved long-context routing.
• It regressed two code-review cases.
• It reduced premium-model handoffs by 18%.
• It added 300ms of router latency.

That is an engineering conversation. There are tradeoffs on the table, and you can decide whether the trade is worth it.

Score live behavior, but do not confuse it with ground truth

Mastra can also attach scorers directly to agents and workflow steps. Live scorers run asynchronously, store results in your configured database, and support sampling so you do not score every production response unless you mean to.

Useful. Also a different job.

import { Agent } from "@mastra/core/agent";
import { validRouterJsonScorer } from "../scorers/route-accuracy";

export const routerDecisionAgent = new Agent({
  id: "router-decision-agent",
  instructions: "Choose the best specialist route...",
  model: process.env.ROUTER_MODEL ?? "openai/gpt-5-mini",
  scorers: {
    validRouterJson: {
      scorer: validRouterJsonScorer,
      sampling: { type: "ratio", rate: 1 },
    },
  },
});

Live scoring tells you the router is still emitting valid decisions. It catches malformed output, toxic content, forbidden tool calls, missing evidence markers, and suspiciously low confidence.

It usually cannot tell you route accuracy, because production traffic does not arrive with ground truth stapled to it.

Live scoring is monitoring. Dataset experiments are controlled tests. You want both. They answer different questions.

What to measure after route accuracy

Route accuracy is the first rung. It tells you the request reached the expected specialist. It says nothing about whether the specialist did good work.

Once the router passes the basics, score the system in layers:

runEvals supports agent-level, workflow-level, step-level, and trajectory scorer configurations, so you do not have to pretend the final answer is the only artifact.

For a workflow, the shape looks like this:

const result = await runEvals({
  target: supportWorkflow,
  data: supportCases,
  scorers: {
    workflow: [finalAnswerQualityScorer],
    steps: {
      "route-request": [routeAccuracyScorer],
      "check-policy": [policyGroundingScorer],
    },
    trajectory: [expectedPathScorer],
  },
});

That is the mental model I want for agents in production:

Score the decision. Score the path. Score the answer.

If you only score the answer, the model can pass by accident.

The router should get more boring over time

The first routing prompt is usually a paragraph of judgment calls. Fine for a prototype.

As the evals teach you things, parts of the router should become less magical:

• Clear lexical cases become deterministic rules.
• Risky tasks require explicit approval or a workflow branch.
• Ambiguous tasks ask a clarifying question instead of guessing.
• Expensive routes require higher confidence or a second signal.
• Known failure cases become dataset items.

The goal is not to make the router “smarter” forever. The goal is to make the system easier to reason about.

Sometimes that means a better model. Sometimes a tighter prompt. Sometimes a workflow step, a scorer, a hard cap, or a boring if statement that saves you four figures a month.

That is the whole point of measuring behavior. You stop arguing from taste and start arguing from evidence.

A practical starting checklist

If you are building a Mastra router today, start here:

1. Make the routing decision structured, even if users never see it.
2. Write deterministic scorers for valid JSON, expected route, and forbidden routes.
3. Use runEvals with 10 to 20 cases before changing router prompts or models.
4. Promote real failures into a versioned dataset.
5. Run dataset experiments for meaningful prompt, model, route, or workflow changes.
6. Add live scorers for cheap production invariants.
7. Compare experiments by route, not only by average score.

The average matters less than the failure cluster.

If every regression is in long-context policy synthesis, you do not have “a worse router.” You have a route boundary problem. If every failed case uses one specific tool, you have a tool contract problem. If every cheap model fails the same two ambiguous cases, you need escalation logic, not a more expensive default.

This is where evals become useful. They are not a ceremony, or a dashboard that makes everyone feel temporarily adult. They show you which part of the system is failing, so you can fix that part instead of the whole thing.

Resources

• Mastra scorers overview
• Mastra createScorer reference
• Mastra runEvals reference
• Mastra datasets overview
• Mastra dataset experiments
• Don’t Marry Your Model
• Fight Evils with Evals!