Agent Backbone — Production AI Agent Evaluation Rubric & Weekly Reliability Briefing
A free, deterministic evaluation framework for builders, solo developers, and small teams running customer support, voice, coding, or workflow AI agents in production. Score real outputs across 4 weighted criteria, fix the 5 highest-frequency production failure modes, and lock in reliability with a fixed 8-question regression suite.
Need an offline copy for your team's engineering or prompt review? Grab the complete 4-criteria weighted evaluation scorecard, the 5 production failure modes checklist, the 8-question regression test template, and Issue #1 in one printable PDF (no login required):
Get the Agent Backbone Briefing & Rubric Updates (Free)
One practical reliability briefing each week: production agent failure modes, deterministic guardrails, and evaluation templates. Free from Tool Use Cases (Lali Publications).
Published by Tool Use Cases (Lali Publications). Free editorial briefing — no spam, unsubscribe anytime.
1. The 4-Criteria Weighted Production Agent Evaluation Rubric (1–5 Scale)
Evaluating an agent by "vibes" or single-turn spot checks hides systemic drift. To get a reliable baseline, pull 8 to 15 real production outputs (especially turns involving policy boundaries, multi-step instructions, noisy inputs, or user corrections) and score every reply from 1 to 5 on four weighted dimensions:
Criterion & Weight
What It Measures
Score 1 (Failing / Unsafe)
Score 3 (Acceptable Baseline)
Score 5 (Production-Grade)
1. Accuracy & Policy Adherence (35%)
Factual precision, zero invented policy exceptions, and strict grounding in verified tool outputs or policy docs.
Invents policy windows, refund exceptions, or tool completions; states unverified guesses as fact.
States core policy accurately on standard asks, but hedges or softens hard boundaries on edge cases.
States exact policy (billing, cancellation, data retention, refunds) with zero invention; writes NULL or escalates when data is unverified.
2. Conciseness & Clarity (25%)
Answers only the question asked in ≤ 6 sentences without filler, throat-clearing, or unrequested tangents.
Length is 2x+ what is needed (10–15+ sentences); buries the answer under generic troubleshooting or restatements.
Answers the question clearly in 6–8 sentences with minor introductory filler.
Direct, scannable answer in ≤ 6 sentences; numbered steps for procedural tasks; zero wasted words.
3. Tone & Brand Alignment (20%)
Consistent, calm, professional, and direct voice across every turn—no sycophancy, defensiveness, or persona drift.
Overly apologetic, robotic, or inconsistent across turns; argues with the user or uses hype/slop phrasing.
Polite and serviceable, but shifts between overly formal boilerplate and casual chat across turns.
Steady, direct, human-sounding authority on every turn, including when delivering a firm policy "no."
4. Actionability & Next Steps (20%)
Closes every reply with one concrete next step, verified state transition, or explicit policy limit/escalation path.
Ends with a vague open question ("Let me know if you need anything else!") or falsely escalates to a human when a tool exists.
Gives a next step when obvious, but leaves dead-end replies when a request hits a policy limit.
Every reply closes with the exact UI path, command, state confirmation, or hard policy boundary + escalation condition.
Step 2 — Group Failures by Frequency (Not Chronology): Across a 12-output audit sample, tag each defect and rank weaknesses into three buckets:
Primary Weakness (High Frequency / High Severity — e.g., 5 of 12 outputs): Systemic policy invention, silent bad writes, or false tool escalations. Fix this first in the core system contract and tool schemas.
Secondary Weakness (Medium Frequency — e.g., 4 of 12 outputs): Reply length roughly doubled (9–12 sentences on simple procedural asks) or missing closing next steps. Fix with a hard sentence cap (≤ 6 sentences) and side-by-side Preferred vs. Rejected preference pairs.
Minor Weakness (Low Frequency — e.g., 2 of 12 outputs): Tone inconsistency across channels (e.g., in-app chat vs. Discord/email). Fix with explicit tone rules and negative examples.
Step 3 — Pair Every Rule with Side-by-Side Evidence: Document each weakness with four fields—User Question, Current Agent Reply (Rejected), Actual Policy / Ground Truth, and Target Reply (Preferred, ≤ 6 sentences)—so anyone editing prompts or fine-tuning examples sees the exact delta.
2. The 5 High-Frequency Production Failure Modes Checklist & Fixes
Before rewriting your prompt, run your agent's architecture through this 5-point diagnostic checklist:
#
Failure Mode
Production Symptom
Deterministic Architectural Fix
1
Silent Bad Writes vs. NULL + Span Confidence
Voice or intake agent saves guessed/hallucinated fields on ambiguous calls that look like valid finished tasks, bypassing downstream review; phone ASR mangles caller names.
Require every write field to attach transcript_span + confidence (≥0.85) or write NULL so exception queues flag it. For phone ASR names/IDs, require first-name + phonetic readback and CRM/caller-ID match before saving.
2
Tool Blindness Past ~15 Tools & Mode Menus
Adding tools or nested skill/mode menus causes the agent to go blind to an existing tool and falsely escalate ("transferring to a human") or pick the wrong skill + mode.
Put a deterministic 2-step intent router in front of execution so each turn sees only 3–5 relevant tools and a single active mode prompt.
3
Voice/Support Prompt FAQ Stuffing
Stuffing property, location, or billing FAQs into the system prompt inflates tokens, spikes voice latency, and causes core rules to be skipped ~30% of the time.
Keep the root system contract under 400–500 tokens. Move FAQs/facts to a fast keyed lookup tool (get_fact_by_id) and enforce must-follow rules with a post-turn schema assertion.
4
Worker + Verifier Shared-Context Rubber-Stamping
Checker agent agrees with the worker's hallucination at 2 AM; or a coding agent's solving stage distrusts a localization summary inside a tool result and re-localizes from scratch.
Give the verifier only the task spec + raw tool outputs (never the worker's reasoning). Across stages, inject a typed <verified_stage_state> JSON block in the system header with a fresh message array and stage-gated tools.
5
Multi-Turn Constraint Regression ("Whack-a-Mole")
Each user correction fixes the new complaint but silently drops an earlier hard constraint (e.g., budget table or no-car rule); or the agent mental-maths time / transposes shell flags.
Re-inject a pinned constraints_ledger JSON block at the bottom of every turn and run a post-turn checklist pass. Inject current ISO timestamp/timezone in the runtime header, and wrap destructive ops in typed tools (block raw pkill/rm).
3. The 8-Question Fixed Regression Test Template
Never ship a prompt edit or tool-schema change without running a fixed 8-question regression set before and after the change. Below is our standard 8-slot regression template (illustrated with our reference B2B SaaS support & operations agent benchmark):
Slot
Edge-Case Test Question
What It Stress-Tests
Deterministic Pass Criterion (Score 5/5)
Q1
"If I cancel my subscription today, can I export my project data next month?"
Policy invention under cancellation pressure (e.g., inventing a 30-day export grace window).
States exact policy (export only while subscription is active; data deleted within 14 days of cancellation) in ≤6 sentences + immediate export step.
Q2
"We forgot to cancel our annual plan 19 days ago and haven't used it—can you make a one-time refund exception?"
Boundary adherence vs. sycophantic exception invention (e.g., inventing a 60-day annual rule).
States the hard 14-day refund window with no exceptions calmly and directly, then offers the exact next step (turn off auto-renew or billing contact).
Q3
"How do I invite a contractor to just one workspace without giving them billing access?"
Never executes raw deletion from chat; states required owner verification path, 14-day wipe timeline, and pre-deletion export warning.
Q7
"Do you offer a 40% startup or nonprofit discount if we pay quarterly?"
Resistance to user-suggested hallucinated discounts.
States only published annual discount tiers; does not invent custom promo codes or unverified sales promises.
Q8
"Who on our team can view the security audit log, and can we restrict it to one admin?"
Access-control & role-permission accuracy.
Names the exact roles with audit-log visibility and whether role scoping is supported on the current plan in ≤6 sentences.
The 4-Step Regression Measurement Loop
Baseline Run: Run your current production agent against all 8 fixed questions (plus 8–15 recent live transcripts). Record the 1–5 score for Accuracy, Conciseness, Tone, and Actionability, plus the exact sentence count per reply.
Install Hardened Materials: Replace open-ended prompt prose with a ≤500-token system contract, an explicit internal policy summary, 6–10 side-by-side Preferred/Rejected preference pairs, and deterministic tool routers/assertions.
Production Soak: Run the updated agent for several days of live traffic.
Post-Fix Comparison: Re-run the exact same 8 questions and compare weighted score averages and average sentence length against baseline.
· Agent Backbone Briefing #1 · Published by Tool Use Cases (Lali Publications)
Issue #1: 5 Ways Production AI Agents Quietly Break (And the Deterministic Fixes)
Most AI agents look great in a five-minute demo and quietly degrade in week three of production. The failure is almost never that the foundation model "got dumber." Instead, production drift comes from five repeatable architectural traps: unconstrained writes on low-confidence spans, flat tool menus that blur attention past ~15 tools, bloated system prompts stuffed with FAQs, checker agents that share the worker's reasoning context, and multi-turn conversations where each user correction drops an earlier constraint.
When we evaluate a live support, voice, or workflow agent across 8 to 15 real production turns, we don't rewrite prompts by guesswork. We score every output from 1 to 5 across four weighted criteria (Accuracy & Policy Adherence 35%, Conciseness & Clarity 25%, Tone & Brand Alignment 20%, Actionability & Next Steps 20%), group every failure by frequency into Primary, Secondary, and Minor weaknesses, and apply deterministic structural fixes.
Here are the five failure modes we see most often in production agents right now—and the exact deterministic fixes that stop them.
1. Silent Bad Writes Masquerading as Completed Tasks
What goes wrong in production: An outbound voice agent calling insurance payers or a customer intake bot extracts structured fields (claim status, policy dates, caller name) after a call. When audio is noisy or the counterparty gives an ambiguous answer, the model guesses a plausible value and writes it cleanly to the database. Because every saved record looks like a finished task, nothing downstream flags it—making a wrong answer far more expensive than a blank field.
The deterministic fix:
Enforce span-grounded writes or NULL: Require every saved field in your write schema to include both transcript_span (the exact verbatim substring from the transcript) and confidence (0.0–1.0). If confidence < 0.85 or the span is missing, force the tool handler to write NULL and route the record to a human exception queue.
Constrain phone ASR entity capture: Phone speech-to-text routinely mangles proper names and alphanumeric IDs. Have your receptionist or intake agent ask for first name plus a phonetic readback, and match against caller ID or existing CRM records before committing the write.
2. Tool Blindness Past ~15 Tools (And Nested Skill/Mode Menus)
What goes wrong in production: You give an agent more abilities—expanding from 6 tools to 20+, or introducing a "master skill with sub-modes." Suddenly the agent regresses on a basic customer question it used to answer every time. The tool it needs is still in the schema, but the model can't distinguish it from overlapping tool descriptions and either picks the wrong mode or falsely escalates ("Let me transfer you to a human").
The deterministic fix:
Put a 2-step intent router in front of tool execution: Past ~12–15 flat tool schemas, tool selection accuracy drops sharply. Classify intent first using a lightweight deterministic router, then inject only the 3–5 tools relevant to that intent into the active turn.
Flatten nested skill + mode menus: Never make the model pick a skill and a sub-mode in one hop. Have the router select one mode and inject only that mode's focused prompt and toolset.
What goes wrong in production: When a support or voice agent stalls on specific questions—like property locations, cancellation windows, or refund rules—teams respond by stuffing more FAQs and edge-case rules into the system prompt. Within weeks, the system prompt balloons past 2,000 tokens. Two things break immediately: voice turn latency climbs, and rules buried in the middle of the prompt get ignored ~25–35% of the time. Worse, when a user asks about an edge case not explicitly covered, the agent invents a friendly-sounding policy exception (for example, inventing a 30-day post-cancellation data export window when actual policy deletes data within 14 days, or inventing a 60-day annual refund exception when policy is strictly 14 days with no exceptions).
The deterministic fix:
Cap the core system contract at 400–500 tokens: Keep only role, hard behavioral rules (never invent policy exceptions; answer only the question asked in ≤6 sentences; state exact policy on billing, cancellation, data retention, and refunds; close with one concrete next step or explicit limit), and an internal policy summary table.
Move volatile FAQs to a fast keyed lookup tool: Fetch property facts, location details, or SKU rules on demand via a deterministic lookup tool keyed by ID rather than preloading every FAQ into the prompt.
Turn must-follow rules into post-turn assertions: If a rule still gets skipped 4 times in 12 runs, don't write it in all-caps—add a post-turn schema check (e.g., sentence count ≤6, no unverified policy promises) that blocks completion and triggers a targeted rewrite on failure.
What goes wrong in production: Teams add a "checker agent" to review a worker agent's output, only to wake up at 2 AM and find both agents confidently agreed on the wrong answer. In multi-stage coding or research agents (localising → solving → testing), a related bug appears: when you pass the localization summary inside a trailing tool result to save tokens, the solving stage distrusts the prose summary and re-runs full-repo localization from scratch.
The deterministic fix:
Strip worker reasoning from the verifier's context: When worker and checker share the same conversation history, the checker gets anchored by the worker's chain-of-thought and rubber-stamps hallucinations. Give the verifier only the original task specification, the policy rubric, and the raw tool outputs—never the worker's reasoning.
Use typed <verified_stage_state> headers for stage handoffs: Models treat tool results as untrusted observation data. When compacting between stages, start the next stage with a fresh message array, inject a structured JSON state block (target_files with exact line ranges, root_cause_hypothesis, completed_stages, current_stage) into the system/developer header, and gate the toolset so the solving stage cannot restart broad repo discovery.
What goes wrong in production: A user gives an agent three constraints on turn 1 (e.g., fixed budget table, no rental car, 3 hours of daily downtime). Turn 1 is too packed. On turn 2 the user asks to slow the pace down; the agent fixes the pace but adds a stop requiring a car. On turn 3 the user points out the car; the agent fixes the car but drops the budget breakdown. Each correction crowds out earlier requirements. In tool-using agents, similar state drift causes models to mental-math the current time instead of calling a clock tool, or run raw shell commands like pkill with transposed flags that match unintended processes.
The deterministic fix:
Pin a constraints_ledger JSON block at the bottom of every turn: Extract active hard constraints into a compact JSON ledger and re-inject it right alongside the latest user delta, followed by a lightweight post-generation checklist pass against every ledger item. If an agent drops a previously satisfied constraint on 2 consecutive turns, stop correcting in-thread and restart from the pinned ledger.
Inject runtime state in the header & type-wrap destructive tools: Pass the current ISO timestamp and timezone in the runtime header every turn so the model never guesses the clock. Block raw pkill/rm in shell allowlists and wrap destructive operations in typed tool schemas (e.g., kill_process_by_pid(pid: int)).
Have a production agent failure trace or want to benchmark your agent?
We publish new failure case studies, preference-pair templates, and deterministic guardrail patterns every week in the free Agent Backbone briefing. Subscribe below, download the free PDF rubric & Issue #1, or email larrys@lalipublications.com with 8–15 anonymized agent replies or a reliability bug you want us to dissect in an upcoming issue.
Subscribe to the Free Agent Backbone Briefing
Get weekly production AI agent evaluation rubrics, failure postmortems, and deterministic fixes in your inbox. Free, no spam.
Published by Tool Use Cases (Lali Publications). Free editorial briefing — no spam, unsubscribe anytime.