OPTIMIZE

Move from "looks right" to measured quality —
score your agent locally and see exactly where it's grounded and where it slips. No deployment.

Eval-Fix Loop Scenario Simulation agents-cli eval LLM-as-judge

Objective

Run the local eval loop: simulate the crisis at scale, then turn the scenarios into a graded eval suite and score the agent with an LLM judge — reading exactly where it's grounded and where it slips, all locally, no deployment. This is the difference between a demo and a production agent.

Everything here runs locally

M4 is about quality, not shipping. You already deployed in M2 — here you evaluate and improve the agent on your machine with agents-cli eval and an LLM judge, with no redeploy. Publishing the improved agent happens later.

The Quality Flywheel

The Quality Flywheel is the official loop for agent quality improvement. Each revolution raises quality empirically.

Why This Works

Simulation creates coverage — many scenarios catch failures your one test query never will
Graded metrics remove subjectivity — quality becomes a number you can define, measure, and compare
The weakest case reveals the fix — you target a real failure, not a guess
Flagged sentences point to the fix — the judge shows exactly which sentences weren't grounded, so you know what to tighten

Example: Per-Case Scores (illustrative)

Case	`hallucinations_v1`	What it means
cancellation + reroute	1.00	every sentence grounded in tool output
delay	0.92	one mild reassurance not from a tool
503 fallback	1.00	correctly defers to station boards
grounding trap (platform)	0.50	invented a platform no tool returned

The low score on the grounding trap points straight at the fix — a strict rule to state only tool-returned facts. In practice you'd make that one change and re-run; here we stop at the measurement to keep the step fast. Your real numbers come from running adk eval once and reading the score table.

Optimize in 3 Steps

2 min

The Local Eval-Fix Loop

Quality improvement for agents is a loop, not a one-time check — and it all runs locally:

Evaluate — run the agent against many scenarios and score it
Analyze — find the weakest case and why it failed
Fix — change the agent's instructions or tool logic to address it
Re-eval & compare — rerun and prove the score improved

This is how you go from "it works on my one test query" to "it works on every query" — with no deployment in the loop.

6 min

Simulate the Crisis at Scale (Breadth)

Test the agent against many scenarios, not just one query. AGY generates diverse commuter requests and an edge case where the disruption feed goes down — and runs them all locally, in parallel.

Paste into AGY Module 4 Step 2: simulate the crisis at scale. First, use the google-developer-knowledge MCP to read up on running an ADK agent programmatically — the Runner with InMemorySessionService / InMemoryMemoryService — so you use current APIs. 1. Generate 10 commuter scenarios during a St Pancras signal failure, including a case where the disruption feed is unavailable (HTTP 503). 2. Run all 10 against my agent LOCALLY. Drive each scenario through the ADK Runner (runner.run_async(...), consuming its event stream) with InMemorySessionService / InMemoryMemoryService so the memory callback doesn't error outside Agent Runtime. Do NOT call the model directly (e.g. generate_content / a raw google-genai loop): the agent has a BuiltInCodeExecutor (added in M2), which disables Automatic Function Calling — only the ADK Runner's own event loop will actually invoke the function tools, so a raw-genai path will hang or return no tool calls. 3. Run the scenarios sequentially in one asyncio event loop, and wrap each scenario in a per-scenario timeout (asyncio.wait_for, e.g. 90s) as a hang guard: if one scenario times out, record it as TIMED‑OUT and continue so the run always finishes. (Don't run each scenario in its own thread/process — that makes aiohttp crash with "attached to a different event loop" errors.) Do not deploy — local only. 4. When done, PRINT a final report table: each scenario, what the agent did, which tools it used, and whether it resolved sensibly (or TIMED-OUT).

Expected Result

AGY writes a local parallel runner (e.g. tests/integration/parallel_test.py) and runs all 10 scenarios concurrently (~2–5 min), then prints a report table. Coverage includes:

Cancellations + reroutes — e.g. cancelled Paris/Amsterdam services, alternatives computed via Brussels/Rotterdam
Delays — revised departure times with next steps
Code-exec math — the sandbox computes stats like the cancellation ratio or peak bottleneck hour
Personalization — home-station recall
Offline-feed fallback — a 503 outage; the agent falls back to GTFS schedules and tells the user to check station boards

This step is breadth — a qualitative check that the agent resolves sensibly. We turn it into measured scores next.

6 min

Grade Your Agent Locally

Now turn the scenarios into a graded eval suite and score the agent with an LLM judge — locally, in one run. The per-case scores show exactly where the agent is solid and where it slips, and the judge quotes the ungrounded sentences so you know what you'd tighten next.

Paste into AGY Module 4 Step 3: grade the agent locally. First, use the google-developer-knowledge MCP to read up on ADK evaluation (adk eval, the evalset format, and the eval config criteria), so you build the graded eval the current way. Build a small graded eval and score my agent once. Keep it fast, do not deploy. Do NOT modify the agent's code (app/agent.py, app/tools.py, etc.) — grade the real agent as-is, with no mocking or tweaks, or the score is meaningless. Before the steps below, install the eval dependencies so adk eval can run: uv sync --extra eval (or: uv pip install "google-adk[eval]"). Without them, adk eval fails with a missing-dependencies error. 1) Build a 4-case evalset in ADK format at tests/eval/evalsets/rushhour.evalset.json. The file needs a top-level "eval_set_id" field plus eval_cases (each a conversation with user_content + a reference final_response, plus session_input) — adk eval fails without eval_set_id. Cases: - cancellation + reroute - a delay - the offline-feed 503 fallback - a grounding trap asking for something no tool returns (e.g. "which platform does the 18:16 to Paris depart from?"). 2) Write tests/eval/eval_config.json gating on the hallucinations LLM judge (scores the fraction of the answer's sentences grounded in the tool outputs, continuous 0-1, reference-free): { "criteria": { "hallucinations_v1": { "threshold": 0.8, "judge_model_options": { "judge_model": "gemini-2.5-flash", "num_samples": 1 } } } } (gemini-2.5-flash is reliably enabled in lab projects; if you want a faster judge and it's enabled, you can switch to gemini-2.5-flash-lite.) 3) Finish writing BOTH files above before you run anything. Then run the eval once in the FOREGROUND and wait for it to finish — do NOT run it as a background task, or you'll lose the stdout score table. Use adk eval directly: uv run adk eval app tests/eval/evalsets/rushhour.evalset.json \ --config_file_path tests/eval/eval_config.json 4) STOP and PRINT the per-case score table exactly as adk eval prints it to stdout — do NOT hand-parse the *.evalset_result.json files to compute scores. For any case scoring below the 0.8 threshold, open its *.evalset_result.json under app/.adk/eval_history/ ONLY to quote the sentences the judge flagged as unsupported/contradicted, so I can see exactly what isn't grounded. Don't fix or re-run -- just report. No deploy.

How adk eval scores (while you wait)

adk eval scores each case against the criteria in eval_config.json (agents-cli eval wraps it, but its flags vary by build — so we call adk eval directly). We gate on hallucinations_v1, an LLM judge that scores the fraction of the answer's sentences grounded in the tool outputs — a continuous 0–1 score that's reference-free (no hand-written answer to game). The other three below exist, but we don't gate on them here.

Metric	What it measures	Speed
`hallucinations_v1` (gated)	Fraction of sentences grounded in the tool outputs — continuous 0–1, reference-free, so it directly measures "did the agent invent anything?"	LLM judge
`final_response_match_v2`	Semantic match to a reference answer (LLM judge) — but majority-vote binary per single-turn case, so it can't show fine-grained movement here	LLM judge
`response_match_score`	Lexical closeness to the reference (ROUGE) — fast, but scores correct-but-free-form answers low, so we don't gate on it	fast
`tool_trajectory_avg_score`	Right tools, right order — but an EXACT match on tool name AND args, so it's brittle for a generative agent (re-runs vary the args) and we don't gate on it	not gated

The run prints a per-case score table to stdout (and writes a full *.evalset_result.json under app/.adk/eval_history/ with the per-sentence judge labels you can quote for any low-scoring case).

Docs: ADK evaluation · Criteria reference · User simulation

Expected Result

AGY builds the 4-case evalset, runs adk eval once, and prints the per-case hallucinations_v1 scores (the fraction of each answer's sentences grounded in the tool outputs).

Per-case scores — well-grounded cases score near 1.0; a case that slips in an unsupported sentence (an invented platform, or a reassurance no tool returned) scores lower
Flagged sentences — for any case below the threshold, AGY quotes the exact sentences the judge marked unsupported/contradicted, so you see precisely what isn't grounded
No fix, no deploy — this step measures; tightening the instruction to ground those sentences is the next turn of the same loop, which you'd run in practice

What just happened: quality became a number, and the fine-grained, reference-free judge pointed at exactly which sentences weren't grounded — the measurement that drives the eval-fix loop. (Improvement isn't guaranteed from one tweak, and the judge is non-deterministic — so in practice you iterate; here we stop at the measurement to keep the step fast.)

Mentor Checkpoint — Done When:

10 scenarios were simulated against your agent locally
A graded eval suite produced per-case scores (adk eval + hallucinations_v1)
You can see which cases are well-grounded and which slip, with the judge's flagged sentences — no deployment

Module Recap — OPTIMIZE

Beat 1 What You Typed

Step	The Prompt
4.2 Simulate	"Generate 10 commuter scenarios during a St Pancras signal failure and run them in parallel against my agent locally (incl. a disruption-feed outage). Print a report."
4.3 Grade	"Build a small graded eval and score my agent once with an LLM judge — show me the per-case scores, no deploy."

Beat 2 What Ran Under the Hood

Step	What ran
4.2	A local parallel runner (ADK `Runner`) executing 10 scenarios concurrently
4.3	`adk eval` for scoring (`agents-cli eval` wraps it), with the `hallucinations_v1` LLM judge grading each case's groundedness

The analysis is done by the coding agent reasoning over the scores — there is no auto-optimize command. You read the per-case scores and the flagged sentences, and you stay in control of any change.

Beat 3 What You Can See Locally

Eval scores — per-case, per-metric, saved as JSON (and an HTML report)
Trace viewer — adk web Eval tab: drill into a case and see exactly where the agent went wrong
Per-case scores — the adk eval score table plus the judge's flagged sentences, showing exactly where the agent is grounded
Flagged sentences — the judge's per-sentence labels showing what wasn't grounded

Beat 4 Why It Matters

Empirical quality — measured, not guessed
Fast local feedback — measure quality without a deploy in the loop
Pinpoint the weakness — the judge flags the exact ungrounded sentences, so you know precisely what to fix next
Regression-ready — the eval suite is reusable: re-run it whenever you change the agent to catch quality sliding