Your First Agent Evaluation

Capture an approved correction, replay it against a Response agent, and verify a behavior change with MCP tools.

WORK INResponse MCP

BRING

Agent and response IDs · Approved correction · Generation access

FINISH WITH

A saved case and reviewed before/after results

Outcome: one saved evaluation and an observed before/after result for an agent change. This walkthrough uses Response MCP. It does not publish an agent or send a customer message.

An evaluation captures what a good answer should accomplish. Running it generates a new reply and grades its substance against that reference. Saving an evaluation alone does not change instructions, update knowledge, or train a model.

Prerequisites

• A Response MCP connection to the intended organization.
• An existing agent and a real response you are authorized to review.
• A corrected answer approved by someone who knows the applicable policy.
• Read access to responses/evaluations and responses:write for creation and evaluation runs.

Evaluation runs use the real generation path and consume usage. Choose one case first; do not start by replaying your entire history.

1. Inspect the original case

Discover the current tool schemas. Use get_response to inspect the response and confirm its agent and organization before constructing a correction. Preserve the response UUID and agent UUID; do not use the conversation ID or a workflow run ID in their place.

Ask the reviewer to explain which fact, policy, or action was wrong and provide the desired answer. The reference should match the evidence available to the agent. Do not invent an order status or a refund commitment merely to make a more helpful-looking expected answer.

Success: you have a specific response, a policy-backed correction, and an identified agent.

2. Check whether the case already exists

Use list_evals filtered by the response ID. These blocks are MCP tool arguments, not REST request bodies. Replace each angle-bracket placeholder with your actual value before calling.

For list_evals:

{
  "response_id": "<response UUID>",
  "limit": 25,
  "offset": 0
}

Inspect the returned cases and follow pagination if needed. If an existing case already records the approved correction, retain its ID rather than creating another copy.

3. Save the approved correction

For a new case, call create_eval with the actual response ID and the reviewer’s correction:

{
  "response_id": "<response UUID>",
  "agent_id": "<agent UUID>",
  "name": "Shipping policy: avoid an unsupported arrival promise",
  "description": "The original answer promised a date not supported by our shipping policy.",
  "preferred_output": "<approved corrected reply>",
  "reason_type": "wrong_policy",
  "idempotency_key": "<stable unique key for this creation attempt>"
}

The server reads the original message and reply from the stored response. Do not retype them into user_message or non_preferred_output when using response_id. A hypothetical case can instead supply user_message, but keep it distinguishable from a real customer case.

Retain the same idempotency key and input for a retry of this creation attempt. A different correction is not the same request. Read the new case using get_eval with its returned id and verify its reference answer and agent association.

Success: a saved evaluation ID with the intended original question and approved reference.

4. Run the baseline

Call run_eval:

{
  "eval_id": "<evaluation UUID>",
  "agent_id": "<agent UUID>"
}

Inspect the returned eval_id, agent_id, user_message, expected, actual, and grading fields. The implementation returns the run result directly; save the result for comparison rather than assuming a separate persisted evaluation-run history exists.

Result	What it establishes	Next step
graded: true, passed: true	The generated reply passed the grader’s substantive comparison for this case	Inspect the reply and retain the baseline; this case may already pass
graded: true, passed: false	The generated reply failed that comparison	Inspect the actual reply and grading explanation before changing the agent
graded: false, passed: null	The case was not graded	Inspect reason; this is neither a pass nor a failure of agent quality
Tool/request error	The requested run did not return a usable result	Resolve access, input, or service errors before another run

Ungraded reasons can include an unavailable judge, a failed judge request, invalid judge output, or an agent returning no reply. Do not rewrite the reference answer to hide these failures.

5. Change the behavior, then rerun

Find the cause before editing: missing source material, incorrect instructions, an ineffective rule, or missing access to the information needed for the answer. Use the app or an authorized MCP mutation to correct that specific cause.

Inspect the saved change and list_changes, then call run_eval for the same case and agent. Keep the before/after outputs and relevant change identifiers. If the agent or knowledge is already serving customers, understand that editing it can affect live behavior; this walkthrough does not create an isolated agent copy automatically.

A new pass supports this case under the tested configuration. It does not establish that every intent works or that all future generations will match the same wording. Review neighboring cases, such as a missing order identifier and a request for an exception, before expanding use.

6. Check a small suite

Once you have several relevant cases, call run_evals with an explicit limit:

{
  "agent_id": "<agent UUID>",
  "limit": 10
}

The current implementation runs sequentially, defaults to ten cases, and caps a call at twenty. Inspect passed, failed, ungraded, individual run results, and has_more. These counts cover the returned subset, not necessarily every evaluation belonging to the agent.

If has_more is true, enumerate the remaining case IDs through list_evals using its pagination and call run_eval for those you need. Repeating run_evals with the same limit does not advance a cursor. Record how many cases were checked and which remain untested.

Let your assistant perform the walkthrough

Review response [response UUID] for agent [agent UUID] in my Response workspace.
Use this approved correction: [corrected reply and policy reason].
Check for an existing evaluation first. If none matches, I authorize creating
one with a stable idempotency key. Read it back and run it once as a baseline.

Report the saved case ID, actual reply, expected reply, and grading result.
If it fails, propose a targeted change for my review. If it is ungraded, diagnose
that instead. Do not change the agent, publish it, send a customer message,
or launch more evaluations without further authorization.

Keep evaluation systems distinct

Response evaluations belong to the Response agent/response system. Workstream golden cases belong to workflow brands and are evaluated through the engine’s configuration tools. Creating a Response case does not satisfy Workstream’s require_golden_pass gate automatically.

Use the first-week guide to combine evaluations, reviewed live outcomes, and measured improvement without treating them as the same evidence.

Next steps

Review live outcomes

Compare your evaluation evidence with real customer outcomes.

Diagnose a failed answer

Trace missing knowledge, incorrect behavior, or blocked tests.

Related: Response MCP reference.