Run an agent's eval suite (regression check)

Each eval costs a real generation plus a judge call, so runs are sequential and capped per call; has_more reports when the suite was longer than the cap.

POST /api/v1/agents/{id}/evals/run

cURL

curl --request POST \
  --url 'https://response.stateset.com/api/v1/agents/{id}/evals/run' \
  --header "Authorization: Bearer $RESPONSECX_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
  "limit": 10,
  "status": "pending"
}'

200

{
  "object": "eval.run_summary",
  "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "total": 102,
  "passed": 1,
  "failed": 1,
  "ungraded": 1,
  "has_more": true,
  "note": "Two-person tent, green — replacement for damaged pole set.",
  "runs": [
    {
      "object": "eval.run",
      "eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "user_message": "string",
      "expected": "string",
      "actual": "string",
      "passed": true,
      "graded": true,
      "reason": "judge_unavailable",
      "error": "string",
      "duration_ms": 8,
      "verdict": "pass",
      "equivalence": 1.5,
      "contradicts": true,
      "omits": [],
      "differences": "string",
      "explanation": "string"
    }
  ]
}

Each eval costs a real generation plus a judge call, so runs are sequential and capped per call; has_more reports when the suite was longer than the cap.

Path parameters

id · string (uuid) · required

The agent id.

Request body

object

limit · integer

How many of the agent’s evals to run (newest first). Capped at 20 per call.

status · string

Only run evals with this status.

Response

object

object · string · required

agent_id · string (uuid) · required

total · integer · required

Number of evals run (runs.length).

passed · integer · required

Runs with passed: true.

failed · integer · required

Runs with passed: false.

ungraded · integer · required

Runs with passed: null.

has_more · boolean · required

True when more evals exist for the agent than were run — the suite stopped at limit.

note · string

Present only when has_more is true: states that only the first N were run.

runs · object[] · required

Show runs

object · string · required

eval_id · string (uuid) · required

agent_id · string (uuid) · required

The agent the eval was replayed against (the request agent_id, else the eval’s own).

user_message · string,null

The replayed customer message. Absent when the run itself failed (reason: 'run_failed').

expected · string,null

The eval’s preferred_output. Absent when the run itself failed.

actual · string,null

The agent’s reply on this run; null when the agent returned no reply. Absent when the run itself failed.

passed · boolean,null · required

True when verdict is pass, false when it is partial or fail, and null when the reply was not graded — ‘the judge was unavailable’ and ‘the agent got it wrong’ are kept distinct.

graded · boolean · required

Whether the judge produced a verdict. When false, reason says why and the verdict fields are absent.

reason · string

Present only when graded is false.

error · string

Present only when reason is run_failed (a run inside a suite threw); the public error message.

duration_ms · integer

Generation plus judge time. Absent when the run itself failed.

verdict · string

Present only when graded is true. Judged on substance, not wording.

equivalence · number,null

Present only when graded is true. 0–5, how closely the substance matches (5 = equivalent); one decimal.

contradicts · boolean,null

Present only when graded is true.

omits · string[]

Present only when graded is true. Essential points from the reference that the reply is missing.

differences · string,null

Present only when graded is true.

explanation · string,null

Present only when graded is true. The judge’s reasoning, so a caller can disagree with the grade.

Status codes

Code	Meaning
200	The suite result. Evals run sequentially; a run that throws is recorded as an ungraded eval.run with reason: 'run_failed' rather than failing the whole suite.
400	Invalid request. issues names the offending field(s).
401	Missing, invalid, or expired API key.
403	The key lacks a required scope.
404	Not found, or not in the organization that owns the key.
429	Rate limit exceeded.

Access

Required scope	responses:write
Rate limit	30 requests per minute

cURL

curl --request POST \
  --url 'https://response.stateset.com/api/v1/agents/{id}/evals/run' \
  --header "Authorization: Bearer $RESPONSECX_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
  "limit": 10,
  "status": "pending"
}'

200

{
  "object": "eval.run_summary",
  "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "total": 102,
  "passed": 1,
  "failed": 1,
  "ungraded": 1,
  "has_more": true,
  "note": "Two-person tent, green — replacement for damaged pole set.",
  "runs": [
    {
      "object": "eval.run",
      "eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "user_message": "string",
      "expected": "string",
      "actual": "string",
      "passed": true,
      "graded": true,
      "reason": "judge_unavailable",
      "error": "string",
      "duration_ms": 8,
      "verdict": "pass",
      "equivalence": 1.5,
      "contradicts": true,
      "omits": [],
      "differences": "string",
      "explanation": "string"
    }
  ]
}