> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stateset.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Run an agent's eval suite (regression check)

> Each eval costs a real generation plus a judge call, so runs are sequential and capped per call; has_more reports when the suite was longer than the cap.

Each eval costs a real generation plus a judge call, so runs are sequential and capped per call; has\_more reports when the suite was longer than the cap.

### Path parameters

<ParamField path="id" type="string (uuid)" required>
  The agent id.
</ParamField>

### Request body

`object`

<ParamField body="limit" type="integer">
  How many of the agent's evals to run (newest first). Capped at 20 per call.
</ParamField>

<ParamField body="status" type="string">
  Only run evals with this `status`.
</ParamField>

### Response

`object`

<ResponseField name="object" type="string" required />

<ResponseField name="agent_id" type="string (uuid)" required />

<ResponseField name="total" type="integer" required>
  Number of evals run (`runs.length`).
</ResponseField>

<ResponseField name="passed" type="integer" required>
  Runs with `passed: true`.
</ResponseField>

<ResponseField name="failed" type="integer" required>
  Runs with `passed: false`.
</ResponseField>

<ResponseField name="ungraded" type="integer" required>
  Runs with `passed: null`.
</ResponseField>

<ResponseField name="has_more" type="boolean" required>
  True when more evals exist for the agent than were run — the suite stopped at `limit`.
</ResponseField>

<ResponseField name="note" type="string">
  Present only when `has_more` is true: states that only the first N were run.
</ResponseField>

<ResponseField name="runs" type="object[]" required>
  <Expandable title="runs">
    <ResponseField name="object" type="string" required />

    <ResponseField name="eval_id" type="string (uuid)" required />

    <ResponseField name="agent_id" type="string (uuid)" required>
      The agent the eval was replayed against (the request `agent_id`, else the eval's own).
    </ResponseField>

    <ResponseField name="user_message" type="string,null">
      The replayed customer message. Absent when the run itself failed (`reason: 'run_failed'`).
    </ResponseField>

    <ResponseField name="expected" type="string,null">
      The eval's `preferred_output`. Absent when the run itself failed.
    </ResponseField>

    <ResponseField name="actual" type="string,null">
      The agent's reply on this run; null when the agent returned no reply. Absent when the run itself failed.
    </ResponseField>

    <ResponseField name="passed" type="boolean,null" required>
      True when `verdict` is `pass`, false when it is `partial` or `fail`, and null when the reply was not graded — 'the judge was unavailable' and 'the agent got it wrong' are kept distinct.
    </ResponseField>

    <ResponseField name="graded" type="boolean" required>
      Whether the judge produced a verdict. When false, `reason` says why and the verdict fields are absent.
    </ResponseField>

    <ResponseField name="reason" type="string">
      Present only when `graded` is false.
    </ResponseField>

    <ResponseField name="error" type="string">
      Present only when `reason` is `run_failed` (a run inside a suite threw); the public error message.
    </ResponseField>

    <ResponseField name="duration_ms" type="integer">
      Generation plus judge time. Absent when the run itself failed.
    </ResponseField>

    <ResponseField name="verdict" type="string">
      Present only when `graded` is true. Judged on substance, not wording.
    </ResponseField>

    <ResponseField name="equivalence" type="number,null">
      Present only when `graded` is true. 0–5, how closely the substance matches (5 = equivalent); one decimal.
    </ResponseField>

    <ResponseField name="contradicts" type="boolean,null">
      Present only when `graded` is true.
    </ResponseField>

    <ResponseField name="omits" type="string[]">
      Present only when `graded` is true. Essential points from the reference that the reply is missing.
    </ResponseField>

    <ResponseField name="differences" type="string,null">
      Present only when `graded` is true.
    </ResponseField>

    <ResponseField name="explanation" type="string,null">
      Present only when `graded` is true. The judge's reasoning, so a caller can disagree with the grade.
    </ResponseField>
  </Expandable>
</ResponseField>

### Status codes

| Code  | Meaning                                                                                                                                                            |
| ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `200` | The suite result. Evals run sequentially; a run that throws is recorded as an ungraded `eval.run` with `reason: 'run_failed'` rather than failing the whole suite. |
| `400` | Invalid request. `issues` names the offending field(s).                                                                                                            |
| `401` | Missing, invalid, or expired API key.                                                                                                                              |
| `403` | The key lacks a required scope.                                                                                                                                    |
| `404` | Not found, or not in the organization that owns the key.                                                                                                           |
| `429` | Rate limit exceeded.                                                                                                                                               |

### Access

|                |                        |
| -------------- | ---------------------- |
| Required scope | `responses:write`      |
| Rate limit     | 30 requests per minute |

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url 'https://response.stateset.com/api/v1/agents/{id}/evals/run' \
    --header "Authorization: Bearer $RESPONSECX_API_KEY" \
    --header 'Content-Type: application/json' \
    --data '{
    "limit": 10,
    "status": "pending"
  }'
  ```
</RequestExample>

<ResponseExample>
  ```json 200 theme={null}
  {
    "object": "eval.run_summary",
    "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "total": 102,
    "passed": 1,
    "failed": 1,
    "ungraded": 1,
    "has_more": true,
    "note": "Two-person tent, green — replacement for damaged pole set.",
    "runs": [
      {
        "object": "eval.run",
        "eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
        "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
        "user_message": "string",
        "expected": "string",
        "actual": "string",
        "passed": true,
        "graded": true,
        "reason": "judge_unavailable",
        "error": "string",
        "duration_ms": 8,
        "verdict": "pass",
        "equivalence": 1.5,
        "contradicts": true,
        "omits": [],
        "differences": "string",
        "explanation": "string"
      }
    ]
  }
  ```
</ResponseExample>