Replay one eval and grade it against the recorded answer

Runs the real generation path, then grades the reply against preferred_output on substance rather than wording…

POST /api/v1/evals/{id}/run

cURL

curl --request POST \
  --url 'https://response.stateset.com/api/v1/evals/{id}/run' \
  --header "Authorization: Bearer $RESPONSECX_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
  "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
}'

200

{
  "object": "eval.run",
  "eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "user_message": "string",
  "expected": "string",
  "actual": "string",
  "passed": true,
  "graded": true,
  "reason": "judge_unavailable",
  "error": "string",
  "duration_ms": 8,
  "verdict": "pass",
  "equivalence": 1.5,
  "contradicts": true,
  "omits": [
    "string"
  ],
  "differences": "string",
  "explanation": "string"
}

Runs the real generation path, then grades the reply against preferred_output on substance rather than wording. passed is null — never false — when grading could not run, so an unavailable judge is not mistaken for a failing agent.

Path parameters

id · string (uuid) · required

The eval id.

Request body

object

agent_id · string (uuid)

Agent to replay against. Defaults to the eval’s own agent_id; 400 if neither is set.

Response

object

object · string · required

eval_id · string (uuid) · required

agent_id · string (uuid) · required

The agent the eval was replayed against (the request agent_id, else the eval’s own).

user_message · string,null

The replayed customer message. Absent when the run itself failed (reason: 'run_failed').

expected · string,null

The eval’s preferred_output. Absent when the run itself failed.

actual · string,null

The agent’s reply on this run; null when the agent returned no reply. Absent when the run itself failed.

passed · boolean,null · required

True when verdict is pass, false when it is partial or fail, and null when the reply was not graded — ‘the judge was unavailable’ and ‘the agent got it wrong’ are kept distinct.

graded · boolean · required

Whether the judge produced a verdict. When false, reason says why and the verdict fields are absent.

reason · string

Present only when graded is false.

error · string

Present only when reason is run_failed (a run inside a suite threw); the public error message.

duration_ms · integer

Generation plus judge time. Absent when the run itself failed.

verdict · string

Present only when graded is true. Judged on substance, not wording.

equivalence · number,null

Present only when graded is true. 0–5, how closely the substance matches (5 = equivalent); one decimal.

contradicts · boolean,null

Present only when graded is true.

omits · string[]

Present only when graded is true. Essential points from the reference that the reply is missing.

differences · string,null

Present only when graded is true.

explanation · string,null

Present only when graded is true. The judge’s reasoning, so a caller can disagree with the grade.

Status codes

Code	Meaning
200	The replay result: the agent’s reply on this run, graded against the eval’s preferred_output by a reference-based judge. passed is null (not false) when grading did not happen.
400	Invalid request. issues names the offending field(s).
401	Missing, invalid, or expired API key.
403	The key lacks a required scope.
404	Not found, or not in the organization that owns the key.
429	Rate limit exceeded.

Access

Required scope	responses:write
Rate limit	30 requests per minute

cURL

curl --request POST \
  --url 'https://response.stateset.com/api/v1/evals/{id}/run' \
  --header "Authorization: Bearer $RESPONSECX_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
  "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
}'

200

{
  "object": "eval.run",
  "eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "user_message": "string",
  "expected": "string",
  "actual": "string",
  "passed": true,
  "graded": true,
  "reason": "judge_unavailable",
  "error": "string",
  "duration_ms": 8,
  "verdict": "pass",
  "equivalence": 1.5,
  "contradicts": true,
  "omits": [
    "string"
  ],
  "differences": "string",
  "explanation": "string"
}