Config evaluate

Eval scores for the candidate config, with a baseline comparison when one was supplied.

POST /v1/brands/{brand_id}/config/evaluate

cURL

curl --request POST \
  --url 'https://api.workstream.stateset.com/v1/brands/{brand_id}/config/evaluate' \
  --header "x-api-key: $STATESET_TEMPORAL_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
  "config": {},
  "cases": [
    {
      "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "channel": "string",
      "message": "Two-person tent, green — replacement for damaged pole set.",
      "context": {},
      "gold": "string"
    }
  ],
  "provider": "string",
  "model": "string",
  "baseline": {},
  "pass_threshold": 1.5,
  "simulate_tools": true
}'

200

{
  "summary": {
    "total": 102,
    "passed": 1,
    "pass_rate": 7.5,
    "avg_score": 7.5,
    "by_dimension": {},
    "flags": {},
    "regressions": [
      {
        "case_id": null,
        "baseline_score": null,
        "candidate_score": null,
        "delta": null
      }
    ],
    "action_fidelity": {}
  },
  "cases": [
    {
      "case_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "draft": "string",
      "tool_calls": [],
      "action_fidelity": "string",
      "verdict": {
        "score": null,
        "passed": null,
        "dimensions": null,
        "rationale": null,
        "flags": null
      }
    }
  ],
  "baseline_summary": {
    "total": 102,
    "passed": 1,
    "pass_rate": 7.5,
    "avg_score": 7.5,
    "by_dimension": {},
    "flags": {},
    "regressions": [
      {
        "case_id": null,
        "baseline_score": null,
        "candidate_score": null,
        "delta": null
      }
    ],
    "action_fidelity": {}
  }
}

Path parameters

brand_id · string (uuid) · required

Request body

object

config · object

cases · object[] · required

provider · string

model · string

baseline · object

pass_threshold · number

simulate_tools · boolean

Response

object

summary · object · required

Show summary

total · integer · required

passed · integer · required

pass_rate · number · required

avg_score · number · required

by_dimension · object · required

flags · object · required

regressions · object[] · required

Show regressions

case_id · string · required

baseline_score · number · required

candidate_score · number · required

delta · number · required

action_fidelity · object

Present only when non-empty

cases · object[] · required

Show cases

case_id · string · required

draft · string · required

tool_calls · string[] · required

action_fidelity · string

verdict · object · required

Show verdict

score · number · required

passed · boolean · required

dimensions · object · required

rationale · string · required

flags · string[] · required

baseline_summary · object

Show baseline_summary

total · integer · required

passed · integer · required

pass_rate · number · required

avg_score · number · required

by_dimension · object · required

flags · object · required

regressions · object[] · required

Show regressions

case_id · string · required

baseline_score · number · required

candidate_score · number · required

delta · number · required

action_fidelity · object

Present only when non-empty

Status codes

Code	Meaning
200	Eval scores for the candidate config, with a baseline comparison when one was supplied.
400	Validation error
401	Unauthorized
404	Not found
429	Rate limited
500	Internal error

cURL

curl --request POST \
  --url 'https://api.workstream.stateset.com/v1/brands/{brand_id}/config/evaluate' \
  --header "x-api-key: $STATESET_TEMPORAL_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
  "config": {},
  "cases": [
    {
      "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "channel": "string",
      "message": "Two-person tent, green — replacement for damaged pole set.",
      "context": {},
      "gold": "string"
    }
  ],
  "provider": "string",
  "model": "string",
  "baseline": {},
  "pass_threshold": 1.5,
  "simulate_tools": true
}'

200

{
  "summary": {
    "total": 102,
    "passed": 1,
    "pass_rate": 7.5,
    "avg_score": 7.5,
    "by_dimension": {},
    "flags": {},
    "regressions": [
      {
        "case_id": null,
        "baseline_score": null,
        "candidate_score": null,
        "delta": null
      }
    ],
    "action_fidelity": {}
  },
  "cases": [
    {
      "case_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "draft": "string",
      "tool_calls": [],
      "action_fidelity": "string",
      "verdict": {
        "score": null,
        "passed": null,
        "dimensions": null,
        "rationale": null,
        "flags": null
      }
    }
  ],
  "baseline_summary": {
    "total": 102,
    "passed": 1,
    "pass_rate": 7.5,
    "avg_score": 7.5,
    "by_dimension": {},
    "flags": {},
    "regressions": [
      {
        "case_id": null,
        "baseline_score": null,
        "candidate_score": null,
        "delta": null
      }
    ],
    "action_fidelity": {}
  }
}