StateSet Agents

The RL improvement control plane: turn an agent’s production conversations into a training set, and prove the retrained model got better.

Your agent already produces conversation logs. StateSet Agents treats them as a training set: it grades every turn against a reward, curates the ones that worked, trains on those, and evaluates the result against prompts the model never saw.

The framework is stateset-agents on PyPI. Multi-turn generation exists in several trainer libraries now; what this keeps coherent is the whole loop — traces, rewards, training, and a deployed model with lineage back to the conversations that produced it.

Earlier revisions of these docs called this framework grpo-agent-framework, which is not a published package. The install is pip install stateset-agents.

The loop

production logs ──▶ ingest ──▶ grade ──▶ curate ──▶ train ──▶ evaluate
      ▲                                                          │
      └──────────────── deploy with lineage ─────────────────────┘

Each step is a command, and each writes a file the next one reads — so you can stop after any of them, inspect what happened, and resume.

Step	Command	Produces
Ingest	stateset-agents ingest	Per-conversation transcript JSONL
Grade and curate	stateset-agents improve run	curated.jsonl, improve_summary.json, next_steps.md
Train	stateset-agents train or train-remote	An adapter, plus eval_results.json
Talk to it	stateset-agents chat / chat-remote	An interactive session against the result

The whole loop, end to end:

pip install stateset-agents

stateset-agents ingest --format openai --input logs.jsonl --output transcripts/
stateset-agents improve run --transcripts transcripts/ --reward customer_support --output improved/
stateset-agents train-remote --provider runpod --dataset improved/curated.jsonl \
  --base-model meta-models/Muse-Glimmer-30B --eval-prompts held_out.txt --max-cost 5

next_steps.md is worth reading rather than skipping: the improve step prints the exact training command for what it just curated, so step 3 is a copy rather than a decision.

Rewards

Grading is rule-based and offline. Four rewards ship:

Reward	Grades
customer_support	Support conversations — resolution, tone, escalation behaviour
tool_calling	Whether the right tool was called with the right arguments
gsm8k	Grade-school maths, as a correctness baseline
nsr	Decisions against a symbolic policy — see Verified decisions

There are deliberately no LLM-judge rewards in this list. A judge needs an API key and a network call per turn, which makes grading non-reproducible and turns a local loop into a billed one. Judge-based evaluation exists in the framework for benchmarking; it is not part of the offline improve loop.

Algorithms

GRPO is the default and the one the quickstart uses. Six other objectives ship as trainers in stateset_agents.training, each with its own config and the same reward and environment interfaces, so switching is a one-line change:

Trainer	Import	Reach for it when
GRPOTrainer, MultiTurnGRPOTrainer	stateset_agents	The default — group-relative advantages, no critic
GSPOTrainer	stateset_agents.training.gspo_trainer	You want GRPO’s stability at the sequence level; GSPOConfig and train_with_gspo come with it
GEPOTrainer	stateset_agents.training.gepo_trainer	Heterogeneous or distributed groups — expectation-based rather than sample-based
DAPOTrainer	stateset_agents.training.dapo_trainer	Reasoning-heavy tasks — decoupled clipping and dynamic sampling
VAPOTrainer	stateset_agents.training.vapo_trainer	Math and reasoning — value-augmented group optimization; fits a 16 GB card as of 0.42.5
PPOTrainer	stateset_agents.training.ppo_trainer	A standard PPO baseline to compare against
RLAIFTrainer	stateset_agents.training.rlaif_trainer	Reward from a judge model rather than a hand-written function

All seven run under one measured protocol: the 0.42.5 release reports a 15-run matrix across five of them on matched hardware, and 0.42.4 adds framework-neutral comparison gates against TRL and measured 1/2/4/8-GPU weak and strong scaling. The benchmark harness that produced those numbers ships in the repository, so a claim about one algorithm beating another is reproducible rather than quoted.

Training backends

The framework does not implement its own trainer. It drives version-pinned backends behind a fail-closed adapter — PPO, GRPO and GSPO through OpenRLHF, PPO/GRPO through verl, and distributed maths GRPO through NeMo RL.

Every backend adapter is fail-closed: a version it has not been conformance-tested against refuses to run rather than proceeding on the assumption that the interface is unchanged. Live GPU conformance is an explicit evidence gate, not something an upgrade inherits. If a backend refuses after you bump a dependency, that is the gate doing its job.

Next steps

Quickstart

Logs to a retrained model, with no GPU of your own.

CLI reference

Every command, and which ones cost money.

Agents API

The REST gateway: agents, conversations, training jobs, and Anthropic/OpenAI-compatible inference.

MCP server

Drive the improve loop from Claude Code or your own agent.

Agent objectives

Choosing what to reward before you train on it.