Reinforcement learning and fine-tuning

Choose SFT or RL, verify rollout and update evidence, isolate validation from test, and assess the current StateSet training workflows.

StateSet Agents trains and evaluates conversational policies. Use supervised fine-tuning for reviewed demonstrations and RL when an executable task or reward can distinguish sampled behaviors. Both need retained input lineage and separate held-out evaluation.

This guide reflects the October 20, 2026 source checkout, including development work beyond the local v0.66.1 release commit. The checkout declares 0.67.0; this page does not verify package publication or a hosted training service.

Choose a training path

Path	Input and purpose	Operational entry point
SFT	Reviewed chat/tool demonstrations; learn the target behavior through supervised loss	Fine-tuning quickstart
Native local RL	Support scenarios, supplied policy facts, and scored sampled trajectories	Terminal studio
Managed River SFT	Assistant-only supervised training with provider-hosted checkpoints	River guide
Native River RL	Stateful sandbox actions, on-policy sampled tokens, and explicit CISPO update evidence	River RL workflow
Industry preparation	Domain recipes and synthetic or reviewed datasets with explicit split roles	Industry fine-tuning

The studio supports local SFT/GRPO/GSPO/DAPO and remote SFT through RunPod, Modal, or River. Other trainer and external-engine surfaces have their own configuration and maturity. A model catalog entry is metadata, not proof that the checkpoint fits the selected hardware or implements every loader capability.

What group-relative RL learns from

For a shared prompt or task, sample multiple trajectories, score their outcomes, and compute advantages relative to that group. The configured estimator determines centering and normalization. GRPO uses token-level policy ratios; GSPO uses sequence-level ratios. A valid rollout needs the actual generated token sequence and behavior likelihoods, with unambiguous task/group ownership.

Current work rejects malformed rollout samples, incomplete/ambiguous groups, and missing required vLLM token log probabilities rather than fabricating them. Native studio RL uses support reward v3, zero-dropout LoRA, a frozen reference, and a positive KL coefficient. Its requested step budget counts committed optimizer updates; a short run is a failure.

A group with identical rewards contains no relative preference signal. Do not interpret nonzero sampled variance as an acknowledged update, or an update as a held-out quality gain. The River study separately records sampled groups, native datums, optimizer acknowledgements, and evaluation outcomes.

Data ownership and split isolation

Capture training source bytes and parsed rows before model loading. Retain source hashes, training-content identity, effective row counts, model revision, seed, settings, and adapter identity. New local SFT completion checks a structurally complete LoRA export and its lineage; it does not establish numerical quality or that every intended update occurred.

Use training data for optimization, validation for checkpoint selection, and fresh test cases for final assessment. Generated industry bundles retain all three roles and reject held-out declarations in training loaders. Exact overlap checks and deterministic audits catch specific leakage; paraphrases, changed annotations, and external contamination still need independent review.

Execution and recovery

stateset-agents finetune schema
stateset-agents finetune validate --config config.json
stateset-agents finetune prepare --config config.json --workspace outputs/finetune

Preparation creates a private snapshot; launch separately binds the retained plan checksum. A successful launch response confirms submission. Inspect terminal state, failure metadata, metrics, and completion receipts. An interrupted supervisor can leave execution unconfirmed; never prepare a replacement paid run merely because its client response disappeared.

A cooperative local stop saves at a safe training boundary. Exact checkpoint recovery needs complete learner state, published receipts, matching code/model/data, a stopped parent, and remaining original budget. Reusing settings, loading model-only weights, and restoring the optimizer are different operations. See checkpoint recovery.

Evaluate before promotion

Local paired evaluations run baseline, adapter, and a pinned independent judge sequentially from cached safetensors models. They retain transcripts and two reversed-order judge passes. A successful case needs both passes to satisfy resolution, accuracy, policy, and non-repetition checks, with no token truncation. This is a fixed-dialogue diagnostic; it does not exercise adaptive customers or real tools.

Validation comparisons rehash named checkpoint bytes and replay judge recordings. Candidates must improve over the base without case-level success, policy, truncation, or required-check regressions. A high mean cannot hide a new failing case. A selected checkpoint needs an explicit nomination and separately reviewed fresh test before its test evaluation.

The recorded multi-seed quality gate checks paired protocols, at least six training seeds, artifact identities, case counts, significance, safety, and pinned model/judge/rubric metadata. One-seed gains stay descriptive. Imported reports do not themselves execute models, verify every external artifact, or approve deployment.

Current measured results

The October 20 native River recovery replication completed six acknowledged one-batch CISPO updates from one shared SFT parent. Candidate test outcomes were 960/960 versus 954/960; the exact seed sign-flip p = 0.125 and bootstrap lower endpoint was zero. The prespecified replication gate failed and no checkpoint was promoted.

Other continuation pilots also retained policy/family regressions despite higher aggregate scores. These synthetic studies support specific execution and evaluation observations; they do not establish general RL superiority, independent merchant results, or production readiness. Read the study details.

Next steps

Start with a prepared local run, choose the studio or JSON agent controls, then retain checkpoint and evaluation evidence.

Source basis: stateset-agents/docs/FINETUNING_TUI.md, FINETUNING_AGENTS.md, TRAINING_BACKENDS.md, INDUSTRY_FINETUNING.md, the rollout/trainer code, and the retained October 20 River recovery replication report. This replaces earlier duplicated examples and ungrounded training-scale and business-impact claims.