Showcase  /  Benchmark

We trained a model with reinforcement learning on an Affordance app, from 40% to 77%, on exactly the tokens it saw

Reinforcement learning lets a model get better at a task by trying it many times and learning from the attempts that worked. We used it to train Qwen3-4B on a small Affordance app, and it went from 40.2% right answers to 77.3% on episodes it had never seen.

The part we cared most about is the training data. Every one of the 345,983 model calls the model trained on matched, token for token, what the inference server had actually processed. In a typical pipeline that is not guaranteed, and on this task the usual way of rebuilding the data changed 17% of the model's replies.

Reinforcement learning with a checkable answer

The method is often called RLVR, reinforcement learning with verifiable rewards. You need a task where a program can check the answer. The model attempts it many times. Attempts that got the right answer are rewarded, and training makes the steps the model took in those attempts more likely next time.

Training works on tokens, the pieces of text a model reads and writes. To make a step more likely, the trainer has to know exactly which tokens the model was shown and exactly which tokens it wrote. If either is slightly off, the trainer adjusts the model for something it did not do.

The task is an app

The model is shown a notebook of 16 to 48 four-digit numbers, but it can only see about 40 tokens of it at a time, a few rows. It has three tools: scroll the notebook, write a note to itself, and submit an answer. The reward is 1 if it submits the largest number in the notebook and 0 otherwise.

The task is easy to state and easy to fail. Before training, the model often submitted the largest number it had seen so far without scrolling to the end. It reached the last row in only 47% of episodes.

In Affordance the environment is an app like any other. The tools are functions, and the screen is a box with a 40-token limit:

/** Scroll the notebook by a number of rows (negative scrolls up). */
function scroll({ by }: { by: number }) {
  const next = Math.min(Math.max(0, offset + Math.trunc(by)), numbers.length - 1);
  setOffset(next);
  return `The notebook now starts at row ${next}.`;
}
/** Replace your notes with new text. */
function note({ text }: { text: string }) {
  setNotes(text);
  return "Noted.";
}
/** Submit your final answer: the largest number in the notebook. This ends the task. */
function submit({ answer }: { answer: number }) {
  harness.stop({ answer });
  return "Submitted.";
}
...export function Screen({ numbers, offset, notes, window, fns }) {
  return <>
    <box label="notebook" max_tokens={window} more="...">
      {`Notebook (${numbers.length} rows), starting at row ${offset}:\n`}
      {numbers.slice(offset).map((n, i) => `${offset + i}: ${n}\n`)}
    </box>
    {`\nYour notes: ${notes || "(empty)"}\n`}
    {fns.map(fn => <Tool fn={fn}/>)}
  </>;
}

Exporting what the model actually saw

Affordance builds every prompt itself, so it knows the exact token ids at every step. Each new prompt in this app is the previous prompt with the model's reply and the app's response added to the end. So a whole episode is one sequence of tokens. The model's own tokens are trained on, and the tokens the app wrote are not.

One real episode from the final evaluation: 1,146 tokens, 7 model calls, a 20-row notebook, answered correctly.

To check the export, we recorded every request to the inference server and its response, byte for byte. Then we compared every exported episode against those recordings.

What the usual way gets wrong

Most agent frameworks store a conversation as a list of messages: the user's text, the model's reply, the tool call it made. To build training data, the trainer turns that list back into tokens with the model's chat template. We did the same to the base model's 1,766 replies from this task and compared.

In 297 of them, 17%, the rebuilt reply was not what the model wrote. The most common case is small. The model put a blank line before its tool call, and the template writes a single newline:

 will now submit this as the largest number. \n\n

<tool_call>
{"name": "

 will now submit this as the largest number.\n
<tool_call>
{"name": "note

It looks harmless, but it changes what the trainer learns. Read in the rebuilt context, the token the model actually sampled there became about 23 nats less likely, a factor of roughly 14 billion. 60 of the 113 affected calls had a token move by more than 5 nats. A trainer working from rebuilt history is training on a conversation the model never had.

Decoding the model's tokens to text and encoding them again also changes 15 of the 1,766 replies: the same text, split into different tokens.

The training run

Each step, the model attempted 32 notebooks 8 times each. Within each group of 8, an attempt that did better than the group's average was made more likely, and one that did worse was made less likely. This is GRPO. Then the new weights were loaded into the inference server and the next step began.

We ran 70 steps, 17,920 episodes, in 45 minutes: two GPUs trained the model and two served it with SGLang. Every 10 steps we measured it on the same 256 held-out episodes.

What the model learned is the obvious missing habit. The share of episodes in which it scrolled to the last row before answering went from 47% to 100%. On a second set of new notebooks it went from 37.5% to 80.5%.

Training did not drift the model away from what the server sampled. At the same weights, the trainer's probabilities for the sampled tokens differed from SGLang's by 0.0059 nats per token on average.

One run on one machine, with a small model and a toy task, and no repeated seeds. Accuracy stopped rising after 40 steps, once the model answered each notebook the same way every time. The trained model scrolls in small steps, so on 56-row notebooks it needs more than the 16 turns it trained with: with 32 turns it scores 71.5%, against 25.4% before training.

Try it

The environment is the app above, run with affordance run against SGLang through Affordance's token-id binding, with every request recorded by setting AFFORDANCE_CAPTURE_DIR. The trainer is about 350 lines of PyTorch and is not published yet.