SiddharthOne Score Is All You Get: GRPO From Scratch, With a Lava-Hopping Robot
Siddharth
October 3, 202635 min read
DeepSeek-R1 learned to reason from one number per answer. This is the algorithm behind it, GRPO, rebuilt from scratch in NumPy on a toy game where a robot also gets one number per run, with every experiment run for real and one update traced to the last decimal.
DeepSeek-R1-Zero learned to reason without a single worked solution to imitate. It was trained with pure reinforcement learning, starting from the DeepSeek-V3 base model with no supervised fine-tuning first . Over that training, its pass@1 on AIME 2024 rose from 15.6% to 77.9%
.
What makes that striking is how little the model is told. Each answer earns one score, delivered at the end. Nobody marks the line where the reasoning went wrong or the step that rescued it. The algorithm that turns a signal that thin into learning is GRPO, Group Relative Policy Optimization, introduced in the DeepSeekMath paper as a variant of PPO .
This post builds GRPO from nothing, on a game small enough to hold in your head. A robot crosses a foundry floor laced with lava, and it gets one number per run: how far it got. We will climb the ladder the field climbed (REINFORCE, then PPO, then GRPO), run real experiments at every rung, and then take a single GRPO update apart, number by number. It is all NumPy, and every figure and number below comes from code that runs end to end in about ten seconds.
Fig 1 is where we are headed. Untrained, seven of eight runs end in the first two channels. After 300 GRPO updates the bot crosses every one-tile channel and falls only at the two-tile ones, where, as we will see, even a perfect policy loses half the time. Two exits in eight sounds modest until you know that on this floor the best possible policy reaches the exit 25% of the time.
The floor has 40 tiles. The bot starts on tile 0, and the exit dock is tile 39. Between tiles 5 and 36 lie three to seven lava channels, each one or two tiles wide, with at least two safe tiles between neighbors. Every training round draws a fresh floor.
Each turn, the bot reads three sensors: is there lava one, two, or three tiles ahead? Append a constant 1 as a bias input and you have its entire view of the world, a four-number vector such as [1, 0, 0, 1] for "lava directly ahead." Then it chooses one of two moves. STEP advances exactly one tile. LEAP advances two or three tiles, decided by a fair coin the bot does not control.
That coin is the heart of the game. With lava directly ahead, stepping is certain death, while a leap clears a one-tile channel whichever way the coin lands. A two-tile channel is different: a 2-tile leap lands on its far half. From the tile in front of it, the best any policy can do is a 50/50 crossing. Training cannot remove that risk; it can only approach the limit it sets.
Landing on lava ends the run, and reaching tile 39 ends it well. Either way, the bot receives exactly one number: its final tile divided by 39. There is no reward along the way and no hint about which move was the mistake.
FLOOR, EXIT = 40, 39 # tiles 0..39; the exit dock is tile 39
STEP, LEAP = 0, 1 # the bot's two moves
def new_floor(rng):
"""3-7 lava channels, 1-2 tiles wide, in tiles 5..36, >= 2 apart."""
k = rng.integers(3, 8) # how many channels
widths = rng.integers(1, 3, size=k) # each 1 or 2 tiles wide
spare = 32 - widths.sum() - 2 * (k - 1) # leftover safe tiles
gaps = rng.multinomial(spare, np.ones(k + 1) / (k + 1))
lava, t = np.zeros(FLOOR, dtype=bool), 5 + gaps[0]
for w, extra in zip(widths, gaps[1:]):
lava[t:t + w] = True
t += w + 2 + extra # 2 safe tiles + any spare
return lava
def sensors(lava, pos):
"""What the bot sees: lava at pos+1, pos+2, pos+3, then a bias 1.0."""
ahead = [float(t < FLOOR and lava[t]) for t in range(pos + 1, pos + 4)]
return np.array(ahead + [1.0]) # past tile 39 reads safe
def step(pos, action, rng):
"""STEP moves exactly 1 tile. LEAP is a coin flip: 2 or 3 tiles."""
return pos + 1 if action == STEP else pos + int(rng.integers(2, 4))
def score(pos):
"""The only feedback, given once at the end: how far the bot got."""
return min(pos, EXIT) / EXITThat is the whole game, about thirty lines. Notice what score() receives: a position. Not the path, not the moves, not what the sensors saw.
The policy is a 4×2 matrix W, with one row per input (lava +1, lava +2, lava +3, bias) and one column per move. Multiply the observation by W and you get two scores; a softmax turns them into the probabilities of STEP and LEAP. Because the sensors read 0 or 1, an observation simply selects rows of W and adds them up.
Training starts from all zeros. Both scores are then zero, and every move is a fair coin flip. On the 200 held-out test floors used throughout this post, that untrained bot scores 0.276 on average and reaches the exit 0.6% of the time.
Run = namedtuple("Run", "states actions path reward")
def policy(W, s):
"""pi(.|s) = softmax(s @ W). W is 4x2: one column of weights per move.
Accepts one sensor row (shape 4) or a stack of rows (shape N x 4)."""
z = s @ W
e = np.exp(z - z.max(axis=-1, keepdims=True)) # max-shift for safety
return e / e.sum(axis=-1, keepdims=True)
def play(W, lava, rng):
"""One run: look, sample a move from pi, move; until lava or the exit."""
pos, states, actions, path = 0, [], [], [0]
while pos < EXIT and not lava[pos]:
s = sensors(lava, pos)
a = LEAP if rng.random() < policy(W, s)[LEAP] else STEP
states.append(s)
actions.append(a)
pos = step(pos, a, rng)
path.append(min(pos, EXIT))
return Run(np.array(states), np.array(actions), path, score(pos))A run records what the bot saw, what it did and where it landed, step by step. The learning algorithms get those records and the final score, nothing else.
One property of this tiny brain pays off later. Three binary sensors means only 2³ = 8 situations the bot can ever face, so there are only 2⁸ = 256 deterministic policies, each a fixed move for each situation. That is few enough to score every one exactly, so for once we will know the true best score instead of guessing at it.
The oldest idea in policy-gradient learning is also the most intuitive. After a run, go back over every move and make it more likely in proportion to the score the run received. Good runs push their moves up a lot. Bad runs push their moves up a little.
For a softmax policy the push has a tidy closed form. The gradient of log π(a|s) with respect to W is the outer product of the sensor row s with onehot(a) − π(·|s). It raises the chosen move's score and lowers the other's, and only on the rows the sensors switched on. REINFORCE scales that by the run's reward, averages over every step in the batch, and takes a step.
def batch(runs, per_run):
"""Stack every step of every run; each step inherits its run's number."""
S = np.concatenate([run.states for run in runs]) # N x 4 sensor rows
acts = np.concatenate([run.actions for run in runs]) # the N moves taken
vals = np.concatenate([np.full(len(run.actions), x)
for run, x in zip(runs, per_run)])
return S, acts, vals
def reinforce_update(W, runs, lr=0.5):
"""REINFORCE: push every move up in proportion to its run's raw reward."""
S, acts, R = batch(runs, [run.reward for run in runs])
push = np.eye(2)[acts] - policy(W, S) # onehot(a) - pi = grad log pi
return W + lr * S.T @ (R[:, None] * push) / len(acts)Now reread the last sentence of the first paragraph: bad runs push their moves up a little. Our rewards run from 0 to 1 and are never negative, so a run that walks straight into the first channel still scores around 0.2, and every move it made, the fatal one included, gets reinforced.
REINFORCE is not wrong. In expectation, good moves are pushed harder than bad ones, and those relative differences do steer the policy, eventually. The trouble is that the useful signal is a small difference between large, all-positive pushes, so most of each update says only "do more of whatever you just did." After 300 updates REINFORCE reaches 0.339 on the test floors, up from 0.276 but nowhere near what is possible. It also picks up a bad habit: across five training seeds it ends up leaping 56% to 67% of the time with lava two tiles ahead, more often than the coin flip it started with, and that is exactly where a 2-tile leap lands in the lava.
The fix is to stop asking "was this run good?" and start asking "was it better than expected?"
PPO adds two ideas to REINFORCE. The first is the advantage: judge each run against a forecast of what it should have scored. A 90 on an exam is great if you usually get 70 and terrible if you usually get 98. Subtract the forecast, and runs that fall short finally push their moves down.
The forecast comes from a critic, a second model trained alongside the policy to predict the final score from what the bot can see. Ours is the smallest possible: V(s) = v·s, four more weights, fitted by least squares to the scores runs actually earned. Every step's advantage is A = r − V(s).
The second idea is reuse. Playing is the expensive part, so PPO squeezes several gradient steps out of each batch of runs; we take four passes. The catch is that after the first pass the policy has changed, while the runs were played by the old one. PPO accounts for that with the ratio ρ = πnew(a|s) / πold(a|s), which says how much more or less likely each recorded move has become since it was played, and it weights every step's push by that ratio.
Then comes the clip, PPO's signature move. The ratio is held to the band [1 − ε, 1 + ε], with ε = 0.2. Think of it as a lock on a buy button: once a move has become 20% more likely than when it was played, pushing it further earns nothing, so its gradient switches off for the rest of the update. Formally, each step contributes min(ρ·A, clip(ρ, 0.8, 1.2)·A), and whenever the clipped branch wins that min, no gradient flows.
def ppo_update(W, v, runs, lr=0.5, eps=0.2, passes=4, lr_critic=0.5,
trace=None):
"""Toy PPO: advantage = actual score - critic's forecast V(s) = v.s"""
S, acts, R = batch(runs, [run.reward for run in runs])
A = R - S @ v # did the run beat the forecast?
W = clipped_update(W, S, acts, A, lr, eps, passes, trace)
for _ in range(passes): # critic: fit forecasts to scores
v = v + lr_critic * S.T @ (R - S @ v) / len(R)
return W, vThe clip sounds like a detail, but it is what makes reuse safe. Without it, four passes over the same eight runs would chase their noise four times over. You will see it engage on real numbers in the walkthrough below (Fig 8).
On our floors PPO is a clear step up: 0.493 after 300 updates, against 0.339 for REINFORCE. But it came with a second model that has to learn, and that model has a blind spot.
In the setting GRPO was designed for, the critic is not four weights. In PPO for language models, the value function is "typically another model of comparable size as the policy model," which brings "substantial memory and computational burden" . You end up training two large models to improve one.
It is also hard to train. Usually "only the last token is assigned a reward score" , yet the value model is supposed to estimate, at every token along the way, what that final score will be. In effect it reads half an answer and predicts whether the finished one will be right.
Our toy has a small, clean version of the same problem. From the start line every floor looks identical: no lava within three tiles, so the sensors read [0, 0, 0, 1]. Whatever the critic forecasts there, it forecasts for every floor, and floors are not alike.
No single number can serve both floors. Put the forecast below 0.46 and every easy-floor run looks like a success; put it above 0.44 and every hard-floor run looks like a failure. Our trained critic says 0.66, so all eight hard-floor runs get negative advantages. That includes the best of them, which made the right move at every step and then lost the fair coin flip at a two-tile channel. From the start line its advantage is −0.22: the critic tells the bot to do less of a flawless run.
The forecast also overshoots this bot's true average of 0.49, for a second reason: our critic cannot tell where on the floor it is. "No lava in view" reads the same on tile 0 as on tile 36, a step from the exit, so its forecast for that view blends the two.
A language-model critic sees the whole prefix, so its blind spot is subtler, but its job has the same shape: predict the final score of an answer from a partial view of it. GRPO's answer is to stop predicting.
GRPO replaces the forecast with a measurement. Play the same floor G = 8 times. The group's mean score becomes the baseline: runs that beat it are pushed up, runs that miss it are pushed down. Dividing by the group's standard deviation then puts every group on the same scale, whether its floor was gentle or brutal.
For run i on a floor, the advantage is Ai = (ri − mean(r)) / std(r), and every step of run i shares that one number. The ratio, the clip, and the four passes stay exactly as they were in PPO.
Time for a pop quiz. Below are eight runs on one hard floor, played by a half-trained bot. Every one of them ended in lava. Which runs get a positive advantage?
The ones that got furthest. GRPO does not need anyone to reach the exit. It needs runs that differ, and it pushes the policy toward whatever the better ones did. Apply the same arithmetic to the two floors in Fig 4 and each is graded against its own mean (0.83 and 0.26). The hard floor's flawless run, the one the critic punished, now earns the largest advantage in its group: +2.23.
In code, the critic's two jobs, forecasting and learning to forecast, collapse into a single line.
def grpo_update(W, runs, lr=0.5, eps=0.2, passes=4, trace=None):
"""GRPO: one floor played G times; each run graded against its group."""
r = np.array([run.reward for run in runs])
adv = (r - r.mean()) / (r.std() + 1e-8) # grade on a curve: no critic
S, acts, A = batch(runs, adv) # every step shares its run's A
return clipped_update(W, S, acts, A, lr, eps, passes, trace)
def clipped_update(W, S, acts, A, lr=0.5, eps=0.2, passes=4, trace=None):
"""PPO's machinery, unchanged in GRPO: reuse one batch for several
passes, weight each step by rho = pi_new / pi_old, stop at the clip."""
N, rows = len(acts), np.arange(len(acts))
onehot = np.eye(2)[acts]
pi_old = policy(W, S)[rows, acts] # frozen: the policy that played
for _ in range(passes):
pi = policy(W, S)
rho = pi[rows, acts] / pi_old # exactly 1 on pass 1
clipped = ((A > 0) & (rho > 1 + eps)) | ((A < 0) & (rho < 1 - eps))
coef = np.where(clipped, 0.0, A * rho) / N # clipped: no gradient
# gradient = sum over steps of coef * outer(s, onehot(a) - pi)
grad = S.T @ (coef[:, None] * (onehot - pi))
if trace is not None: # the walkthrough reads these
trace.append(dict(W=W.copy(), pi=pi, rho=rho, clipped=clipped,
coef=coef, grad=grad))
W = W + lr * grad
return WCompare it with ppo_update above. The critic's forecast and its training loop are gone, and adv = (r - r.mean()) / (r.std() + 1e-8) stands in for both. Everything after that line is PPO's machinery, untouched. This is the advantage the DeepSeekMath paper describes: rewards normalized by subtracting the group mean and dividing by the group standard deviation, with every token of an output sharing that normalized reward .
The paper calls this outcome supervision , and it has a price: resolution. One number per run means the fatal move and the sound moves before it all receive the same advantage. The finer-grained alternative, process supervision, scores intermediate steps; we stick with outcomes, because outcomes are all our game provides.
The full objective has one more term: a KL penalty that keeps the policy close to a frozen reference model, added directly to the loss rather than folded into the reward . We switch it off (β = 0) for the toy. The penalty exists to protect a starting policy worth protecting, and ours starts as a coin flip.
Let's run a single GRPO update by hand. I froze the bot from our first training run after 20 updates (test score 0.392, up from 0.276) and gave it a hard floor: six channels, two of them two tiles wide. It played the floor eight times; these are the runs from the pop quiz.
One disclosure: I picked this group because every run dies and, as you will see, the clip engages. At our learning rate the clip is a guardrail that rarely gets touched. Over that whole training run it engaged in 42 of 300 updates, on 0.14% of all step-passes.
The floor. Lava sits on tiles 7, 15–16, 21, 26–27, 30 and 35.
| 0 | 5 | 10 | 15 | 20 | 25 | 30 | 35 | 39 |
The group. Eight runs with 76 steps between them, so N = 76:
| Run | Moves | Path (tiles) | Fell on | Score r | Advantage |
|---|---|---|---|---|---|
| 1 | LLSLSSLSLLL | 0 3 5 6 8 9 10 13 14 17 19 21 | 21 | 0.538 | +0.55 |
| 2 | SLLLLLSSLLLSSL | 0 1 3 6 8 10 12 13 14 17 20 22 23 24 27 | 27 | 0.692 | +1.38 |
| 3 | SSSLL | 0 1 2 3 5 7 | 7 | 0.179 | −1.38 |
| 4 | SSLSSS | 0 1 2 4 5 6 7 | 7 | 0.179 | −1.38 |
| 5 | SLLLSSLS | 0 1 4 6 9 10 11 14 15 | 15 | 0.385 | −0.28 |
| 6 | LSSLLLSSL | 0 3 4 5 8 10 12 13 14 16 | 16 | 0.410 | −0.14 |
| 7 | LLLLSSSL | 0 2 5 8 11 12 13 14 16 | 16 | 0.410 | −0.14 |
| 8 | LSSLSSLSLSLLLSL | 0 3 4 5 8 9 10 13 14 17 18 20 22 24 25 27 | 27 | 0.692 | +1.38 |
The group's mean score is 0.436 (tile 17.0) and its standard deviation is 0.186 (7.26 tiles), so each run's advantage is its distance from tile 17.0, measured in units of 7.26 tiles. The two runs that reached tile 27 get +1.38; the two that died on tile 7 get −1.38.
One step, in full. Take run 4's last move. Standing on tile 6, its sensors read lava directly ahead: s = [1, 0, 0, 1]. Only two rows of W are switched on, row 0 (lava +1) and row 3 (bias).
| Run 4, step 6, on tile 6 | STEP | LEAP |
|---|---|---|
| Row 0 of W (lava +1), switched on by s[0] = 1 | −1.1851 | 1.1851 |
| Row 3 of W (bias), switched on by s[3] = 1 | −0.0553 | 0.0553 |
| Scores s·W = row 0 + row 3 | −1.2404 | 1.2404 |
| Probabilities π = softmax(scores) | 0.0772 | 0.9228 |
| Move taken | STEP → tile 7, lava | |
| Run 4's advantage (shared by all its steps) | −1.3769 |
The policy gave STEP a 7.7% chance, and the bot stepped anyway, into the lava on tile 7. Run 4 scored 0.179, and its advantage is −1.38. Nobody tells the update that this was the fatal move. It receives the same −1.38 as the five harmless moves before it.
Pass 1. The policy has not moved yet, so all 76 ratios are exactly 1. Each step's coefficient is A·ρ/N, and the step's push is outer(s, onehot(a) − π) times that coefficient:
| Pass 1 for the same step | STEP | LEAP |
|---|---|---|
| onehot(STEP) − π | +0.9228 | −0.9228 |
| × coefficient A·ρ/N = −1.3769 × 1 / 76 = −0.01812: push on rows 0 and 3 | −0.0167 | +0.0167 |
| × learning rate 0.5: this step's change to rows 0 and 3 | −0.0084 | +0.0084 |
| Rows 1 and 2 (lava +2, +3 read 0) | 0 | 0 |
Less STEP and more LEAP when lava is directly ahead: the right lesson, delivered by a run that knew nothing about which of its moves was wrong. Sum the pushes of all 76 steps, multiply by the learning rate, and you have the first pass:
| Row of W | Before: STEP | LEAP | ΔW: STEP | LEAP | After pass 1: STEP | LEAP |
|---|---|---|---|---|---|---|
| lava +1 | −1.1851 | 1.1851 | −0.0148 | +0.0148 | −1.1999 | 1.1999 |
| lava +2 | 0.0656 | −0.0656 | −0.0091 | +0.0091 | 0.0565 | −0.0565 |
| lava +3 | 0.6707 | −0.6707 | −0.0034 | +0.0034 | 0.6673 | −0.6673 |
| bias | −0.0553 | 0.0553 | −0.0453 | +0.0453 | −0.1006 | 0.1006 |
Two things stand out. The columns move as mirror images: every row of onehot(a) − π sums to zero, so whatever STEP gains, LEAP loses. And the largest change is not on a lava row at all but on the bias row, which every step touches.
Passes 2 to 4. Now the policy has moved, and the ratios fan out.
| Pass | Lowest ρ | Highest ρ | Outside 0.8–1.2 | Clipped (no gradient) |
|---|---|---|---|---|
| 1 | 1.000 | 1.000 | 0 | 0 |
| 2 | 0.881 | 1.095 | 0 | 0 |
| 3 | 0.780 | 1.194 | 1 | 1 |
| 4 | 0.698 | 1.290 | 5 | 4 |
The first clip lands on pass 3. Run 5 stepped into the two-tile channel at tile 15, a move the policy had given only 8.7%; its run's advantage is a mild −0.28. By pass 3 that move is 22% less likely than when it was played (ρ = 0.78), and the clip switches it off. The update has pushed it as far as it is allowed to.
By pass 4, four steps are clipped. Here is every clipped step-pass in the update:
| Pass | Step | Tile | Sensors | Move | π_old(move) | A | ρ |
|---|---|---|---|---|---|---|---|
| 3 | run 5, step 8 | 14 | 2-wide lava, directly ahead | STEP | 0.087 | −0.28 | 0.780 |
| 4 | run 2, step 14 | 24 | 2-wide lava, 2 ahead | LEAP | 0.204 | +1.38 | 1.290 |
| 4 | run 4, step 6 | 6 | lava directly ahead | STEP | 0.077 | −1.38 | 0.729 |
| 4 | run 5, step 8 | 14 | 2-wide lava, directly ahead | STEP | 0.087 | −0.28 | 0.698 |
| 4 | run 8, step 11 | 18 | lava 3 ahead | LEAP | 0.226 | +1.38 | 1.235 |
Two of them are moves you want pushed down, and they have been pushed far enough: run 4's fatal step, our worked example, whose ratio went 1.00, 0.89, 0.80, 0.73 across the four passes, and run 5's step into the channel, now at 0.70. The other two are more interesting. Run 2, one of the two best runs, ended with a leap from tile 24 into the two-tile channel at 26–27, a leap the policy had given 20%. Its run's advantage is +1.38, so the update pushes that fatal leap up, 29% by pass 4, until the clip stops it. Run 8 took a similar gamble on tile 18, leaping with lava three tiles ahead, and happened to land safely. Same story: clipped at 1.24.
That is outcome supervision's blind spot, and the clip is the guardrail on it. A run that beats its group lends its advantage to every move it made, its worst included. The clip cannot tell which moves those are, but it switches off each move's own push once its probability has moved 20%. Shared weights can still carry a move a little further, as run 2's 1.29 shows, but the incentive to keep pushing is gone.
One more detail. Five ratios sit outside the band on pass 4, but only four are clipped. The fifth belongs to run 5's leap on tile 4, with lava three ahead. Its run fell short (A = −0.28), yet through the shared weights that leap has become 24% more likely (ρ = 1.24). The min in PPO's objective keeps the gradient on in that case: the clip stops pushes in the direction the advantage already wants, never the corrections.
What the update did. Here is how the policy moved across the four passes:
| Sensors | Before | Pass 1 | Pass 2 | Pass 3 | Pass 4 | Change |
|---|---|---|---|---|---|---|
| lava directly ahead | 0.923 | 0.931 | 0.938 | 0.944 | 0.946 | +0.023 |
| 2-wide lava, directly ahead | 0.913 | 0.923 | 0.932 | 0.939 | 0.941 | +0.028 |
| no lava in view | 0.528 | 0.550 | 0.572 | 0.592 | 0.600 | +0.072 |
| lava 3 ahead | 0.226 | 0.244 | 0.261 | 0.279 | 0.281 | +0.055 |
| lava 2 ahead | 0.495 | 0.522 | 0.548 | 0.572 | 0.579 | +0.085 |
| 2-wide lava, 2 ahead | 0.204 | 0.223 | 0.243 | 0.263 | 0.264 | +0.060 |
Look at the direction. This update made leaping more likely for every pattern, including "lava two ahead," the exact move that killed run 3. The runs that beat the group mean leapt on 24 of their 40 moves (60%); the runs below it, on 17 of their 36 (47%). Leaping covers ground, so in this group it went with getting further, and the update cannot separate the good leaps from the bad ones. Measured on the test floors, this single update made the bot slightly worse: 0.392 before, 0.384 after.
That is not a flaw in the walkthrough; it is how the method works. One group is a small, noisy sample, and each update is a small, noisy step. The errors average out over hundreds of floors, which is why the training curves in the next section climb anyway, and why it pays to cap how far any single batch can pull.
Back to full training. Each algorithm gets 300 updates, and each update is a fresh floor played eight times, so all three see the same kind of experience and differ only in what they do with the eight scores. I trained five seeds of each and scored the policy every 10 updates on the same 200 test floors.
The scoring is exact rather than sampled. The only randomness in a run is coin flips, the policy's and the leap's, so a short backward pass over the 40 tiles computes a policy's expected score precisely. As a cross-check, simulations that actually play each test floor 50 times land within 0.003 of the exact values.
Exactness also buys the ceiling. With only 256 deterministic policies, we can score them all. The best reaches 0.574. Eight policies tie for first, differing only on patterns that never occur or do not matter, and all of them follow one rule: leap if and only if lava is directly ahead. No policy of any kind can beat it, because it crosses every one-tile channel with certainty and every two-tile channel at the unavoidable 50%. The runner-up among the 256 manages only 0.435.
| Policy after 300 updates | Test score | Seed range | Share of best | Reached the exit |
|---|---|---|---|---|
| Untrained (all weights zero) | 0.276 | — | 48% | 0.6% |
| REINFORCE | 0.339 | 59% | 2.8% | |
| PPO | 0.493 | 0.488–0.504 | 86% | 14.9% |
| GRPO | 0.553 | 0.545–0.559 | 96% | 21.5% |
| Best possible (exact) | 0.574 | — | 100% | 24.0% |
GRPO finishes at 0.553, 96% of the ceiling, and passes 90% of it after 170 updates; neither PPO nor REINFORCE gets there within 300. The exit rates tell the same story in more human terms. Even the ceiling reaches the exit only 24.0% of the time, because the average test floor has 2.6 two-tile channels, each a coin flip, and only 2.5% of floors have none.
A fair objection: dividing by the standard deviation also makes GRPO's steps bigger. Its advantages average 0.94 in size per step, against 0.25 for PPO's raw score differences, at the same learning rate. So I reran the others with larger rates. PPO at 1.0 and 1.5 reaches 0.537 and 0.546, closing most of the gap but still short of GRPO at its default rate (0.553), let alone GRPO at 1.0 (0.565). REINFORCE at 1.5 gets to 0.423, with one of its five seeds stuck at 0.319. Step size explains part of GRPO's edge here, but not all of it.
One GRPO seed deserves a closer look: the flat teal line in Fig 9. After a few early leaps paid off, it learned to leap on every sensor pattern at least 89% of the time by update 10. A bot that almost never steps rarely produces the contrast it would need to learn when stepping is better, so it sat near 0.32 until about update 160 before climbing out. Grading on a curve removes the critic, not the need to explore.
What did it learn? Fig 10 reads the trained policy back out, one sensor pattern at a time.
With lava directly ahead, stepping is certain death, so it leaps 99.5% of the time. With lava two tiles ahead, a 2-tile leap lands squarely in the channel, while a step sets up a clean leap on the next turn, so it almost never leaps there (2.1%). With lava three ahead, it is the 3-tile leap that lands in it: 0.3%.
The most interesting number is the second bar. With a two-tile channel directly ahead, the bot leaps 97.3% of the time, not 99.5%. Eight weights cannot store a lookup table; they add up evidence. The lava-two-ahead weight learned "wait," and in this pattern lava is also two ahead, so that weight pulls against the leap. The leftover 2.7% is the price of a brain that sums instead of memorizing, and it is exactly how one of the trained runs in Fig 1 died: stepping into the channel on tile 17.
With nothing in view, the bot leaps 10.6% of the time, and across the five seeds that value ranges from 0.10 to 0.59. That is not instability. Leaping over safe tiles lands on safe tiles, so once the rest of the policy is right this choice does not change the score, and the gradient has little to say about it.
| Row of W (input) | STEP column | LEAP column | LEAP − STEP |
|---|---|---|---|
| lava +1 | −3.698 | 3.698 | +7.40 |
| lava +2 | 0.848 | −0.848 | −1.70 |
| lava +3 | 1.818 | −1.818 | −3.64 |
| bias | 1.067 | −1.067 | −2.13 |
For a two-move softmax only the difference between the columns matters, and because every update moves the columns by equal and opposite amounts, the eight weights are really four numbers: a strong +7.40 toward leaping when lava is directly ahead, a default lean toward stepping (−2.13), and two vetoes for lava further out (−1.70 and −3.64).
Three swaps turn the toy into the real thing. The floor becomes a question. The run becomes a sampled answer, produced token by token the way our bot produces moves. And the distance score becomes a verifier's verdict on the final answer: right or wrong.
The groups get bigger. DeepSeekMath sampled 64 outputs per question, and its policy took a single update per exploration stage, with a KL coefficient of 0.04 and a policy learning rate of 1e-6 . If each batch feeds exactly one gradient step, every ratio is exactly 1, as in our pass 1, and the clip never engages; it earns its keep when samples are reused, as in our four passes.
DeepSeek-R1-Zero, the pure-RL run on the V3 base model, used groups of 16. Each training step drew 32 unique questions with 16 rollouts apiece, a batch of 512; the 8,192 rollouts were split into 16 mini-batches and trained for a single inner epoch, and the reference model was replaced every 400 steps . Over training, its AIME 2024 pass@1 rose from 15.6% to 77.9%, and to 86.7% with self-consistency over 16 samples
. The final DeepSeek-R1 model reached 79.8% on AIME 2024 and 97.3% on MATH-500, and the work was published in Nature
.
Nothing in the update itself changes. A group of answers to one question is a group of runs on one floor. Answers that beat the group's mean are pushed up, token by token, and the rest are pushed down. Swap "how far did the bot get" for "is the final answer correct," and grpo_update above is, structurally, the algorithm. What the real version adds is scale, and the reference model.
That reference model is the one box our toy leaves empty. The real objective subtracts β·KL(πθ ‖ πref), a penalty for drifting from a frozen copy of the starting policy, directly in the loss . A language model's starting policy already writes fluent, sensible text, and the penalty keeps reinforcement learning from trading that away for reward. Our bot starts as a coin flip with nothing worth keeping, so β = 0.
The trained bot ends up where it should: it leaps when lava is directly ahead, waits otherwise, and loses only the coin flips nobody can win. It got there from one number per run, graded on a curve. That is the whole trick, and it scales.
Every number, chart and table in this post comes from three small files. Download them and follow along, change the hyperparameters, break the game — that is when GRPO actually clicks.
That is the whole setup:
pip install numpy matplotlib # the only dependencies
python test_lava_leap.py # 7 checks pass
python run_experiments.py # every chart and number in this postGood first experiments: raise the learning rate until the clip engages every update, set G = 2 and watch GRPO degrade toward REINFORCE, or make every channel two tiles wide and watch the ceiling fall to chance.
Thoughts and essays, published with Yokush. See more posts
Siddharth20 min
Siddharth14 min
Comments 0