Classic RL models — where does reward magnitude matter?
Gong, Martell, Dudman & Coddington (2026).
Reward magnitude determines reinforcement learning efficiency. Science.
Task: 2-arm deterministic bandit. Arm 0 → reward R; arm 1 → 0.
Core question: Does increasing R (while holding α fixed) speed up learning —
i.e., shorten the trials needed to reach P(correct) ≥ 90%?
The answer depends entirely on where R appears in the update rule or policy.
Plots show reinforcements to criterion vs reward size (0.1 → 1.0); red ticks = mean ± SEM.
Group A
ε-greedy —
R cancels out: only the rank of Q values matters for action selection
Q-learning
+ ε-greedy
slow α = 0.001
artifact
init noise
2 params: α = 0.001, ε = 0.1
TD Error
δ = r − Qa
Value Update
Qa ← Qa + α · δ
Action Selection
a = argmax Q (w.p. 1−ε)
= TD(0) / Schultz dopamine model. On a bandit γV(s′) = 0, so δ = r − Qa. δ scales with R because Q scales with R — not because learning is faster.
reward size →
Apparent effect — initialization artifact
Q starts with small noise (std 0.05), so some seeds wrongly prefer the bad arm. Larger R overcomes this bias faster. With zero init, the curve is flat for any R > 0.
α = 0.001 · ε = 0.1 · Q init ~ 𝒩(0, 0.05)
Q-learning
+ ε-greedy
fast α = 0.1
NO effect
rank-based selection
2 params: α = 0.1, ε = 0.1
TD Error
δ = r − Qa
Value Update
Qa ← Qa + α · δ
Action Selection
a = argmax Q (w.p. 1−ε)
Same rule as above — only α differs. Q0 overtakes Q1 after 1–2 correct trials for any R > 0. Flat at the floor.
reward size →
Key contrast with slow α
Same rule, different α shows that α — not R is the speed knob for ε-greedy. One converges in <100 trials, the other takes thousands — purely because of α.
α = 0.1 · ε = 0.1
Group B
Softmax policy —
R enters the policy directly via absolute Q values; normalization removes it
Q-learning
+ softmax
Luce's choice rule
YES effect
policy artifact
2 params: α = 0.001, τ = 0.05
Value Update
Qa ← Qa + α · (r − Qa)
Action Selection
π(a) = exp(Qa/τ) / Σ exp(Qb/τ)
At asymptote Q0 → R, so π(a0) = σ(R/τ). Larger R → higher P(correct) even after learning is done.
reward size →
Effect lives in the policy, not the update
The learning rule is identical to Group A. The effect appears because softmax converts absolute Q values to probabilities — and Q0 asymptotes to R, so larger R → faster P(correct) ≥ 0.9.
α = 0.001 · τ = 0.05 (temperature)
Normalized
Q-learning + softmax
Reward-normalized δ rule
NO effect
normalization removes artifact
2 params: α = 0.001, τ = 0.05
Reward Normalization
r̃ = r / maxt′≤t rt′
Value Update
Qa ← Qa + α · (r̃ − Qa)
Action Selection
π(a) = exp(Qa/τ) / Σ exp(Qb/τ)
After first reward, r̃ = 1 always → Q0 → 1 for any R. Softmax sees identical values regardless of R.
reward size →
Proof that the softmax effect was an artifact
Same softmax policy as above — only the value update is normalized. Result: perfectly flat across R. Confirms the effect in Q-learning+softmax was due to raw Q magnitudes entering the policy.
α = 0.001 · τ = 0.05
Group C
Policy-gradient —
R multiplies the gradient update directly: genuine learning-speed effect
REINFORCE
Williams 1992
Monte-Carlo PG
YES effect
gradient scaling
1 param: α = 0.002
Policy Update
Δθa = α · r · (𝟙[a] − π(a))
Action Selection
a ~ π(·|θ) (softmax over θ)
No value function — R is not subtracted or normalized. It scales the step size of every gradient update directly.
reward size →
R multiplies the gradient directly
Each update is α·R·∇log π — bigger R means bigger steps. A genuine effect in the learning rule. Ceiling at R=0.1: many seeds don't converge in T=20k.
α = 0.002 (only parameter — no value function)
Actor-Critic
REINFORCE
+ learned baseline
YES effect
δ ∝ R early
2 params: αa = 0.002, αc = 0.02
TD Error (Critic)
δ = r − V
Value Update (Critic)
V ← V + αc · δ
Policy Update (Actor)
Δθa = αa · δ · (𝟙[a] − π(a))
Action Selection
a ~ π(·|θ) (softmax over θ)
Early: V ≈ 0 so δ ≈ r (same as REINFORCE). As V → R, δ shrinks — lower variance, same expected gradient.
reward size →
Actor-Critic = REINFORCE + a running average
REINFORCE weights each update by raw r; Actor-Critic weights by surprise (r−V). Same expected gradient ∝ R, lower trial-to-trial noise.
αa = 0.002 (actor) · αc = 0.02 (critic)