Skip to content
← All writingKaon Labs / Research

Online RL: Learning From User Behavior in Real Time

We train our generative model directly on the live conversations and interactive stories it is running with real users. Behavior is the only reward, and updated weights reach the serving engines within about a second of every training step.

Online RL on live production traffic has long been something of a myth: discussed in principle, while post-training in practice still runs offline on static snapshots. We first saw credible production accounts of it from Cursor, first with Cursor Tab and later with Composer, where accepted and rejected code edits provide a direct reward. More recently, Applied Compute published a recipe for production RL, demonstrated on chess puzzles with a verifier that knows the right move. We have been running online RL in production on a different kind of traffic: people talking to characters and consuming interactive stories on our products. There is no verifier, and nobody labels anything. The only signal is what the person does next.

What we want is a continual learning system. Once a model ships, it should keep learning from the users it serves and put what it learns back online immediately. In a recommender system, the part that learns online is a ranking layer over fixed items. Here, the model is the content. What it learns is how to write, how to catch what a user is reaching for, and how to keep the story and conversation going. Every interaction, including the implicit signal that a user stayed or left, becomes an update to the policy generating the next turn.

The end state is a model that adapts to each user. Different people want different pacing, length, and style from the same character or story. Online RL reaches that the same way continuous training works in a recommender system: as users interact with the product, their revealed preferences are folded continuously into the weights alongside user representations, moving from one global policy toward group- and individual-level adaptation rather than freezing a model at release. Two engineering problems make that hard:

  1. Infrastructure: capturing exact token trajectories, archiving, batching, training, and pushing weights back to serving without pausing live traffic or letting data go stale.
  2. Reward and optimization: learning from sparse, noisy, gameable behavioral signals—under a single rollout per live user turn and low-precision serving—without learning to game the signal instead.

This post covers the system architecture, how we keep low-precision serving and asynchronous training aligned, how we move from grouped session outcomes to single-rollout turn-level optimization, how we monitor live runs, and what broke along the way.

Infrastructure: non-stop serving

Real users

A slice of live traffic through the A/B platform. Implicit behavioral feedback only; no labels or verifiers.

Servingnever stops

Gateway

OpenAI-compatible. Pins each conversation to one engine. Journals requests on a bypass and drops capture before it slows a user.

Engine 1

FP8 + LoRA

vk

Engine 2

FP8 + LoRA

vk

Engine 3

FP8 + LoRA

vk

… Engine N

FP8 + LoRA

vk

TITO capture: each engine records the exact sampled token IDs, behavior probability q(y), top-k head, and weight version.

Trainingseparate machines

Trainer

  • Updates the LoRA adapter only
  • External rollout: waits for next batch
  • TIS + Score Centering for off-policy & FP8 mismatch
  • One optimizer step per batch
vk → vk+1
Data pathbypass, resumable

Local journal

requests and token-level evidence

Archive

shared storage, incremental, indexed

Batch builder

trajectory → reward module → advantage (grouped or online critic) → batch, lag ≤ K

Serving always runs the newest version; each training step consumes finished conversations from earlier versions and puts the next version live about a second later.ServingTrainingv0 serves usersv1 serves usersv2 serves usersv3 serves usersv4 …step 1step 2step 3step 4finished conversations become a batchnew weights live, about one second
Figure 1. The loop. Serving and training run on separate machines and never wait for each other. Two channels connect them: token-level evidence flows from the engines through the journal, archive and batch builder to the trainer, and the updated adapter is broadcast back to every engine after each step. Below, the same loop in time: serving always runs the newest version, and each training step uses finished conversations from the previous few versions.

Serving and training run on separate machines connected by only two asynchronous channels, and neither side waits for the other:

  • Data channel (serving → training): Every engine records, on a bypass, the exact tokens it sampled, the behavior probabilities it sampled them under, the top-k distribution head, and the weight version that produced each token. Records land in a local write-ahead journal, mirror incrementally to shared storage as checksummed shards, and feed a batch builder that assembles trajectories, invokes the reward module, computes advantages, and packs training batches.
  • Weights channel (training → serving): After each optimizer step, the trainer broadcasts the updated LoRA adapter to every serving engine over NCCL in about a second. Engines pause in place, keep their KV caches and in-flight requests, upsert the adapter without unloading, and resume immediately on version vk+1v_{k+1}.

Six design principles govern this loop:

  1. Never stop serving. Weight updates, data capture, and training cannot pause live traffic or reset the running experiment.
  2. Single-source sampling config. Engine sampling settings are generated directly from the trainer configuration so behavior and training distributions match by construction.
  3. Version-tagged tokens. Every generated token carries <experiment>:<step>. Staleness bounds, off-policy correction, experiment isolation, and post-hoc exclusion of bad windows all depend on that tag.
  4. Best-effort bypass capture. Journaling runs on an asynchronous bounded queue; under load, capture drops before user latency suffers.
  5. Resumable, commit-after-checkpoint delivery. The data pipeline delivers at least once with downstream deduplication, and a batch is committed in the ledger only after the checkpoint trained on it is on disk.
  6. Low-precision first. Serving (SGLang) and training forward passes (Megatron) use the lowest precision the hardware supports—FP8 today (with BF16 LoRA weights), and FP4 on B300 next—absorbing numerical mismatch in the algorithm rather than paying full-precision compute.

In our deployment, because SGLang built-in data parallelism (--dp-size) does not support distributed dynamic LoRA updates, serving runs as a pool of single-GPU SGLang instances behind an OpenAI-compatible gateway that pins each conversation to one engine. The Megatron trainer performs no rollouts of its own; it blocks on the next batch and takes one optimizer step per batch.

Data: token-in, token-out (TITO)

Rebuilding training sequences by re-tokenizing stored transcripts breaks online RL. What the model conditioned on at generation time is not always identical to what is stored in chat history afterwards—for instance, stripped empty thinking blocks, template normalization, or later edits. Training on reconstructed text reinforces token choices the policy never actually made.

Three data rules keep replay faithful:

  1. Token-in, token-out (TITO). Following the same rule Applied Compute calls TITO, a reply's training sequence is the exact token IDs emitted by the serving engine, paired with its recorded behavior probabilities. Each eligible assistant turn is trained against the exact prefix the engine saw when generating it.
  2. Truncation never teaches termination. A reply cut off by max_tokens has no end-of-sequence token, and none is appended, so the policy is never taught that hitting a length cap was a choice to stop.
  3. Bounded version staleness (lag≤K\mathrm{lag} \le K). Online RL is off-policy by construction because behavioral outcomes mature after generation. Batches consume newest trajectories first, enforce a strict upper bound KK on policy version lag, and exclude any window where an engine restarted onto unadapted base weights.

Reward: an isolated module

We do not disclose our specific reward formulation. Architecturally, the reward is a replaceable module inside the batch builder with a narrow contract:

Reward module contract
input
session or turn trajectory (context, token evidence, behavioral signals)
output
reward rtr_t per eligible reply, or discard

Downstream advantage estimation and policy gradients depend only on rtr_t. Isolating that boundary let us iterate on how user behavior is turned into session- and turn-level rewards without touching serving, capture, or the trainer. Because any implicit behavioral signal is gameable, guardrails such as reply length, regeneration behavior, and safety classifiers are evaluated alongside the primary metric.

Algorithm

1. Serving–training consistency: TIS + Score Centering

Even with TITO and bounded staleness KK, the trainer's probability p(y)p(y) for a sampled token diverges from the serving engine's behavior probability q(y)q(y) for two reasons: the trajectory was generated up to KK policy versions ago, and FP8 SGLang inference is numerically distinct from FP8 Megatron training.

To correct both at once, each engine records the true behavior probability q(y)q(y)—renormalized over the exact top-p and top-k support actually sampled from—along with the top-k distribution head. Training minimizes a per-token objective combining Truncated Importance Sampling (TIS) and Score Centering (SC):

Ltoken=−A⋅(wlog⁡p(y)−C)(w,C,A stop-grad)w=min⁡ ⁣(p(y)q(y), C)TIS (upper-only clip)C=∑y′∈Hk(m(y′)−α p(y′)) log⁡p(y′)Score Centering on top-k head\begin{aligned}\mathcal{L}_{\mathrm{token}} &= -A \cdot \bigl( w \log p(y) - \mathcal{C} \bigr) && \textcolor{#8f8f9e}{\text{\small ($w, \mathcal{C}, A$ stop-grad)}} \\[6pt]w &= \min\!\left(\frac{p(y)}{q(y)},\, C\right) && \textcolor{#8f8f9e}{\text{\small TIS (upper-only clip)}} \\[6pt]\mathcal{C} &= \sum_{y' \in \mathcal{H}_k} \bigl(m(y') - \alpha\, p(y')\bigr)\,\log p(y') && \textcolor{#8f8f9e}{\text{\small Score Centering on top-}k\text{\small{} head}}\end{aligned}

where m(y′)=min⁡(p(y′), C q(y′))m(y') = \min\bigl(p(y'),\, C\, q(y')\bigr), α=min⁡ ⁣(C qtailptail, 1)\alpha = \min\!\left(C\,\frac{q_{\mathrm{tail}}}{p_{\mathrm{tail}}},\, 1\right), and ptail, qtailp_{\mathrm{tail}},\, q_{\mathrm{tail}} are the residual probability masses outside the top-kk head Hk\mathcal{H}_k.

Score Centering subtracts the non-zero drift gradient that appears whenever p≠qp \neq q even under constant reward, while TIS caps the upper importance ratio p/qp / q so stale or quantized tail tokens cannot blow up gradient variance. We set the explicit KL penalty coefficient to zero and monitor reference KL passively. In delayed-window asynchronous FP8 ablations before launch, TIS + SC outperformed pure SC and matched or beat two-sided MIS masking, becoming our default off-policy loss.

Running this objective in FP8 surfaced two practical lessons:

  • Adapter drift requires periodic merge-and-reset. Off-policy mismatch compounds as the LoRA norm grows: continuing a single adapter indefinitely caused online gains to erode. Merging the adapter into the base weights and resetting the adapter, optimizer, and reference anchor recovered and exceeded the earlier peak.
  • Low precision introduces a reference-KL floor. Dynamic activation quantization in FP8 forward passes produces a non-zero reference KL even when weights have barely moved. An early run at a conservative learning rate looked dead because that floor hid the update; replaying a single batch confirmed every adapter parameter moved by the exact step size, showing the pipeline was sound and the learning rate was too small.

2. Learning from a single live rollout: from grouped baselines to SAO

Standard RL recipes for LLMs such as GRPO sample a group of GG candidate replies for the exact same prompt to compute a prompt-local baseline. On live traffic, a user at a given point in a story experiences exactly one reply—you cannot replay the person GG times from the same state.

We progressed through two single-trajectory credit-assignment designs:

  • Grouped session baselines (critic-free). In our initial closed-loop test, we grouped finished sessions from comparable contexts within each batch so positive and negative outcomes baselined each other without a value network, controlling for prompt difficulty.
  • Single-Rollout Asynchronous Optimization (SAO). Session-level grouping is coarse, and naive single-sample PPO baselines trailed group-sampled GRPO in our controlled benchmarks. Following Hou et al. (SAO), our mainline turn-level loop pairs each live reply's reward rtr_t with an online critic V(st)V(s_t) that estimates expected return from the conversation prefix:
At=rt−V(st)A_t = r_t - V(s_t)

Every time the policy trainer consumes a live batch, the critic updates once on that same stream. Rather than SAO's original double-sided DIS clipping, we keep TIS + Score Centering so the same off-policy FP8 loss serves the entire stack. Across two independent controlled ablations, updating the critic online more than doubled the online depth lift compared with a frozen critic, while holding reference KL substantially lower than group-sampled GRPO, where late-training KL spikes collapsed performance.

3. Advantage balancing and the length hack

Turning trajectory rewards into token gradients requires strict alignment between advantage centering and loss normalization. In our grouped run, session depth rose sharply while average reply length fell steadily across training.

The cause was arithmetic. Averaging loss over tokens within each reply before averaging across replies gives every token weight 1/L1/L. Because positive replies outnumbered negatives in the batch, the mean advantage E[A]\mathbb{E}[A] was positive, injecting a c⋅∇E[1/L]c \cdot \nabla \mathbb{E}[1/L] (c>0c > 0) term into the expected gradient that rewarded shorter replies regardless of quality—the length bias analyzed in Dr. GRPO.

Fixing it established two rules, paired with sign-preserving mass redistribution and per-sample clipping adapted from the MiMo-V2.6 Technical Report:

  1. Zero-mean advantage mass within each comparison group. Positive and negative advantage mass sum to zero within each comparison group, with heavier penalty weights on later turns in negative trajectories.
  2. Match the unit of loss aggregation to the unit of advantage balancing. Fix A (global per-token denominator) and Fix B (sample-balanced zero-mean mass) each eliminated length shrinkage alone. Applying Fix A + Fix B together was worse than either: per-token aggregation weights each sample proportionally to its length while Fix B balances by sample count, over-amplifying long positive replies until nearly every training sample hit max_tokens.
VariantSession depthReply lengthMechanism
Continued adapterGain erodesShrinksServing–training drift accumulates
Merged restartStrongest raw gainShrinksPartially inflated by 1/L1/L length bias
Merged restart + Fix A (per-token loss)Positive, smaller gainStableLength bias removed
Merged restart + Fix B (balanced advantage)Positive, smaller gainStableLength bias removed
Merged restart + Fix A + Fix BWeakest gainHits length capAggregation unit (token) ≠ balancing unit (sample)

Removing the hack cost roughly half of the apparent turn gain. We keep the half that is real.

Results

Turns per session (relative lift vs. production baseline)

-10%0%+10%+20%baseline+1.6%step 10stale buffer+12.0%step 20newest-first+14.7%step 30merged restart
Figure 2. Relative lift on turns per session over the frozen production model across our initial closed-loop run in Session A/B (prior to the §3 length-bias fix). Absolute sample sizes and metric values are omitted; the shaded band shows 95% confidence intervals and filled points are statistically significant. Step 10 trained on stale buffered data; step 30 is the merged restart from step 20.

Every checkpoint is merged, exported, and evaluated against the frozen production model in Session A/B on fresh live sessions (with native capture disabled during evaluation so serving latency matches the baseline). Following our Session A/B post, absolute sample sizes and metric values are omitted; results are reported as relative lifts with 95% confidence intervals.

Closing the critic-free loop. Figure 2 shows our initial closed-loop behavioral-outcome run prior to the length-bias fix in §3. Step 10 sat within noise (+1.6% [−3.7, 7.8] on turns per session) because steps 0–9 accidentally consumed the oldest buffered conversations first, inflating the serving–training divergence. Switching to newest-first with bounded version lag jumped step 20 to +12.0% [5.7, 18.3], and merging the adapter into the base weights at step 20 for ten further steps pushed the raw lift to +14.7% [8.1, 21.2] without regressing regeneration rate. As analyzed in §3, part of that step-30 gain came from 1/L1/L reply-length shrinkage; applying either length-bias fix stabilized reply length and preserved roughly half of the turn lift, giving us a clean baseline before moving to turn-level SAO.

Scaling turn-level SAO. Moving from grouped session baselines to the turn-level SAO loop (online critic V(st)V(s_t) + TIS + SC) lifted live Session A/B depth substantially further than the critic-free loop and scaled faster as we increased batch size. At larger batch sizes and longer training horizons, continual optimization also surfaces subtler behavioral reward hacks beyond reply length—such as writing lines or actions on the user's behalf to force a continuation—making live behavior classifiers essential alongside primary depth metrics.

Monitoring live runs: a MiMo-style console

Online RL ConsoleSAO · TIS+SC
● training · sao-tis-sc · step 30
serving okcapture okInterface example · illustrative
step30committed · 14s ago
train/ref_kl0.041above FP8 floor
train/train_rollout_kl0.008trainer p vs. behavior q
pg_tis_clipfrac1.4%ratio > C clipped
critic/ev0.46online V(s_t) fit
perf/weight_sync0.94sNCCL in-place upsert
Session A/B · initial run turns lift %
pre-fix Phase 1 vs. baseline · 95% CI
+14.7%
0%+10%+20%0102030
train/ref_kl
K3 vs. initial anchor · reset at step 20 merge
0.0330.0390.060
0.0000.0400.080FP8 floor2102030
train/train_rollout_kl
drops at step 12 after switching oldest → newest-first
0.0080.0110.020
0.0000.0200.0402102030
data/reply_length_index
relative to baseline · catches 1/L length shrinkage
1.01×1.00×0.71×
0.60×0.80×1.00×2102030
noticesoperational & guardrail log
16:40evalsao-tis-sc step 30 merged & exported → Session A/B gate queued
15:52resetstep 20 LoRA merged into base weights; adapter, optimizer & ref anchor reset
14:18gategrouped-unbalanced stopped at step 20: reply-length guardrail regression (1/L bias)
13:05rulebatch builder switched from oldest-first buffer to newest-first within lag ≤ K
12:24exclexcised 4m engine restart window (base-weight fallback) via version tag filter
next batch · step 31committed: step 30 (after ckpt)
78% of target trajectories ready
policy version lag mix (lag ≤ K)mean lag 0.61
lag 0 · 54%lag 1 · 31%lag 2 · 15%> K · dropped
Figure 3. Interactive mock of our MiMo-style live online-RL console (sanitized, illustrative telemetry). Switch tabs or toggle runs to inspect off-policy KL diagnostics, batch version-lag composition, operational notices, and asynchronous safety and reward-hack detectors.

Inspired by the public RL training board Xiaomi released alongside MiMo-V2.6, we built a unified live console (Figure 3) that puts trainer dynamics, batch freshness, and live stream detectors on one screen, fed asynchronously from the local journal replica so monitoring adds zero latency to user requests:

  1. Off-policy and low-precision actor telemetry (TIS / SC). Alongside live Session A/B checkpoint lifts, the console plots train/ref_kl (separating genuine policy movement from the FP8 quantization floor), train/train_rollout_kl between serving q(y)q(y) and trainer p(y)p(y), directional TIS clip fractions by advantage sign, effective sample size (sc/ess_fraction), Score Centering head magnitude, and per-tensor LoRA zero-gradient checks.
  2. Batch accumulation, version-lag mix, and operational notices. A live sampler panel separates the last checkpoint-committed batch from the next batch currently accumulating, breaks down every step's policy-age mix (lag 0 through lag≤K\mathrm{lag} \le K), and pairs the curves with a timestamped notices log of adapter merges, builder rule changes, and excised fallback windows.
  3. Out-of-band behavior and safety detectors. Asynchronous sidecar classifiers and rule detectors annotate the replicated stream continuously—tracking safety categories, relative reply length, within-session repetition, and reward-hack signatures such as writing lines on the user's behalf—flagging regressions before the next formal evaluation gate completes.

Is it safe to train on live users?

The safety is in the evaluation and gating architecture, not in hoping online gradients behave:

  • Safety evaluations on all traffic-serving checkpoints. Just like candidate models in regular online evaluation, checkpoints serving live traffic run through our safety evaluations, supplemented during training by lightweight classifiers monitoring safety categories and reward-hack behaviors on the live stream.
  • Bounded traffic slice inside Session A/B. The online policy serves only a bounded experiment slice; the frozen production model serves the rest of the product, including the control arm every evaluation compares against.
  • Versioned post-hoc exclusion. Because every token records <experiment>:<step>, any window affected by a serving fallback or bad update is excised cleanly from downstream training batches.
  • Scheduled guardrail gates and instant rollback. At fixed step intervals, merged checkpoints are tested on fresh live sessions against the production baseline across primary and guardrail metrics—which is what caught the length hack on day one. Two consecutive regressions stop the run, every evaluated checkpoint is retained for rollback, and promotion beyond the experiment slice requires passing the same release gates as any offline-trained model.

Two directions remain open:

  1. Richer behavioral credit assignment. Implicit user behavior spans immediate turn-level reactions and longer-horizon session outcomes; combining horizons while keeping the signal resistant to subtle reward hacking is a valuable direction.
  2. From one global policy to individual weights. The runs above update a shared policy across the traffic slice; conditioning updates on user representations so preferences fold into user-specific behavior in real time is the next step.

Sources

Interested in this work?

We’d love to hear from you. For research questions or collaboration, reach out to alex@kaonlabs.com. If you’d like to help build what’s next, explore our open roles.

Continue reading

Rubric Generation in an Open Domain: From Reward Model to Verifiable Rubrics

Online Evaluation at Scale: How We Evaluate 100+ Model Variants per Week

Reading a roleplay model before it speaks: what differs after post-training

Build the medium while
it is still becoming.

Join the team building consumer AI across product, research, and infrastructure.

View open roles