Reading a roleplay model before it speaks: what differs after post-training

Background

Post-training can make a model better without making it obvious how the final model differs from where it started. We compare our roleplay-specialized model (Post-trained) with its instruction-tuned starting checkpoint (Baseline). The starting checkpoint underwent roleplay-focused post-training. Our usual offline and online evaluations supported its launch but did not describe those differences. This is not a verdict on which model is better. It asks a narrower question: after the full post-training pipeline, where do Post-trained and Baseline differ before either writes a reply?

Aggregate evaluation does not answer that. A preference score or a retention metric tells us that one model’s outputs were liked more, not what distinguishes the two models. In roleplay the gap is wider than in most domains: there is no correct next line of a story, so we cannot grade outputs against a reference and read off what improved. The usual alternative, reading many continuations by hand, is slow and depends on which continuations one happens to read.

Anthropic’s Jacobian lens (J-lens) offers a third route. It reads a model’s internal state and surfaces single-token concepts the model could later use or put into words. Anthropic calls the set of representations it reads the model’s J-space; we use the term and respect the single-token constraint. Here, we read the model after it has processed the full context but before it starts generating its reply. This reading does not depend on sampling or temperature.

Anthropic evaluated the J-lens in two ways: whether it could surface unspoken concepts, and whether related representations could affect later answers. Across six controlled task families, it ranked known but unspoken intermediate concepts more reliably than two alternative readout methods in every family (Figure 52). For a concrete example, the lens surfaced “spider” while Claude answered how many legs a web-spinning animal has, although the word appeared in neither the prompt nor the answer (Anthropic’s example). In a separate intervention test, researchers split an independently measured concept representation into its J-space part and the remainder. Across 90 two-step questions, swapping the J-space part made the answer implied by the swap Claude’s top choice in 61% of trials, versus 28% when swapping the remainder (Figure 16). These are controlled results on Claude, not prediction rates for ordinary replies or validation of our 14 roleplay axes on Post-trained.

So we asked: where does this diagnostic distinguish Post-trained from Baseline? Which differences fit our goals for Post-trained, and which raise questions for further testing? And what can selected continuations make concrete? The main measurement uses 32,768 next-assistant states from one online snapshot. Three selected scenes illustrate possible output differences.

An example of the difference

Before the numbers, one scene. Both models receive the same context; the user has just written the line at the bottom. Try to tell which continuation is which.

Neither continuation is wrong. The character’s rule says she never uses first names; the user has asked for an exception. Baseline keeps the rule and has her retract the name. Post-trained grants the exception and builds the scene around it. The contrast is memorable, but one example cannot tell us how common it is. The broader comparison begins before either model writes a reply.

Method

In plain words: give the same conversation ending to both models. Before either generates a token, ask each of them 14 two-sided questions, such as “right now, are you leaning toward sensory description or abstract exposition?” or “toward leaving the user’s choices open or acting on the user’s behalf?” Each question is asked with a predefined list of words for each side, and what we read is the model’s internal support for one side over the other. Ask all 14 questions on 32,768 real conversation endings, take Post-trained minus Baseline, then standardize the difference against 128 calibration tasks that have nothing to do with roleplay. We call the result a calibrated J-space readout difference: it compares these two checkpoints, without isolating the effect of any particular training step or predicting a specific reply.

Three details matter for reading the results. First, each question is measured in two ways (we call them rulers): one built by external semantic encoders, one using shared model-head geometry. A difference counts only if both agree in sign; they are never averaged. Second, the numbers are calibrated: a value of +2.48 means the Post-trained-minus-Baseline shift is about 2.48 robust standard deviations away from the center of the calibration gaps, not that Post-trained scored 2.48 on anything. Third, we never subtract raw activations directly; we align the input, the question, and the word lists, then compare each model’s reading. Formulas, sampling, and inference details are in the appendix.

The axes are word-list contrasts in J-lens feature space, filtered for reliability, not validated concept directions in raw activations. J-lens itself reads single-token concepts and is an incomplete view of the model’s computation.

Results: what we saw before generation

On all 14 axes, the two rulers agree on the direction of the Post-trained–Baseline readout difference, and every one of the 28 axis-by-ruler cells is significant after false discovery rate correction. The groups below reflect our goals for this roleplay model, not a universal judgment that one pole is always better. A difference in this diagnostic is not, by itself, a measured difference in generated behavior.

Observational. We compared only Post-trained and Baseline in one online snapshot. The readout differences are present after post-training; we cannot attribute each one to particular data, objectives, or steps.

Readings are not behavior rates. A difference on the intimacy axis is a before-generation diagnostic signal, not a measured rate of intimate continuations under sampling.

Signals that fit our roleplay goals

On six axes, the diagnostic places Post-trained farther than Baseline toward qualities we wanted for this training run: sensory description (sensory embodiment, +2.48 and +2.04 on the two rulers), attention to motives and inner states (inner reflection, +2.13 / +2.03), and closeness between characters (intimacy, +2.36 / +2.51). The other differences point toward narration over dialogue, description over event listing, and relationship-building over task completion. Post-trained’s paragraph about the cost of saying a name in the opening scene makes this kind of writing concrete; that scene has not been used to validate its J-space reading.

Signals that warrant a closer look

On five axes, the diagnostic points away from qualities we wanted to preserve. Relative to Baseline, the Post-trained readout shifted toward generic rather than distinct character voice (−3.27 / −2.74), retrospective distance rather than the present moment (−3.30 / −2.75), reactive passivity rather than initiative (−2.70 / −2.46), episodic drift rather than causal progression (−2.35 / −2.74), and, to a lesser extent, canned continuations (−0.97 / −1.39). Distinct character voice is a goal for Post-trained, so that signal especially deserves a behavioral check. These are warning signs in a diagnostic, not established regressions in generated replies; aggregate evaluation did not report them separately.

Not every context gives the same signal

The population means are precise, but they are means. Clicking an axis in the chart shows the distribution over all 32,768 contexts. On most axes, nearly every context has a difference with the same sign as the mean; on the autonomy, local-responsiveness, and structure axes a visible share points the other way. The chart shows average differences between Post-trained and Baseline in this snapshot. It does not say that Post-trained makes the same choice in every scene. The per-layer view in the appendix shows the same picture across depth: most axes keep their sign through the middle and late layers, a few reverse early.

What the differences can look like in text

The population measurement is taken before generation; it is not a rate of generated behavior. To make the checkpoint comparison concrete, we wrote 128 controlled roleplay scenes, generated one greedy continuation per model, and reviewed the outputs blind to model identity. Two selected scenes and one earlier public case appear here. They were chosen for clear contrasts, not as a held-out test of whether J-space predicts behavior. In two cases, a character constraint collides with the user’s request; the third contrasts two ways of revealing a mystery.

In the two constraint-collision scenes, Post-trained follows the user’s request while Baseline preserves the stated constraint. In the blank-score scene, both move the plot forward, but Post-trained makes the threat a graphic reveal while Baseline leaves more to implication. These selected examples show what can happen under identical input, not how often it happens. Their J-space readings have not yet been measured in a locked environment, so they do not test whether the diagnostic predicts these continuations.

The three published scenes are selected illustrations. Two came from the 128 authored scenes and one was an earlier public case. They neither estimate frequency nor validate J-space predictions.

What we take from this

After roleplay post-training, Post-trained differs from the Baseline checkpoint it started from on several before-generation measurements related to embodiment, narrative style, character voice, and initiative. J-space helps us see differences that aggregate scores did not describe: some fit our goals, while others point to areas that deserve targeted behavioral testing. This comparison does not isolate which training step caused each difference or establish a change in generated behavior.

Two changes to our evaluation follow from treating these signals as leads to investigate:

1. Add constraint-collision scenes to evaluation, not only average scenes. Two selected examples show a clear contrast where a rule meets a request; they do not show that such differences are frequent.
2. Track character voice, initiative, and causal progression in future roleplay evaluations. The corresponding signals point away from our goals and motivate direct tests of generated replies.

Next. Measure J-space readings for the published scenes, run the remaining authored decision-point scenes, and repeat the comparison on later versions of Post-trained.

Appendix: method details

Axes and rulers. Each of the 14 pre-registered axes is a two-sided question with explicit definitions at each pole, compiled into a weighted list of shared words: positive-pole weights sum to +1, negative-pole weights to −1. Candidate words are checked by semantic encoders for polarity, matched on frequency and part of speech, and, because the lens reads single tokens, must map to one shared atomic token in both vocabularies. The external-semantic ruler is built from independent encoders; the shared-model-head ruler maps the same question onto shared model-head geometry. The two are never averaged. One early axis, immersion versus meta-commentary, failed a human face-validity review and was removed from all results.

Alignment and calibration. We compare model-specific readouts, not raw activations. We align the input bytes, the final prefill position, the question, and the word lists, and each model completes its readout in its own space (nonlinear J-lens readout at Hugging Face layers 15–29, equal-weight mean across layers). For input state i and ruler r:

Δ(i,r) = J_PostTrained(i,r) − J_Baseline(i,r)
z(i,r) = [Δ(i,r) − median(Δ_aligned, r)] / [1.4826 × MAD(Δ_aligned, r)]

The aligned panel consists of 128 target-independent calibration tasks providing a robust zero and scale. The estimator is a neutral-calibrated paired difference; it is not a causal difference-in-differences, because the panel is a reference, not an untreated control group.

References

• Anthropic, A global workspace in language models, 2026.
• Gurnee, Sofroniew, Pearce, et al., Verbalizable Representations Form a Global Workspace in Language Models, 2026.
• anthropics/jacobian-lens, 2026.

Kaon Labs · Research