E2S Finetuning: From Expert Demonstrations to On-Policy Learning

Expert responses can contain valuable information yet follow reasoning paths the student model would rarely produce. Training it to imitate those paths can require a large change in its behavior.

E2S keeps the required expert information while favoring answers the model is already more likely to produce.

MCMC can generate such answers by searching separately for each example. E2S learns a reusable sampler across examples; the student still learns through SFT.

Off-policy SFT creates a distribution mismatch

For a fixed prompt, let \(q_{\mathrm{data}}\) be the distribution of expert responses and \(p_\theta\) the student’s response distribution. SFT minimizes the expected negative log probability of those responses:

SFT pulls the student toward the data distribution. The entropy term is fixed, so minimizing the loss minimizes this KL divergence. In the population limit, a sufficiently expressive model can fit \(p_\theta=q_{\mathrm{data}}\). Fitting distant data means moving toward a distant policy.

Some policy change is necessary to learn a new task. Imitating an expert’s particular trajectory can demand additional change. Larger shifts have been associated with more forgetting, motivating training data closer to the student; the KL identity alone does not prove that other skills will be lost. [1] [2]

The same mismatch appears word by word: SFT asks the student to continue the expert’s reasoning, even when the student would have written something different up to that point.

Existing approaches to the distribution mismatch

Both routes address the same mismatch, but they change different parts of training: where supervision is applied, or which responses become training targets.

Move learning to the student — RL / OPD. Let the current student generate a response, then provide feedback on that trajectory. RL scores the student’s answers with rewards. In on-policy distillation (OPD), a teacher reads what the student has written so far and provides probabilities for the next token. Feedback is applied to the student’s own attempt. [3] [4]

Move the data to the student — MCMC + SFT. Use the expert constraint to define which responses are valid, and the student policy to define their relative probabilities. Sample from that constrained distribution, then train the student on the resulting responses with SFT. Here, we change the training targets themselves. MCMC and E2S are two ways to obtain them. [5]

Rejection sampling can also produce the constrained target: draw from the student and keep valid responses. It wastes most rollouts when success is rare. MCMC instead starts from an expert solution and repeatedly proposes, scores, and accepts or rejects edits—the route developed in Finetuning with Sampling. [5]

Write \(x\) for the prompt, \(\tau\) for its off-policy expert trace, and \(y\) for a new response that satisfies the expert constraint while following the student’s relative probabilities. Both MCMC and E2S turn \(\tau\) into a more on-policy \(y\), then use \((x,y)\) for student SFT. The expert trace guides data preparation; it is the new response \(y\) that becomes the SFT target.

E2S takes this second route. MCMC runs a new chain for every example. That repeated search is the motivation for E2S. Instead, we train a conditional sampler across prompts: work on one example changes the sampler used for the next.

MCMC revises one response at a time; E2S improves a sampler that can generate responses for many examples. E2S moves part of the repeated search cost into sampler training. Whether that saves total compute depends on the training cost and how much useful data the sampler subsequently generates.

The target distribution: preserve expert information with minimal policy change

The expert semantic constraint requires responses to be semantically equivalent to the expert response. Following Finetuning with Sampling, let \(C_\tau\) contain responses satisfying this constraint for expert trace \(\tau\). Math and science use correct outcomes; knowledge tasks require preserving key facts. [5]

For prompt \(x\), write \(p(y)=p_{\mathrm{ref}}(y\mid x)\) for the frozen student’s response probability. The expert requirement selects valid responses; the student determines their relative probabilities.

Among normalized distributions \(q\) supported on this valid set, choose the one closest to the student: [5]

Let \(Z_\tau=\sum_{y\in C_\tau}p(y)>0\) be the student’s total probability of producing a valid response. The solution is:

The solution preserves the student’s relative probabilities among responses that satisfy the expert requirement. The opening illustration keeps both valid modes in their original proportions and removes responses outside the requirement.

For any feasible distribution with finite KL, substituting \(p(y)=Z_\tau q^*(y)\) on the valid set gives:

The second term is constant; the first is minimized at zero. Therefore \(q=q^*\).

Suppose the student assigns 6% to one valid solution and 3% to another, with all remaining responses invalid. Conditioning gives them two-thirds and one-third of the training mass. We remove invalid responses without erasing the student’s 2:1 preference between the valid ones.

We call these responses more on-policy because they retain the student’s relative probabilities within the valid set. Formally, the target is the constrained student distribution, rather than its unconstrained policy.

Amortize the target with GFlowNet

We can score a sampled response, but computing \(Z_\tau\) would require summing over all valid responses. We need to match a distribution known only up to an unnormalized score. We use GFlowNet as the training method for this amortized sampler. [6]

Define the target score \(R_\tau(y)=p(y)\mathbf 1[y\in C_\tau]\). First consider a normalized sampler \(q_\phi\) supported on the valid set, where \(\phi\) denotes its learned parameters. Exact matching requires, for every valid response:

The right-hand side does not depend on the response. Matching the target means making this log gap the same everywhere. Trajectory balance enforces that condition with a shared offset \(z_\tau\): [7]

If the residual is zero across the valid set, normalization forces \(z_\tau=\log Z_\tau\). For text generated one token at a time, a response follows a single sequence of token choices. Its probability is the product of those token probabilities, including the end-of-response token. Earlier GFlowNet work applies this connection to language and visual reasoning. [8] [9]

Remove the normalizer with a group of responses

Draw \(K\ge2\) valid responses for the same prompt and expert trace. Define each log gap as \(a_i=\log q_\phi(y_i\mid x,\tau)-\log p_{\mathrm{ref}}(y_i\mid x)\), and let \(\bar a\) be their mean. The offset minimizing the group’s squared residuals is \(z=-\bar a\). Substitution gives:

Relative to the group, an above-average gap means a response is overrepresented; a below-average gap means it is underrepresented. The loss adjusts those relative probabilities without a separate normalizer network. Related group-relative objectives appear in GFlowNet reasoning methods. [10] [7]

E2S-Offline and E2S-Online

The sampler sees the prompt and expert trace. The student scores responses from the prompt alone. Expert information guides data preparation; it is not an additional input the student can rely on at evaluation time.

E2S-Offline: fit once, generate a dataset, then SFT

E2S-Offline fits the sampler against a fixed starting student, generates a dataset, then fine-tunes on that fixed dataset. All generated targets refer to the same student checkpoint.

E2S-Online: generate data during training

After a student update, the constrained target changes too. E2S-Online refits the sampler against the current student, generates a fresh batch, and runs SFT. Repeating this cycle lets data generation track the evolving policy.

Implementation: one backbone, LoRA sampler

We implement the sampler using LoRA adapters on the student backbone. [11] With the adapters enabled, the model generates responses conditioned on the prompt and expert trace. With them disabled, the student scores those responses from the prompt alone or receives SFT updates.

For a weight matrix \(W_\theta\), the sampler uses \(W_\theta+sBA\). Here \(B\in\mathbb R^{d_{\mathrm{out}}\times r}\), \(A\in\mathbb R^{r\times d_{\mathrm{in}}}\), and \(s\) scales an update of rank at most \(r\). During sampler fitting, the backbone stays frozen. During student SFT, the adapters are disabled.

Input:  expert batch B, frozen student θ,
        sampler φ, group size K ≥ 2,
        update budget U, learning rate ηφ
Output: updated sampler φ

for u = 1, …, U do
    Draw (x, τ) from B
    Sample y₁, …, yK ∼ qφ(· | x, τ),
        retaining only Valid(x, τ, yi) = true
    for i = 1, …, K do
        ai ← log qφ(yi | x, τ)
              − log pθ(yi | x)
    ā ← (1/K) Σi ai
    L ← (1/K) Σi (ai − stopgrad(ā))²
    φ ← φ − ηφ ∇φ L
return φ

Input:  expert corpus D, student θ₀,
        sampler initialization φ₀,
        sampler budget U, samples per example m
Output: fine-tuned student θ

φ ← FitSampler(D, θ₀, φ₀, K, U, ηφ)
D* ← ∅
for each (x, τ) in D do
    Draw m valid responses y ∼ qφ(· | x, τ)
    Add each pair (x, y) to D*
θ ← SFT(θ₀, D*)
return θ

Input:  expert corpus D, student θ₀,
        sampler initialization φ₀, rounds T,
        per-round sampler budget U,
        samples per example m
Output: fine-tuned student θT

θ ← θ₀; φ ← φ₀
for t = 1, …, T do
    Select expert batch Bt from D
    φ ← FitSampler(Bt, θ, φ, K, U, ηφ)
    Dt* ← ∅
    for each (x, τ) in Bt do
        Draw m valid responses y ∼ qφ(· | x, τ)
        Add each pair (x, y) to Dt*
    θ ← SFT(θ, Dt*)
return θ

Experiments: does E2S learn more without forgetting more?

Setup

We follow the math setting in Finetuning with Sampling: Qwen2.5-3B, MATH levels 3–5, 8,230 training problems, and 1,024 held-out MATH problems. Math avg. is the equal-weight mean of MATH, AMC, MATH500, and GSM8K; Prior avg. averages Chemistry, MMLU, and GPQA. [5]

The baseline scores come from that study; the E2S rows are our project results. Offline and online use the same starting checkpoint, training corpus, and evaluation suite. The comparison measures accuracy and retention, rather than speed at matched total compute.

Results

Offline. E2S-Offline improves Math avg. by 3.9% relative to MCMC + SFT, while keeping the evaluated prior-task average slightly closer to the starting model. It learns more from expert data while retaining prior skills at least as well in this comparison.

MCMC starts a new search for each example. E2S learns a sampling strategy across examples and reuses it to generate training responses. The practical advantage is that search experience becomes reusable knowledge in the sampler, rather than remaining in a separate chain for each prompt.

Online. E2S-Online adds a 7.0% relative improvement over offline, or 11.2% over MCMC + SFT, while matching the starting model’s displayed Prior avg. The trajectory below tracks online learning; the horizontal lines mark the other methods’ final scores.

Conclusion

These results support adapting the training-data distribution as a way to improve the learning–retention tradeoff. Expert information can support new-task learning without a comparable decline in the evaluated prior-task average. The online result further supports treating data generation as part of learning, rather than only preparing targets once.

The comparison evaluates complete training schedules. A controlled comparison at equal compute is needed to isolate which parts of online training drive the gain; an average over three prior benchmarks also does not establish retention of every capability.

Diversity scaling: more responses from the same expert examples

One expert example can teach the student through more than one response. E2S learns a distribution of valid responses, so we can draw again from the same expert example rather than simply copy one answer. The question is whether those additional draws keep improving SFT.

Setup

We keep the 8,230 expert problems and the fitted offline sampler fixed, then generate 1, 2, or 4 valid responses per problem. That gives 8,230, 16,460, or 32,920 SFT responses without adding new expert problems. We evaluate Math avg. on the same four math benchmarks. This is a separate scaling sweep from the main method comparison above.

Results

Math avg. rises from 55.0% with one response per example to 56.3% with two and 57.2% with four—a 2.2-point gain from 1× to 4×. The expert corpus stays the same; the sampler supplies additional training responses.

Conclusion

Additional samples improve math performance in this sweep. This is a useful scaling direction: we can expand the training responses without collecting more expert solutions. Separating the benefit of different responses from extra SFT compute requires a matched-compute duplication control; accuracy alone does not measure how many reasoning paths the sampler covers.

How E2S differs from RL, OPD, and MCMC

RL starts from student trajectories and primarily optimizes reward. GRPO increases probability on better-rewarded trajectories using policy constraints and clipping. E2S explicitly specifies a constrained target and trains a sampler to match it. This is a distinction between these objectives, not a claim that RL cannot be understood through distributions: KL-regularized RL can also have a reward-tilted target. [3]

OPD gives teacher feedback on the student’s own writing. The student generates a response, and the teacher provides next-token probabilities after each part the student has written. E2S instead starts from expert information and constructs an SFT target distribution for the student. [4]

MCMC and E2S move expert information toward the student. Both target the student conditioned on validity. MCMC searches separately for each example; E2S stores reusable sampling behavior in learned parameters. [5]

This distinction matters for diversity. A binary correctness reward alone does not require every valid reasoning mode to remain represented. If the E2S target assigns mass to several modes, exact distribution matching requires preserving their relative mass. Diversity is a property of the target distribution, not an extra diversity bonus. A finite learned sampler still needs sufficient coverage to achieve that goal.

Looking ahead

Offline E2S makes data preparation specific to the learner. During mid-training or post-training, an expert corpus can be rewritten into valid trajectories that a particular student is more likely to produce. A learned sampler also makes each expert example reusable: the scaling sweep shows gains from additional valid responses. A matched-compute study can test how much of that gain comes from useful diversity rather than additional training.

Online E2S makes data generation part of learning. An expert-conditioned sampler can guide generation toward valid responses while tracking the student as it changes. The broader idea is to learn the data-generation policy itself: not only which information to teach, but how to express it for the model that will learn from it.

References

[1] Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RL’s Razor: Why Online Reinforcement Learning Forgets Less. arXiv:2509.04259v1, 2025. See §4–5 and Appendix A for the KL analysis and its assumptions.

[2] Howard Chen, Noam Razin, Karthik Narasimhan, and Danqi Chen. Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting. ICML, 2026. See §3–4 for distributional analysis and approximately on-policy SFT; Appendix A.5 discusses limits of KL as a predictor.

[3] Zhihong Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024. See §4.1 for GRPO.

[4] Kevin Lu and Thinking Machines Lab. On-Policy Distillation. Thinking Machines Lab: Connectionism, 2025.

[5] Aayush Karan, Sitan Chen, and Yilun Du. Finetuning with Sampling: SFT Learns Better Than You Think. arXiv:2610.02140v1, 2026.

[6] Yoshua Bengio. Generative Flow Networks. 2022. Discusses learning sequential construction policies and contrasts them with MCMC sampling.

[7] Xuekai Zhu et al. FlowRL: Matching Reward Distributions for LLM Reasoning. arXiv:2509.15207v3, 2025.

[8] Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples. ICML, 2025.

[9] Haoqiang Kang, Enna Sachdeva, Piyush Gupta, Sangjae Bae, and Kwonjoon Lee. GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networks. CVPR, 2025.

[10] Xiaodong Liu et al. GFlowRL: Scaling Distribution-Matching RL to Large Language Models. arXiv:2607.13394v1, 2026.

[11] Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.

[12] Siyan Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv:2601.18734, 2026.

[13] Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. UFT: Unifying Supervised and Reinforcement Fine-Tuning. arXiv:2505.16984, 2025.

Citation

If you found this post useful, please cite it as:

Murray Kang. “E2S Finetuning: From Expert Demonstrations to On-Policy Learning.” October 2026.

@misc{kang2026onpolicysft,
  author = {Kang, Haoqiang},
  title = {{E2S Finetuning: From Expert Demonstrations to On-Policy Learning}},
  year = {2026},
  month = oct,
  howpublished = {Research blog},
  url = {https://mk322.github.io/blog/learned-on-policy-sampler/}
}