Blog · Kernels

Optimizing KL loss: how to fit long context distillation into your VRAM

At long context, the largest tensor in knowledge distillation is not in the model. It is the student's vocabulary-sized logits. This post explains a kernel that computes the exact same loss and gradients without ever storing them.

Paper Code

To distill a large language model into a smaller one, we typically use a training method that compares the teacher's (large model) and the student's (scaller model) next-token distributions at every position of every sequence. The comparisson is done, by applying the Kullback-Leibler divergence formula to the two distributions. Each distribution spans the whole vocabulary, which is 128K to 256K entries in current models. At long context, the tensors holding them become the largest objects on the GPU, and the first thing that runs out of memory.

We hit this wall when we tried to distill models at 32K tokens and beyond. The fix turned out to be a loss kernel that computes the forward KL divergence and its gradients in sequence chunks, fused with the model's output projection, so that the full logit tensor never exists. This post walks through how the kernel works and what it buys.

11.6×less peak memory than a forward-chunked loss at 256K tokens
3.3×faster per iteration at 256K tokens, while computing the same loss
4 → 1nodes needed to distill GPT-OSS-20B at a 32K context

This is one part of our paper, Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss, written with Bakbergen Ryskulov, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene and Román Orús. The paper also covers offline teacher caching and the resulting compact model. Here I focus only on the kernel. The implementation is open source.

The logit wall

A language model ends with a linear head. It takes the final hidden states \(h \in \mathbb{R}^{S \times d}\) for a sequence of \(S\) tokens and projects them through the output weights \(W \in \mathbb{R}^{V \times d}\) to get one logit per vocabulary entry:

$$z = h\,W^{\top} \in \mathbb{R}^{S \times V}$$

The hidden size \(d\) is a few thousand. The vocabulary \(V\) is two orders of magnitude larger. For GPT-OSS, \(d = 2{,}880\) and \(V = 201{,}088\), so the logits are about 70 times larger than the hidden states that produced them. And they grow linearly with the sequence length.

How big are the logits of GPT-OSS? One sequence, vocabulary 201,088, fp32. Hover over a column to inspect it.
  • Full logits, S × V
  • One 4,096-token chunk, C × V
0 GBFull logits at 4,096 tokens
3.3 GBOne chunk, at any context length
The context length doubles from 4K to 256K tokens, and so do the logits.
Show data
Context lengthFull logitsOne chunk
4,0963.3 GB3.3 GB
8,1926.6 GB3.3 GB
16,38413.2 GB3.3 GB
32,76826.4 GB3.3 GB
65,53652.7 GB3.3 GB
131,072105.4 GB3.3 GB
262,144210.9 GB3.3 GB
Logit memory is \(S \times V \times 4\) bytes. At 256K tokens a single copy is larger than the whole H200, and a standard loss keeps several. A chunk of \(C\) tokens costs the same at every context length.

That is a single copy. A straightforward distillation loss holds several: the student logits, their log-softmax, the teacher distribution expanded to the same shape, and, during the backward pass, the gradient with respect to the logits. Each of them is \(S \times V\).

For ordinary cross-entropy against hard labels, the standard fix is to fuse the output projection into the loss and chunk it, as Cut Cross-Entropy and Liger Kernel do. Our objective is different: a KL divergence against a sparse, cached top-K teacher. It was not available in the distillation tooling we use, such as ModelOpt, so we wrote our own.

What the loss actually needs

We distill against a cached teacher. For each position we store only the teacher's top \(K = 100\) tokens and their probabilities \(p_v\), for \(v\) in the support set \(\mathcal{S}\). They are not renormalized, so the retained mass \(M = \sum_{v \in \mathcal{S}} p_v\) is slightly below one. Writing the student's log-probabilities as \(\log q_v = z_v - \log Z\), the forward KL for one token splits into three pieces:

$$\mathcal{L}_{\mathrm{KL}} \;=\; \underbrace{\sum_{v \in \mathcal{S}} p_v \log p_v}_{H} \;-\; \underbrace{\sum_{v \in \mathcal{S}} p_v\, z_v}_{C} \;+\; M \log Z, \qquad \log Z = \log \sum_{v=1}^{V} e^{z_v}$$

Look at what each term touches. The teacher entropy \(H\), the cross term \(C\) and the mass \(M\) need only the \(K\) cached entries. Only the log-normalizer \(\log Z\) spans the whole vocabulary, and it reduces each row of logits to a single number.

Anatomy of one row of logits One token position, \(V\) vocabulary entries. Hover over a teacher entry.
  • Student logit \(z_v\)
  • Teacher top-K entry \(p_v\)
The sparse terms read 100 entries out of hundreds of thousands. The full row is needed only to compute \(\log Z\), a single scalar per token.

The gradient is just as compact. Differentiating with respect to a logit gives

$$\frac{\partial \mathcal{L}}{\partial z_v} \;=\; M\,\mathrm{softmax}(z)_v \;-\; p_v,$$

with \(p_v = 0\) outside the teacher's support. To rebuild it you need the logits of that row, \(\log Z\) and \(M\). Nothing else. So the rows can be produced, used and thrown away, as long as you are willing to produce them again in the backward pass.

The full \(S \times V\) tensor is never needed at once. Each row is needed briefly, and only to compute one scalar and one gradient.

The kernel

The fused kernel takes the hidden states and the output weights, not the logits. It splits the sequence into chunks of \(C\) tokens (4,096 by default) and visits them three times. Each visit recomputes that chunk's \(C \times V\) logit tile with one matrix multiply, uses it and frees it before moving on.

One training step through the output head Standard and fused paths on the same clock. Six chunks for readability; real runs use 4,096-token chunks.
Chunk 1 of 6
In the standard path, the logits, their log-softmax, a dense teacher and the logit gradient each take the full \(S \times V\) shape. In the fused kernel, only one \(C \times V\) tile is alive at a time, whatever the sequence length. Both memory meters share one scale, counted in tiles; it is illustrative.

The rest of this section walks through the kernel in code. The snippets are a simplified version of full_chunked.py: one GPU, a batch of one, and no temperature or logit scale. The comments mark where the real kernel adds tensor-parallel reductions. The whole thing is a single torch.autograd.Function in plain PyTorch, with no custom CUDA.

Pass 1: the normalizer

For each chunk, compute the tile \(z = h_{[s_0:s_1]} W^{\top}\), take the row maxima and the sum of shifted exponentials, and store \(\log Z\): one float per token. Then discard the tile. This is the usual max-shifted log-sum-exp, done one chunk at a time. When the vocabulary is sharded across tensor-parallel ranks, the max and the sum are all-reduced, and each rank only ever sees its own slice of \(V\).

Pass 2: the loss

Recompute the tile, gather the logits at the teacher's top-K token ids, and accumulate \(H\), \(C\) and \(M\). Combined with the stored \(\log Z\), they give the per-token loss. Here is the whole forward pass:

class FusedChunkedKL(torch.autograd.Function):

    @staticmethod
    def forward(ctx, hidden, W, t_idx, v_idx, p, C):
        # hidden: [S, d] final hidden states    W: [V, d] output head
        # (t_idx, v_idx, p): the sparse top-K teacher, one entry per
        # (position, token id, probability), sorted by position
        S = hidden.shape[0]
        log_z = hidden.new_empty(S, dtype=torch.float32)
        loss = torch.empty_like(log_z)
        mass = torch.empty_like(log_z)

        # We write the backward pass ourselves, so autograd records nothing.
        with torch.no_grad():

            # ---- Pass 1: the log-normalizer, one C x V tile at a time ----
            for s0 in range(0, S, C):
                s1 = min(s0 + C, S)
                z = (hidden[s0:s1] @ W.T).float()   # [C, V]: the only vocab-sized tensor
                m = z.amax(dim=-1)                  # row max, for a stable log-sum-exp
                # (tensor parallel: all_reduce(m, MAX), each rank sees V / tp columns)
                z.sub_(m[:, None]).exp_()           # shifted exponentials, in place
                total = z.sum(dim=-1)
                # (tensor parallel: all_reduce(total, SUM))
                log_z[s0:s1] = m + total.log()      # one float per token
                del z                               # the tile is gone before the next chunk

            # ---- Pass 2: the sparse terms of the loss ----
            for s0 in range(0, S, C):
                s1 = min(s0 + C, S)
                z = (hidden[s0:s1] @ W.T).float()   # recompute the very same tile
                # The teacher entries are sorted by position, so this chunk's
                # entries are one contiguous slice.
                lo = torch.searchsorted(t_idx, s0)
                hi = torch.searchsorted(t_idx, s1)
                t, v, pk = t_idx[lo:hi] - s0, v_idx[lo:hi], p[lo:hi]
                z_k = z[t, v]                       # gather K logits per row
                rows = torch.zeros(s1 - s0, device=z.device)
                H = rows.scatter_add(0, t, pk * pk.log())   # teacher entropy
                X = rows.scatter_add(0, t, pk * z_k)        # cross term
                M = rows.scatter_add(0, t, pk)              # retained mass
                loss[s0:s1] = H - X + M * log_z[s0:s1]
                mass[s0:s1] = M
                del z

        # Saved for backward: inputs plus two floats per token. No logits.
        ctx.save_for_backward(hidden, W, log_z, mass, t_idx, v_idx, p)
        ctx.C = C
        return loss                                 # per-token KL, [S]

A few details matter more than they look:

  • Everything runs under torch.no_grad(). Autograd would otherwise keep every tile alive for its own backward pass, which is exactly the memory we are trying to avoid. Because we provide the backward ourselves, nothing needs to be recorded.
  • The tile is cast to fp32 before the reductions. The matmul runs in bf16 on the tensor cores, but exponentials and sums over 200K entries need the extra precision. The fp32 tile is the largest temporary, at \(C \times V \times 4\) bytes.
  • The sparse terms use scatter_add. Each teacher entry adds its contribution to the row it belongs to, so a position can have any number of entries and positions without a teacher simply contribute zero.
  • What is saved for backward is tiny: the inputs, which autograd keeps anyway, plus \(\log Z\) and \(M\), two floats per token.

Backward: recompute, do not store

Recompute the tile once more and turn it into the gradient in place. Exponentiate \(z - \log Z\) to get the softmax, scale it by \(M\) and the upstream gradient, and subtract \(p\) at the K support positions. Two more matrix multiplies give this chunk's \(\partial h\) and add its contribution to \(\partial W\). Then the tile is freed.

    @staticmethod
    def backward(ctx, grad_loss):
        # grad_loss: [S], the upstream gradient of each token's loss
        hidden, W, log_z, mass, t_idx, v_idx, p = ctx.saved_tensors
        S, C = hidden.shape[0], ctx.C
        grad_hidden = torch.empty_like(hidden)
        grad_W = torch.zeros_like(W, dtype=torch.float32)   # accumulated in fp32

        for s0 in range(0, S, C):
            s1 = min(s0 + C, S)
            h = hidden[s0:s1]
            z = (h @ W.T).float()                   # third and last projection

            # dL/dz = M * softmax(z) - p, built in place on top of the logits.
            g = z.sub_(log_z[s0:s1, None]).exp_()   # softmax(z): log Z is already known
            g.mul_((mass[s0:s1] * grad_loss[s0:s1])[:, None])
            lo = torch.searchsorted(t_idx, s0)
            hi = torch.searchsorted(t_idx, s1)
            t, v = t_idx[lo:hi] - s0, v_idx[lo:hi]
            g.index_put_((t, v), -p[lo:hi] * grad_loss[t + s0],
                         accumulate=True)           # subtract p at the K support entries

            g = g.to(W.dtype)                       # back to bf16 for the tensor cores
            grad_hidden[s0:s1] = g @ W              # [C, V] x [V, d] -> [C, d]
            grad_W += (g.T @ h).float()             # [V, C] x [C, d] -> [V, d]
            del z, g                                # free the tile

        # (tensor parallel: all_reduce(grad_hidden), since each rank only
        #  contributed its own slice of the vocabulary)
        return grad_hidden, grad_W.to(W.dtype), None, None, None, None

Because \(\log Z\) was stored in the forward pass, the softmax needs no reduction at all here: subtracting it and exponentiating gives normalized probabilities directly. Every step writes into the same buffer, so the fp32 tile, its softmax and the gradient are one allocation. The sparse correction is a scatter of \(K\) values per row with accumulate=True, the only place the teacher enters the backward pass.

The trade is plain gradient checkpointing applied to the output head. The projection is computed three times instead of once: twice in the forward pass, once more in the backward pass. In exchange, vocabulary-sized memory is bounded by \(C \times V\), and the only thing that still grows with sequence length is the hidden state, \(S \times d\), with \(d \ll V\). Loss values and hidden-state gradients match the dense reference within \(10^{-4}\).

At 32K tokens: a real training run

The first test is the one that motivated the work: distilling Llama 3.1 8B Instruct into a 3.2B student on a single H200 with a 32,768-token context. The figure follows GPU memory through one training iteration, component by component.

GPU memory through one training iteration at 32K context Llama 3.1 8B Instruct → 3.2B student, single H200. Hover over a chart to inspect any moment.
  • Loss and logits
  • Activations
  • Gradients
  • Optimizer states
  • Model weights

Dense KL

Fused chunked KL

Show data
At the peak, approximateDense KLFused chunked KL
Model weights18 GB18 GB
Optimizer states30 GB30 GB
Activations77 GB77 GB
Loss and logits125 GB3 GB
Total≈250 GB≈128 GB
Everything except the loss is identical in the two runs. The vocabulary-sized tensors arrive exactly when activations peak, at the turn between the forward and backward passes, which is why they decide whether the run fits.

With the dense loss, the logit and teacher tensors push the peak to roughly 250 GB, far beyond the H200's 141 GB. The fused loss never forms them and peaks at about 128 GB, so the run fits on one GPU.

The same effect shows up at a larger scale. Distilling GPT-OSS-20B at 32,768 tokens previously needed four 8×H200 nodes. With the fused loss it fits on one, with less model parallelism and no inter-node communication:

GPT-OSS-20B distillation at 32K context 8×H200 nodes. Before: TP 4, PP 4, EP 2. After: TP 2, PP 1, EP 4.
Nodes414× fewer
Step time57.0 s12.23 s4.7× faster
Throughput per GPU74.2 TFLOP/s345.7 TFLOP/s4.7× higher

Isolating the kernel: 32K and 256K

End-to-end runs mix the loss with everything else on the GPU. To isolate it, we benchmark the output projection and the loss alone, with no transformer body: hidden size 4,096, a 131,072-token vocabulary, top-K = 100, batch size 1 and 4,096-token chunks, split across two H200s with tensor parallelism. Besides the dense baseline, we compare against a forward-chunked loss. It keeps the teacher sparse and computes the loss in chunks, but takes the logits from the regular output layer, so autograd still stores all of them for the backward pass.

Loss-only benchmark Per GPU, 2 × H200, vocabulary 131,072, chunk size 4,096
  • Dense KL
  • Forward-chunked KL
  • Fused chunked KL

Peak memory · GiB, lower is better

Time per iteration · lower is better

Show data
Loss32K memory32K time256K memory256K time
Dense KL85.2 GiB218 msOut of memory–
Forward-chunked KL17.7 GiB183 ms134.2 GiB5.26 s
Fused chunked KL5.45 GiB199 ms11.6 GiB1.59 s
Dense KL already runs out of memory at 64K tokens. The dashed line marks the 139.7 GiB of memory available on each H200.

At 32K tokens, the fused loss uses 5.45 GiB, against 85.2 GiB for the dense loss: 15.6× less. The forward-chunked loss is the fastest here, by about 8%, since it computes the projection only once and its logits still fit comfortably.

At 256K tokens, that changes. The forward-chunked loss needs 134.2 GiB per GPU, nearly the whole H200, just to keep its logits for the backward pass. The fused loss peaks at 11.6 GiB and is 3.3× faster, even though it computes the projection three times. Its throughput stays near 266 TFLOP/s at both lengths, while the forward-chunked loss falls from 288 TFLOP/s at 32K to 80 TFLOP/s at 256K. Past a certain size, storing and reading the logits costs more than computing them again.

Choosing the chunk size

The chunk size is the one knob. Larger chunks feed bigger matrix multiplies to the GPU but keep a larger tile alive. The floor under the memory curve is everything else: hidden states, weights and their gradients.

Fused chunked KL across chunk sizes Same loss-only benchmark, per GPU

Peak memory · GiB

Time per iteration

Show data
Chunk size32K memory32K time256K memory256K time
5124.13 GiB430 ms10.3 GiB3.47 s
1,0244.32 GiB298 ms10.5 GiB2.38 s
2,0484.70 GiB231 ms10.8 GiB1.84 s
4,0965.45 GiB199 ms11.6 GiB1.59 s
8,1968.46 GiB182 ms14.6 GiB1.45 s
Going from 4,096 to 512 tokens per chunk saves 1.3 GiB at 256K but more than doubles the iteration time. We use 4,096 everywhere else in this post.

When to use it

  • Long context is where it pays off. Up to about 32K tokens, a forward-chunked loss can be slightly faster if its logits fit in memory. Beyond that, the fused loss wins on both memory and speed.
  • The 256K numbers isolate the loss. They come from a benchmark without attention, optimizer state or data loading, and are not end-to-end training throughput.
  • We have tested it on H200 GPUs with Megatron-Bridge and ModelOpt. Other hardware and frameworks are untested.

Use it

Using the kernel requires one change to the model: the output head must not run during training. The loss needs the final hidden states and the head's weight, and does the projection itself. A sketch of the change to a model's forward pass:

class StudentLM(nn.Module):

    def forward(self, input_ids, sparse_teacher=None):
        # Everything up to and including the final norm is unchanged.
        hidden = self.final_norm(self.decoder(self.embed(input_ids)))   # [S, d]

        if sparse_teacher is None:
            # Inference and evaluation keep the usual head.
            return self.lm_head(hidden)                                 # [S, V]

        # Training: do not call the head. Hand its weight to the loss, which
        # projects one chunk at a time and returns the per-token KL.
        W = self.lm_head.weight          # or the embedding, if the two are tied
        return full_chunked_loss(
            hidden, W, sparse_teacher,
            seq_chunk_size=4096,
            tp_group=self.tp_group,      # if the vocabulary is sharded
            output_scale=self.logit_scale,   # anything applied to the logits
            temperature=1.0,                 # after the matmul moves in here
        )

Three things to check when you wire it in. The hidden states must be the ones the head would see, after the final norm. With tied embeddings, pass the embedding matrix as \(W\). And anything applied to the logits after the matmul, such as a logit scale or a distillation temperature, has to move into the loss through output_scale and temperature, because the logits no longer leave it. In a tensor-parallel model such as Megatron, pass the local shard of the head, shaped [V / tp, d], together with the tensor-parallel group, and set sequence_parallel=True if the hidden states are sharded along the sequence. The kernel handles the all-gather and the reductions.

The repository includes the dense and forward-chunked baselines, CPU equivalence tests, and the benchmark behind every number in this post. For offline teacher caching and the distilled model itself, see the paper and our earlier post on Hugging Face.

@misc{ryskulov2026efficientknowledgedistillationllms,
  title         = {Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss},
  author        = {Bakbergen Ryskulov and Iker García-Ferrero and David Montero and David Jansen and Ali Hashemi and Jezabel R. Garcia and Antonio Tiene and Román Orús},
  year          = {2026},
  eprint        = {2608.03796},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.03796}
}