Blog · Kernels
Optimizing KL loss: how to fit long context distillation into your VRAM
At long context, the largest tensor in knowledge distillation is not in the model. It is the student's vocabulary-sized logits. This post explains a kernel that computes the exact same loss and gradients without ever storing them.
To distill a large language model into a smaller one, we typically use a training method that compares the teacher's (large model) and the student's (scaller model) next-token distributions at every position of every sequence. The comparisson is done, by applying the Kullback-Leibler divergence formula to the two distributions. Each distribution spans the whole vocabulary, which is 128K to 256K entries in current models. At long context, the tensors holding them become the largest objects on the GPU, and the first thing that runs out of memory.
We hit this wall when we tried to distill models at 32K tokens and beyond. The fix turned out to be a loss kernel that computes the forward KL divergence and its gradients in sequence chunks, fused with the model's output projection, so that the full logit tensor never exists. This post walks through how the kernel works and what it buys.
This is one part of our paper, Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss, written with Bakbergen Ryskulov, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene and Román Orús. The paper also covers offline teacher caching and the resulting compact model. Here I focus only on the kernel. The implementation is open source.
The logit wall
A language model ends with a linear head. It takes the final hidden states \(h \in \mathbb{R}^{S \times d}\) for a sequence of \(S\) tokens and projects them through the output weights \(W \in \mathbb{R}^{V \times d}\) to get one logit per vocabulary entry:
$$z = h\,W^{\top} \in \mathbb{R}^{S \times V}$$The hidden size \(d\) is a few thousand. The vocabulary \(V\) is two orders of magnitude larger. For GPT-OSS, \(d = 2{,}880\) and \(V = 201{,}088\), so the logits are about 70 times larger than the hidden states that produced them. And they grow linearly with the sequence length.
- Full logits, S × V
- One 4,096-token chunk, C × V
Show data
| Context length | Full logits | One chunk |
|---|---|---|
| 4,096 | 3.3 GB | 3.3 GB |
| 8,192 | 6.6 GB | 3.3 GB |
| 16,384 | 13.2 GB | 3.3 GB |
| 32,768 | 26.4 GB | 3.3 GB |
| 65,536 | 52.7 GB | 3.3 GB |
| 131,072 | 105.4 GB | 3.3 GB |
| 262,144 | 210.9 GB | 3.3 GB |
That is a single copy. A straightforward distillation loss holds several: the student logits, their log-softmax, the teacher distribution expanded to the same shape, and, during the backward pass, the gradient with respect to the logits. Each of them is \(S \times V\).
For ordinary cross-entropy against hard labels, the standard fix is to fuse the output projection into the loss and chunk it, as Cut Cross-Entropy and Liger Kernel do. Our objective is different: a KL divergence against a sparse, cached top-K teacher. It was not available in the distillation tooling we use, such as ModelOpt, so we wrote our own.
What the loss actually needs
We distill against a cached teacher. For each position we store only the teacher's top \(K = 100\) tokens and their probabilities \(p_v\), for \(v\) in the support set \(\mathcal{S}\). They are not renormalized, so the retained mass \(M = \sum_{v \in \mathcal{S}} p_v\) is slightly below one. Writing the student's log-probabilities as \(\log q_v = z_v - \log Z\), the forward KL for one token splits into three pieces:
$$\mathcal{L}_{\mathrm{KL}} \;=\; \underbrace{\sum_{v \in \mathcal{S}} p_v \log p_v}_{H} \;-\; \underbrace{\sum_{v \in \mathcal{S}} p_v\, z_v}_{C} \;+\; M \log Z, \qquad \log Z = \log \sum_{v=1}^{V} e^{z_v}$$Look at what each term touches. The teacher entropy \(H\), the cross term \(C\) and the mass \(M\) need only the \(K\) cached entries. Only the log-normalizer \(\log Z\) spans the whole vocabulary, and it reduces each row of logits to a single number.
- Student logit \(z_v\)
- Teacher top-K entry \(p_v\)
The gradient is just as compact. Differentiating with respect to a logit gives
$$\frac{\partial \mathcal{L}}{\partial z_v} \;=\; M\,\mathrm{softmax}(z)_v \;-\; p_v,$$with \(p_v = 0\) outside the teacher's support. To rebuild it you need the logits of that row, \(\log Z\) and \(M\). Nothing else. So the rows can be produced, used and thrown away, as long as you are willing to produce them again in the backward pass.
The full \(S \times V\) tensor is never needed at once. Each row is needed briefly, and only to compute one scalar and one gradient.
The kernel
The fused kernel takes the hidden states and the output weights, not the logits. It splits the sequence into chunks of \(C\) tokens (4,096 by default) and visits them three times. Each visit recomputes that chunk's \(C \times V\) logit tile with one matrix multiply, uses it and frees it before moving on.
Standard dense
Fused chunked
The rest of this section walks through the kernel in code. The snippets are a simplified version of full_chunked.py: one GPU, a batch of one, and no temperature or logit scale. The comments mark where the real kernel adds tensor-parallel reductions. The whole thing is a single torch.autograd.Function in plain PyTorch, with no custom CUDA.
Pass 1: the normalizer
For each chunk, compute the tile \(z = h_{[s_0:s_1]} W^{\top}\), take the row maxima and the sum of shifted exponentials, and store \(\log Z\): one float per token. Then discard the tile. This is the usual max-shifted log-sum-exp, done one chunk at a time. When the vocabulary is sharded across tensor-parallel ranks, the max and the sum are all-reduced, and each rank only ever sees its own slice of \(V\).
Pass 2: the loss
Recompute the tile, gather the logits at the teacher's top-K token ids, and accumulate \(H\), \(C\) and \(M\). Combined with the stored \(\log Z\), they give the per-token loss. Here is the whole forward pass:
class FusedChunkedKL(torch.autograd.Function):
@staticmethod
def forward(ctx, hidden, W, t_idx, v_idx, p, C):
# hidden: [S, d] final hidden states W: [V, d] output head
# (t_idx, v_idx, p): the sparse top-K teacher, one entry per
# (position, token id, probability), sorted by position
S = hidden.shape[0]
log_z = hidden.new_empty(S, dtype=torch.float32)
loss = torch.empty_like(log_z)
mass = torch.empty_like(log_z)
# We write the backward pass ourselves, so autograd records nothing.
with torch.no_grad():
# ---- Pass 1: the log-normalizer, one C x V tile at a time ----
for s0 in range(0, S, C):
s1 = min(s0 + C, S)
z = (hidden[s0:s1] @ W.T).float() # [C, V]: the only vocab-sized tensor
m = z.amax(dim=-1) # row max, for a stable log-sum-exp
# (tensor parallel: all_reduce(m, MAX), each rank sees V / tp columns)
z.sub_(m[:, None]).exp_() # shifted exponentials, in place
total = z.sum(dim=-1)
# (tensor parallel: all_reduce(total, SUM))
log_z[s0:s1] = m + total.log() # one float per token
del z # the tile is gone before the next chunk
# ---- Pass 2: the sparse terms of the loss ----
for s0 in range(0, S, C):
s1 = min(s0 + C, S)
z = (hidden[s0:s1] @ W.T).float() # recompute the very same tile
# The teacher entries are sorted by position, so this chunk's
# entries are one contiguous slice.
lo = torch.searchsorted(t_idx, s0)
hi = torch.searchsorted(t_idx, s1)
t, v, pk = t_idx[lo:hi] - s0, v_idx[lo:hi], p[lo:hi]
z_k = z[t, v] # gather K logits per row
rows = torch.zeros(s1 - s0, device=z.device)
H = rows.scatter_add(0, t, pk * pk.log()) # teacher entropy
X = rows.scatter_add(0, t, pk * z_k) # cross term
M = rows.scatter_add(0, t, pk) # retained mass
loss[s0:s1] = H - X + M * log_z[s0:s1]
mass[s0:s1] = M
del z
# Saved for backward: inputs plus two floats per token. No logits.
ctx.save_for_backward(hidden, W, log_z, mass, t_idx, v_idx, p)
ctx.C = C
return loss # per-token KL, [S]
A few details matter more than they look:
- Everything runs under
torch.no_grad(). Autograd would otherwise keep every tile alive for its own backward pass, which is exactly the memory we are trying to avoid. Because we provide the backward ourselves, nothing needs to be recorded. - The tile is cast to fp32 before the reductions. The matmul runs in bf16 on the tensor cores, but exponentials and sums over 200K entries need the extra precision. The fp32 tile is the largest temporary, at \(C \times V \times 4\) bytes.
- The sparse terms use
scatter_add. Each teacher entry adds its contribution to the row it belongs to, so a position can have any number of entries and positions without a teacher simply contribute zero. - What is saved for backward is tiny: the inputs, which autograd keeps anyway, plus \(\log Z\) and \(M\), two floats per token.
Backward: recompute, do not store
Recompute the tile once more and turn it into the gradient in place. Exponentiate \(z - \log Z\) to get the softmax, scale it by \(M\) and the upstream gradient, and subtract \(p\) at the K support positions. Two more matrix multiplies give this chunk's \(\partial h\) and add its contribution to \(\partial W\). Then the tile is freed.
@staticmethod
def backward(ctx, grad_loss):
# grad_loss: [S], the upstream gradient of each token's loss
hidden, W, log_z, mass, t_idx, v_idx, p = ctx.saved_tensors
S, C = hidden.shape[0], ctx.C
grad_hidden = torch.empty_like(hidden)
grad_W = torch.zeros_like(W, dtype=torch.float32) # accumulated in fp32
for s0 in range(0, S, C):
s1 = min(s0 + C, S)
h = hidden[s0:s1]
z = (h @ W.T).float() # third and last projection
# dL/dz = M * softmax(z) - p, built in place on top of the logits.
g = z.sub_(log_z[s0:s1, None]).exp_() # softmax(z): log Z is already known
g.mul_((mass[s0:s1] * grad_loss[s0:s1])[:, None])
lo = torch.searchsorted(t_idx, s0)
hi = torch.searchsorted(t_idx, s1)
t, v = t_idx[lo:hi] - s0, v_idx[lo:hi]
g.index_put_((t, v), -p[lo:hi] * grad_loss[t + s0],
accumulate=True) # subtract p at the K support entries
g = g.to(W.dtype) # back to bf16 for the tensor cores
grad_hidden[s0:s1] = g @ W # [C, V] x [V, d] -> [C, d]
grad_W += (g.T @ h).float() # [V, C] x [C, d] -> [V, d]
del z, g # free the tile
# (tensor parallel: all_reduce(grad_hidden), since each rank only
# contributed its own slice of the vocabulary)
return grad_hidden, grad_W.to(W.dtype), None, None, None, None
Because \(\log Z\) was stored in the forward pass, the softmax needs no reduction at all here: subtracting it and exponentiating gives normalized probabilities directly. Every step writes into the same buffer, so the fp32 tile, its softmax and the gradient are one allocation. The sparse correction is a scatter of \(K\) values per row with accumulate=True, the only place the teacher enters the backward pass.
The trade is plain gradient checkpointing applied to the output head. The projection is computed three times instead of once: twice in the forward pass, once more in the backward pass. In exchange, vocabulary-sized memory is bounded by \(C \times V\), and the only thing that still grows with sequence length is the hidden state, \(S \times d\), with \(d \ll V\). Loss values and hidden-state gradients match the dense reference within \(10^{-4}\).
At 32K tokens: a real training run
The first test is the one that motivated the work: distilling Llama 3.1 8B Instruct into a 3.2B student on a single H200 with a 32,768-token context. The figure follows GPU memory through one training iteration, component by component.
- Loss and logits
- Activations
- Gradients
- Optimizer states
- Model weights
Dense KL
Fused chunked KL
Show data
| At the peak, approximate | Dense KL | Fused chunked KL |
|---|---|---|
| Model weights | 18 GB | 18 GB |
| Optimizer states | 30 GB | 30 GB |
| Activations | 77 GB | 77 GB |
| Loss and logits | 125 GB | 3 GB |
| Total | ≈250 GB | ≈128 GB |
With the dense loss, the logit and teacher tensors push the peak to roughly 250 GB, far beyond the H200's 141 GB. The fused loss never forms them and peaks at about 128 GB, so the run fits on one GPU.
The same effect shows up at a larger scale. Distilling GPT-OSS-20B at 32,768 tokens previously needed four 8×H200 nodes. With the fused loss it fits on one, with less model parallelism and no inter-node communication:
Isolating the kernel: 32K and 256K
End-to-end runs mix the loss with everything else on the GPU. To isolate it, we benchmark the output projection and the loss alone, with no transformer body: hidden size 4,096, a 131,072-token vocabulary, top-K = 100, batch size 1 and 4,096-token chunks, split across two H200s with tensor parallelism. Besides the dense baseline, we compare against a forward-chunked loss. It keeps the teacher sparse and computes the loss in chunks, but takes the logits from the regular output layer, so autograd still stores all of them for the backward pass.
- Dense KL
- Forward-chunked KL
- Fused chunked KL
Peak memory · GiB, lower is better
Time per iteration · lower is better
Show data
| Loss | 32K memory | 32K time | 256K memory | 256K time |
|---|---|---|---|---|
| Dense KL | 85.2 GiB | 218 ms | Out of memory | – |
| Forward-chunked KL | 17.7 GiB | 183 ms | 134.2 GiB | 5.26 s |
| Fused chunked KL | 5.45 GiB | 199 ms | 11.6 GiB | 1.59 s |
At 32K tokens, the fused loss uses 5.45 GiB, against 85.2 GiB for the dense loss: 15.6× less. The forward-chunked loss is the fastest here, by about 8%, since it computes the projection only once and its logits still fit comfortably.
At 256K tokens, that changes. The forward-chunked loss needs 134.2 GiB per GPU, nearly the whole H200, just to keep its logits for the backward pass. The fused loss peaks at 11.6 GiB and is 3.3× faster, even though it computes the projection three times. Its throughput stays near 266 TFLOP/s at both lengths, while the forward-chunked loss falls from 288 TFLOP/s at 32K to 80 TFLOP/s at 256K. Past a certain size, storing and reading the logits costs more than computing them again.
Choosing the chunk size
The chunk size is the one knob. Larger chunks feed bigger matrix multiplies to the GPU but keep a larger tile alive. The floor under the memory curve is everything else: hidden states, weights and their gradients.
Peak memory · GiB
Time per iteration
Show data
| Chunk size | 32K memory | 32K time | 256K memory | 256K time |
|---|---|---|---|---|
| 512 | 4.13 GiB | 430 ms | 10.3 GiB | 3.47 s |
| 1,024 | 4.32 GiB | 298 ms | 10.5 GiB | 2.38 s |
| 2,048 | 4.70 GiB | 231 ms | 10.8 GiB | 1.84 s |
| 4,096 | 5.45 GiB | 199 ms | 11.6 GiB | 1.59 s |
| 8,196 | 8.46 GiB | 182 ms | 14.6 GiB | 1.45 s |
When to use it
- Long context is where it pays off. Up to about 32K tokens, a forward-chunked loss can be slightly faster if its logits fit in memory. Beyond that, the fused loss wins on both memory and speed.
- The 256K numbers isolate the loss. They come from a benchmark without attention, optimizer state or data loading, and are not end-to-end training throughput.
- We have tested it on H200 GPUs with Megatron-Bridge and ModelOpt. Other hardware and frameworks are untested.
Use it
Using the kernel requires one change to the model: the output head must not run during training. The loss needs the final hidden states and the head's weight, and does the projection itself. A sketch of the change to a model's forward pass:
class StudentLM(nn.Module):
def forward(self, input_ids, sparse_teacher=None):
# Everything up to and including the final norm is unchanged.
hidden = self.final_norm(self.decoder(self.embed(input_ids))) # [S, d]
if sparse_teacher is None:
# Inference and evaluation keep the usual head.
return self.lm_head(hidden) # [S, V]
# Training: do not call the head. Hand its weight to the loss, which
# projects one chunk at a time and returns the per-token KL.
W = self.lm_head.weight # or the embedding, if the two are tied
return full_chunked_loss(
hidden, W, sparse_teacher,
seq_chunk_size=4096,
tp_group=self.tp_group, # if the vocabulary is sharded
output_scale=self.logit_scale, # anything applied to the logits
temperature=1.0, # after the matmul moves in here
)
Three things to check when you wire it in. The hidden states must be the ones the head would see, after the final norm. With tied embeddings, pass the embedding matrix as \(W\). And anything applied to the logits after the matmul, such as a logit scale or a distillation temperature, has to move into the loss through output_scale and temperature, because the logits no longer leave it. In a tensor-parallel model such as Megatron, pass the local shard of the head, shaped [V / tp, d], together with the tensor-parallel group, and set sequence_parallel=True if the hidden states are sharded along the sequence. The kernel handles the all-gather and the reductions.
The repository includes the dense and forward-chunked baselines, CPU equivalence tests, and the benchmark behind every number in this post. For offline teacher caching and the distilled model itself, see the paper and our earlier post on Hugging Face.
@misc{ryskulov2026efficientknowledgedistillationllms,
title = {Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss},
author = {Bakbergen Ryskulov and Iker García-Ferrero and David Montero and David Jansen and Ali Hashemi and Jezabel R. Garcia and Antonio Tiene and Román Orús},
year = {2026},
eprint = {2608.03796},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2608.03796}
}