Our work on efficient distillation, Low-Rank Clone (LRC), was selected as a Spotlight at NeurIPS 2025.

TL;DR: We train a set of low-rank projection matrices that directly project a strong teacher LLM into a smaller student language model. Instead of creating or training the student’s large weight matrices from scratch, we train only these projections. With just 20B training tokens, our LRC-1.7B outperforms Qwen3-1.7B, a state-of-the-art model trained on 36T tokens, yielding a more than 1,000-fold improvement in training-token efficiency.

Paper: arxiv.org/abs/2505.12781

Code and weights: https://github.com/CURRENTF/LowRankClone https://huggingface.co/JitaiHao/LRC-4B-SFT

1. The initial insight: if an FFN is a key-value memory, can we compress it?

Before introducing the method, I would like to share the thinking that led to it, in the hope that it offers a useful perspective.

LRC takes direct inspiration from the classic paper “Transformer Feed-Forward Layers Are Key-Value Memories” (Geva et al., 2021), which interprets the FFN module as a key-value memory.

Let us revisit the FFN computation, using SwiGLU as an example:

$$o_{ffn} = \text{SwiGLU}(xW_{up}^{\top}, xW_{gate}^{\top})W_{down}$$

Intuitively, this computation has two stages:

1. Querying and matching: the input vector x is multiplied by W_up and W_gate, which can be viewed as key matrices, to compute similarities and produce an activation distribution.
2. Retrieving content: those activations weight the rows of W_down, which can be viewed as a value matrix, to retrieve the stored knowledge.

If an FFN is a key-value memory, a natural question follows: can we compress its key and value matrices without substantially damaging the knowledge they store?

That question is the starting point for LRC.

2. The LRC method

To compress FFN memory, we designed a unified framework with two core components: low-rank projection and activation clone.

1. Low-rank projection: generating student weights through projection

Conventional methods either train a student from scratch or discard parts of the teacher’s weights through hard pruning. These approaches can be inefficient or lose valuable information.

LRC takes a different approach. We train lightweight projection matrices $W_m^P$, rather than the student’s weights. The student weights $W_m^S$ are generated directly by projecting the teacher weights $W_m^T$.

For each weight matrix m, such as q, k, v, o, up, gate, down, in each Transformer layer i, the student weights are generated as follows:

$$W_{m,i}^S = W_{m,i}^T W_{m,i}^P$$

Here, $W_{m,i}^T \in \mathbb{R}^{d_{m}^{T}\times d^{T}}$ denotes the teacher weights, and $W_{m,i}^P \in \mathbb{R}^{d^{T}\times d^{S}}$ is the low-rank projection matrix we train. This softly compresses the teacher’s large weight matrices into the student’s dimensional space while preserving as much of the original knowledge structure as possible.

2. Activation clone: preserving similarity after projection

Projecting the weights alone is not enough. We also need the compressed key-value memory to respond similarly to the teacher when given the same input.

We therefore introduce activation clone, using a mean squared error (MSE) loss to align the student’s intermediate activations with the teacher’s.

Our clone loss $\mathcal{L}_{clone}$ covers multiple intermediate outputs in both the attention and FFN modules:

$$\mathcal{L}_{clone} = \sum_{i}^{l} \left[ \mathcal{E}(o_{attn,i}^S, o_{attn,i}^T W_{o,i}^P) + \mathcal{E}(o_{ffn,i}^S, o_{ffn,i}^T W_{down,i}^P) + \sum_{m \in \{q,k,v,up,gate\}} \mathcal{E}(h_{m,i}^S, h_{m,i}^T) \right]$$

Here, $\mathcal{E}$ denotes the MSE loss. This objective encourages the student to imitate the teacher not only at the final output layer, but also throughout its intermediate computation.

The complete training objective combines three terms: the conventional KL divergence loss $\mathcal{L}_{KL}$, the language modeling loss $\mathcal{L}_{LM}$, and our clone loss $\mathcal{L}_{clone}$.

3. Experiments

We evaluated LRC by training a series of student models using open-source teachers such as Llama-3.2-3B and Qwen2.5-3B/7B.

As shown in the table above, LRC-1.7B achieves an average score of 64.98 with only 20B training tokens, exceeding Qwen3-1.7B (63.17), which was trained on 36T tokens. Meanwhile, LRC-4B reaches 70.30 with approximately 18B training tokens, matching Qwen3-4B (70.29), also trained on 36T tokens. All models were evaluated with lm-eval==0.4.8, as clarified in the Xiaohongshu update of November 30, 2025.

The student inherits the teacher’s knowledge structure

To test our central hypothesis, we conducted a neuron-masking experiment. We found that neurons strongly associated with factual knowledge in the teacher play similar roles at the same positions in the student. Masking those neurons causes a substantial performance drop in both models, providing strong evidence that the student inherits the teacher’s knowledge structure.

If you are interested in the work, please read the paper and consider starring the GitHub repository. We will continue improving the code to make LRC easier to use!