Uni-X: Resolving Modality Conflict with Separate Ends and a Shared Middle

我们关于统一多模态生成与理解模型（Unified Multimodal Models, UMMs）架构探索的论文 Uni-X 被 ICLR 2026 接收。在该工作中，我们在最简洁的自回归（Autoregressive, AR）范式下，尝试给“文本”和“视觉”模态之间的互相冲突问题一个优雅的解决方案。简单来说，我们在模型的浅层和深层设置了不同分支来专门处理文本或者视觉模态。

📄 论文 https://openreview.net/forum?id=IJLIYpCkwz
💻 代码 https://github.com/CURRENTF/Uni-X
🤗 权重 https://huggingface.co/JitaiHao/Uni-X-3B

Motivation：模态间梯度冲突

将视觉输入通过向量量化（Vector Quantization）转化为离散 Token，从而将文本和视觉统一为单一序列进行预测，是目前构建 AR UMMs 的主流思路。然而，通过实验分析，我们发现：使用一个完全共享参数的标准 Transformer 在处理这种多模态输入时，会遭遇严重的“梯度冲突”（Gradient Conflict）。

标准 Transformer 的 FFN 下采样权重梯度冲突分析。浅层和深层的梯度冲突极为剧烈，而在中间层有所缓解。应用Uni-X结构后，梯度冲突得到了进一步地缓解。

为了精确描述这一现象，我们定义了层级的梯度冲突指标 $c_{g}$ 。通过计算纯文本数据梯度 $g_{text}$ 与图文对数据梯度 $g_{img}$ 之间的余弦相似度 $S_{inter}$ ，并减去模型在混合模态数据分布下的基线梯度相似度 $S_{base}$ ，我们得到：

$c_{g}=-(S_{inter}-S_{base})$

冲突来源：底层表示的信息熵差异

为什么会出现这种现象？我们尝试从信息论角度提供一个解释：VQ得到的视觉序列与于自然语言的差异极大。我们计算了基于 N-gram 的条件熵，结果表明，视觉 Token 序列的条件熵显著高于英语、德语或中文等自然语言。这种极高的信息熵意味着视觉序列的预测难度更大，需要模型建模更长程、空间纠缠更深的依赖关系。

当单一网络强制同时处理低熵的语法化文本和高熵的空间化视觉时，模型的浅层（负责底层特征提取）和深层（负责目标分布预测）会被迫调和截然不同的底层统计特性，从而引发强烈的梯度拉扯。相反，在网络的中间层，特征表征逐渐抽象化和语义化，模态间的对齐变得更加自然，冲突显著减小。

2. Uni-X (Two-End-Separated, Middle-Shared)

基于上述底层逻辑，我们不再一味地引入外部视觉语义编码器或复杂的任务分支结构，而是让模型架构的物理设计去贴合不同模态的固有特性。我们提出了 Uni-X，一种“两端分离、中间共享”的 X 型架构：

Uni-X 架构与标准共享 Transformer 的对比。X 型结构通过两端分离避免了极端的梯度冲突，同时依靠共享核心实现了特征对齐。

分离层（Separated Layers）： 将 Transformer 的初始 $N$ 层和最后 $M$ 层拆分为平行的模态专用分支。这保证了在早期特征提取和最终 Token 投射阶段，文本和视觉信息能够被独立处理，彻底隔绝底层分布差异带来的优化冲突。
共享层（Shared Layers）： 中间层保持参数共享，专注于高维语义的跨模态融合与推理。

和 Encoder/Decoder 的关系

我们的模型设计一定程度上也受到了 Encoder/Decoder 这种经典架构的启发，我们希望分离层能分别起到 Encoder和Decoder 的作用，由于时间和资源限制我们没有进行相关分析实验。一个有趣的探索是，直接拿出浅分离层作为双塔模型，能不能有一个还不错的检索性能？

理论计算效率的额外增益

除了优化上的优势，由于分离层中视觉和文本处于严格隔离的计算流，对于长度为 $n$ 的序列（其中视觉 Token 长 $a$ ，文本 Token 长 $b$ ），自注意力机制的计算复杂度从全共享的 $O((a+b)^{2})$ 下降到了与 $a^{2}+b^{2}$ 成正比。这意味着在同等参数规模下，Uni-X 的训练和推理吞吐量具有更高的理论上限。

3. 实验结果

我们在一致训练预算下进行实验对比，Uni-X 的 3B 参数版本展现出了极强的 Scaling 能力与任务竞争力：

图像生成与理解： 在未引入额外语义 Encoder 的前提下，模型在 GenEval 基准测试中达到了 82 分的优异成绩，匹配甚至超越了一些 7B 规模的自回归 UMMs 。
图像编辑（Zero-Shot 泛化）： 在仅使用约 90k 图像编辑数据微调的情况下，Uni-X 凭借中间共享层积累的强大语义对齐能力，在 ImgEdit 榜单上的综合表现与使用了更多数据和更大参数量的 Bagel 相当。

Future Work

不依赖外部视觉特征提取器（CLIP）确实在一定程度上限制了Uni-X的多模态理解性能。但我们计划进一步探索：未来能否进一步移除 VQ-VAE 这一用来 tokenization 的中间件？如果让 Uni-X 架构中“X”的分叉部分直接承担起 Tokenizer 与 Detokenizer 的映射功能，我们或许能实现真正意义上的 Pixel-to-Pixel、端到端的原生多模态统一。

Our paper Uni-X, which explores architectures for unified multimodal models (UMMs) that support both generation and understanding, was accepted to ICLR 2026. Within a simple autoregressive (AR) framework, we aim to resolve the conflict between text and vision. In brief, we introduce separate branches for the two modalities in the shallow and deep layers of the model.

📄 Paper https://openreview.net/forum?id=IJLIYpCkwz
💻 Code https://github.com/CURRENTF/Uni-X
🤗 Weights https://huggingface.co/JitaiHao/Uni-X-3B

Motivation: gradient conflict across modalities

A common approach to building AR UMMs is to convert visual inputs into discrete tokens through vector quantization, then model text and vision as a single sequence. Our experiments show, however, that a standard Transformer with fully shared parameters encounters severe gradient conflict when processing these multimodal inputs.

Gradient conflict in the FFN down-projection weights of a standard Transformer. Conflict is particularly severe in the shallow and deep layers and weaker in the middle layers. The Uni-X architecture reduces it further.

To characterize this phenomenon, we define a layer-wise gradient conflict metric, $c_{g}$ . We compute the cosine similarity $S_{inter}$ between gradients from text-only data, $g_{text}$ , and image–text pairs, $g_{img}$ , then subtract the baseline gradient similarity under the mixed-modality data distribution, $S_{base}$ , giving:

$c_{g}=-(S_{inter}-S_{base})$

Where conflict comes from: entropy differences in low-level representations

Why does this happen? We explore an explanation from information theory: visual sequences produced by vector quantization differ substantially from natural language. Our N-gram conditional entropy analysis shows that visual token sequences have significantly higher conditional entropy than English, German, or Chinese text. This high entropy makes visual sequences harder to predict, requiring the model to capture longer-range and more spatially entangled dependencies.

Visual token sequences have substantially higher conditional entropy than natural languages such as English, German, and Chinese.

When a single network must process both low-entropy, grammatically structured text and high-entropy, spatially structured visual inputs, its shallow layers, which extract low-level features, and its deep layers, which predict the target distribution, must reconcile very different statistics. This produces strong conflict between their gradients. In the middle layers, representations become more abstract and semantic, making cross-modal alignment more natural and reducing conflict.

2. Uni-X: two-end-separated, middle-shared

Instead of relying solely on external visual semantic encoders or complex task-specific branches, we align the architecture with the inherent properties of the modalities. We propose Uni-X, an X-shaped architecture with separate ends and a shared middle:

Uni-X compared with a standard shared Transformer. Separating the two ends avoids severe gradient conflict, while the shared core supports feature alignment.

Separated layers: the first $N$ and last $M$ Transformer layers are split into parallel, modality-specific branches. Text and visual tokens are processed independently during early feature extraction and final token projection, isolating the optimization conflicts caused by their different low-level distributions.
Shared layers: the middle layers retain shared parameters and focus on cross-modal fusion and reasoning over high-level semantic features.

Connections to encoder–decoder architectures

Our design is also partly inspired by the classic encoder–decoder architecture. We hope the separated layers can serve as an encoder and decoder, respectively, although time and resource constraints prevented us from conducting dedicated analyses. One interesting question is whether the shallow separated branches could be used directly as a dual-encoder model with reasonable retrieval performance.

An additional benefit in theoretical compute efficiency

Beyond the optimization benefits, visual and textual tokens follow separate computation paths in the separated layers. For a sequence of length $n$ , containing $a$ visual tokens and $b$ text tokens, self-attention complexity drops from the fully shared $O((a+b)^{2})$ to a quantity proportional to $a^{2}+b^{2}$ . This gives Uni-X a higher theoretical ceiling for training and inference throughput at the same parameter scale.

3. Experimental results

Under matched training budgets, the 3B-parameter version of Uni-X demonstrates strong scaling behavior and competitive performance:

Image generation and understanding: without an additional semantic encoder, the model achieves a score of 82 on GenEval, matching or exceeding some 7B autoregressive UMMs.
Image editing and zero-shot generalization: after fine-tuning on only about 90K image-editing examples, Uni-X achieves overall ImgEdit leaderboard performance comparable to Bagel, which uses more data and a larger model. The shared middle layers provide the semantic alignment that supports this capability.

Future work

Not relying on an external visual feature extractor such as CLIP does limit Uni-X’s multimodal understanding performance to some extent. We plan to explore whether VQ-VAE, the intermediate tokenization component, can also be removed. If the branches of the “X” directly perform tokenization and detokenization, we may be able to achieve truly end-to-end, pixel-to-pixel unified multimodal modeling.