Unified Multimodal Reasoning: Can Different Modalities Share One Thought Space?

Read the original version →

The big idea in 30 seconds

Modern AI systems increasingly need to reason across language, images, 3D geometry, and robot state. Pixels do not look like words, and a point cloud does not look like a robot joint configuration. Yet they may all need to contribute to the same reasoning process.

Do different modalities need separate reasoning processes, or can their intermediate thoughts share one space for reasoning?

Uni-LaDiR explores the second possibility. Rather than forcing modalities into the same raw representation, it maps their task-relevant information into a shared latent thought space. The model learns to generate intermediate thoughts in this space and uses them to predict the answer or action. ^[1]

The argument follows three questions: what should be shared, what should a thought retain, and how can the model generate it? The experiments then test each design choice.

Read the paper →

1. AI speaks many “languages”

Modern multimodal systems work with diverse forms of information:

• Language: abstract concepts and relationships.
• Images and video: appearance, spatial context, and change over time.
• Audio: sounds and temporal cues.
• 3D geometry: explicit physical structure.
• Robot state and actions: an agent’s configuration and interaction with the world.

Each modality has its own structure and strengths. The challenge is how to reason across them.

An illustrative reasoning chain:

If each modality uses a completely separate reasoning representation, every transition becomes a modality-specific interface. As more modalities are introduced, this becomes increasingly complicated.

An alternative lets each modality preserve its input representation while mapping the information needed for reasoning into a common latent space.

This motivates unified latent reasoning. The list above describes the broader multimodal challenge; Uni-LaDiR’s experiments study text, images, 3D observations, and robot state. ^[1]

2. A shared space for reasoning

A shared latent space does not mean all modalities should contain the same information. It provides a common computational format in which different information can participate in reasoning.

• A 3D observation might contribute geometric structure.
• A robot state might contribute the agent’s current configuration.
• Language might contribute a relationship needed to interpret an image.

These pieces of information remain different, but downstream reasoning can operate on them through the same interface.

We call each representation a latent thought: a small continuous representation that carries useful information from one reasoning step to the next.

The aim is to preserve modality-specific information while providing a shared reasoning interface.

3. What should a thought remember?

Once we introduce a latent thought space, an important question follows:

What information should each thought contain?

One possibility is to train it to reconstruct the observation that produced it. This encourages the representation to preserve information about its source. But reasoning does not always require reconstructing everything that came before: only some details may matter for the next step.

This motivates a different objective, continuation prediction:

• Reconstruction asks a thought to recover its source observation.
• Continuation asks it to support prediction of what comes next.

Rather than asking the representation to describe the past, we ask it to preserve the information needed by the future. It becomes an information bottleneck between reasoning steps. ^{[1, §3.3]}

4. Generating thoughts

Continuation prediction defines what a useful thought should contain. During inference, however, the model cannot access the teacher’s intermediate reasoning steps. It needs to generate its own thoughts.

There are therefore two connected problems:

1. Learn useful latent representations.
2. Learn to generate them from the available context.

Uni-LaDiR uses a diffusion model for the second part. Given the same context, multiple intermediate thoughts may be valid. Instead of directly regressing to one latent target, diffusion learns a distribution over plausible next thoughts. ^{[1, §3.4]}

The model generates a thought, adds it to the context, and repeats. The resulting chain supports the final prediction.

With this intuition in place, we can describe the method more precisely.

5. Method

Uni-LaDiR consists of two closely connected components:

• A latent encoder converts intermediate multimodal reasoning steps into latent thoughts.
• A latent generator learns to generate those thoughts during inference.

Both operate through a shared model backbone and are trained jointly. The method diagram also shows continuation prediction, which teaches the encoded thoughts what information to retain. ^{[1, Fig. 2]}

5.1 Encoding multimodal reasoning steps

Each teacher reasoning step can be text, an image, a 3D point cloud, or a robot state. Its corresponding modality representation is processed first; a small set of learnable query tokens then collects information from that step through the shared encoder.

The query outputs are projected into a fixed-size latent block, z_i ∈ ℝ^{K × D}, where K is the number of latent tokens and D is their dimension. In the experiments, each thought contains four 512-dimensional latent tokens. All modalities use the same queries and output projection. ^{[1, §3.2; Appendix B.2]}

Each query block attends only to its own teacher step and itself while encoding. The task input and earlier thoughts provide the wider context when the reasoner uses the block. Sharing the encoder does not mean exposing each block to every teacher step.

5.2 Learning useful thoughts through continuation prediction

Suppose a teacher trajectory contains steps x₁, x₂, …, x_T. The encoder produces z_i = E(x_i).

Rather than reconstructing x_i from z_i, the model uses the task input and accumulated latent thoughts to predict later steps.

Raw earlier teacher steps are hidden from these predictions, as are future latent blocks. Information from an earlier step must therefore survive in its thought tokens. The final-output objective also connects the latent chain to successful end-to-end reasoning. ^{[1, §3.3]}

5.3 Generating latent thoughts with diffusion

Encoded thoughts provide training targets, but the teacher steps that produce them are unavailable at test time. Uni-LaDiR trains a diffusion generator using flow matching. ^{[1, §3.4][2]}

For a target thought z, noise is added at a sampled time t. Conditioned on the task and earlier thoughts, the model learns the velocity along a noisy-to-clean path.

The reasoner starts from random noise and iteratively denoises it into a latent thought.

The generated thought is appended to the context for subsequent thoughts. Because modalities share the same latent format, the same generation procedure applies regardless of a thought’s source modality.

5.4 Jointly learning the encoder and generator

A natural alternative is to learn the representation first, freeze it, and then train a generator. But a representation useful for encoding teacher steps may be difficult to generate from the available input.

Uni-LaDiR trains both jointly, allowing the latent space to become both useful for reasoning and feasible to generate.

The objective combines:

• Continuation prediction: preserve information needed by later steps.
• Final-output prediction: connect latent reasoning to task performance.
• Diffusion training: generate thoughts without teacher intermediates.

The clean targets and earlier thought blocks use stop-gradient during diffusion updates. Continuation and final-output losses train the encoded thoughts; diffusion trains their generator. Both update the shared backbone, coupling representation learning and generation. ^[1]

5.5 Inference

Given the input task, Uni-LaDiR:

1. Initializes a latent thought from noise.
2. Denoises it conditioned on the current context.
3. Adds the thought to the reasoning history.
4. Repeats for subsequent thoughts until the chain stops.
5. Predicts the final answer or action from the latent chain.

Intermediate multimodal supervision defines a useful space during training. At inference, reasoning proceeds through generated latent thoughts, without teacher steps. ^[1]

6. What do the experiments teach us?

The experiments evaluate two settings:

• Visual reasoning: answer questions requiring understanding and reasoning over visual information.
• Robot manipulation: reason over images, 3D geometry, and robot state to produce actions.

These use separate model instances of the same design, rather than one model jointly trained across both domains. We first evaluate the full system, then examine four design lessons. ^{[1, §4]}

Main results: Better visual answers and robot actions

Setup. For visual reasoning, train Qwen2.5-VL-7B on Zebra-CoT traces containing text and intermediate images, and compare against the base model and reasoning methods on the same evaluation questions. ^[6][7] For manipulation, evaluate on RLBench and LIBERO; the RLBench comparison follows LaST₀’s backbone, demonstration, training, and evaluation protocol. ^[5][8][9] At test time, the models generate their own latent thoughts without teacher steps.

Results. The complete system improves performance in both domains:

• Visual reasoning: mean accuracy over seven benchmarks rises from 64.6% for the base model and 72.2% for Mirage to 78.5% for Uni-LaDiR.
• Visual math and logic: mean accuracy over four benchmarks rises from 36.4% for the base model and 40.7% for Mirage to 49.0%.
• Robot manipulation: RLBench success improves from 82.0% for LaST₀ to 87.0%, with nine of ten tasks improved and the remaining task matched. LIBERO mean success is 99.2% across four suites. ^{[1, Tables 1–3]}

Uni-LaDiR runs at 16.5 Hz, versus 15.4 Hz for LaST₀ under the same RTX 4090 timing protocol, without action chunking and excluding simulator stepping and rendering. ^[1]

Lesson 1: A shared latent space can improve multimodal reasoning

Setup. Compare increasingly shared representations while keeping the backbone, data, teacher supervision, continuation objective, and thought budget fixed:

• Visual reasoning: separate text and image latent encoders versus a shared encoder.
• Robot manipulation: separate encoders versus pairwise sharing and sharing all three modalities—images, 3D point clouds, and robot state. All variants retain every teacher modality.

Results. Increased sharing improves performance in both domains:

• Visual reasoning: average accuracy across four benchmarks rises from 63.0% to 70.1% with shared text and image encoding.
• RLBench: success rises from 82.0% with separate encoders to 83.5–84.5% with pairwise sharing and 87.0% with all three shared. ^{[1, Fig. 3]}

These modalities have very different raw structures. The result suggests that their task-relevant information benefits from a common reasoning representation; it does not make the modalities interchangeable.

Lesson 2: Thoughts are more useful when they predict what comes next

Setup. Compare reconstruction and continuation prediction, with comparable latent capacity and downstream tasks. Test each objective with separate and unified encoders:

Results. Continuation prediction performs better across the evaluated visual and robot tasks. Averaged equally over six benchmarks, it improves performance by 6.6 percentage points with separate encoders and 9.2 points with unified encoding. ^{[1, Fig. 5; §4.4]}

Reconstruction may retain accurate details that are less useful for reasoning. Continuation prediction instead pressures the bottleneck to keep what later steps need. This is an interpretation of the objective and results, rather than a direct measurement of each latent token’s meaning.

Lesson 3: The latent space and its generator should be learned together

Setup. Compare two schedules:

• Separate training: learn the latent encoder first, then freeze it while training the generator.
• Joint training: optimize the latent encoder and generator together.

Results. Joint training improves the six-benchmark mean by 12.7% relative to staged training. ^{[1, Fig. 4]}

A useful thought must retain what later reasoning needs, and the model must be able to generate it from the available context.

A separately trained encoder may produce useful teacher-derived representations that are difficult for the generator to reproduce. Joint learning allows the shared space to adapt to both requirements.

Lesson 4: Diffusion generates more effective thoughts than direct prediction

Setup. Compare direct latent prediction with L2 or cosine objectives against flow matching. All variants use a unified encoder and joint training, with the same continuation and final-output supervision. Each generator receives the task input and preceding thoughts. ^{[1, §4.4; Appendix B.3]}

Results. Flow matching improves the six-benchmark mean by 16.3% relative to L2 prediction and 21.0% relative to cosine prediction. ^{[1, Fig. 6]}

This is consistent with the idea that several next thoughts can be valid and diffusion can model that distribution. The ablation establishes a performance advantage in these settings; it does not by itself prove that multiple valid continuations explain the entire gain.

7. Unification should not mean making modalities the same

At first glance, a shared latent space might sound like an attempt to erase the differences between modalities. I think the opposite interpretation is more useful.

We give a robot an image and a point cloud precisely because they tell us different things. The image is rich in appearance. The point cloud makes geometry explicit. Robot state tells the robot where its own body is. Language can express goals and abstractions that may not be directly visible at all.

If these modalities contained the same information, there would be little reason to use more than one of them.

So the goal of a unified representation should not be to force every modality into the same description of the world. What we want instead is a common interface for reasoning: a space in which different kinds of evidence can be expressed in a form that the next computation knows how to use.

This distinction matters. “Shared” does not have to mean “identical.” A visual thought may preserve appearance; a geometric thought may preserve spatial constraints. They can carry different information while still participating in the same reasoning process.

This differs in emphasis from the Platonic Representation Hypothesis, which asks whether representations converge toward common structure in the world. ^[10] That view examines what modalities have in common; our goal is to use their differences and complementarity. The two perspectives can coexist, but they ask different questions.

8. What should a thought preserve?

Once we think of a latent representation as an interface, a deeper question appears: what information should that interface carry?

A common answer in representation learning is that a good representation should preserve the input. If I compress an image, for example, I might judge the representation by how well I can reconstruct the original image.

But reasoning has a different objective.

Suppose a robot is looking at a mug on a crowded shelf. If its next task is to grasp the mug, the useful information may be the handle position, the free space around the object, and the current configuration of the robot arm. The exact texture of the wall behind the mug may be visually real, but computationally irrelevant.

Now change the task. If the robot is asked to sort objects by color, geometry becomes less important and appearance becomes central. If it is asked to count objects, yet another subset of the scene matters.

The observation is the same. What should be remembered is not.

This suggests a different way to think about representation learning:

In this view, forgetting is not necessarily a defect. A useful reasoning system should discard information all the time. The important question is whether it discards the right information.

That is why continuation prediction is conceptually interesting to me. Instead of asking a latent state to reconstruct its past, we ask whether it contains enough information to support what comes next. The representation is defined by its role in a computation. ^{[1, §3.3; Fig. 5]}

You can think of a latent thought as a message passed from one stage of reasoning to another. The sender does not need to transmit everything it has seen. It needs to transmit what the receiver needs to do its job.

This question extends beyond Uni-LaDiR. Implicit CoT and Coconut explore reasoning without writing every intermediate step; Mirage lets visual latents adapt to answering the question; LaDiR learns a space in which thought blocks can be generated and refined. ^{[13][11][3][12]} Across these approaches, a useful question remains: what information should survive to support the next computation?

9. From multimodal input to multimodal learning

This perspective also points to what I think is the more interesting long-term question.

Multimodal systems are often evaluated on whether they can use multiple modalities at the same time. Give the model an image and language, or an image and 3D geometry, and ask whether the combination improves performance.

But there is a stronger form of multimodal intelligence:

Imagine that a model has repeatedly learned about physical clearance through 3D observations. Later, it receives only an RGB image. Does its previous geometric experience help it infer which gaps are large enough for a hand to pass through?

Or imagine that a robot has learned, through interaction, which objects are deformable, movable, or graspable. Can that experience later improve its understanding of a novel scene from vision alone?

This would mean that modalities are doing more than contributing extra information at inference time. They would be teaching one another reusable abstractions about the world.

Uni-LaDiR does not yet demonstrate this. The visual reasoning experiments and the robot manipulation experiments use separate model instances, so the current results should not be interpreted as evidence that 3D or robotic experience improves later visual reasoning. ^{[1, §4]}

But the shared-interface view suggests a concrete next step. If different modalities can express task-relevant information through a compatible reasoning space, then knowledge learned through one modality may eventually become useful to computations initiated by another.

That, to me, is the more ambitious promise of multimodal reasoning.

The goal is not simply to build a model with more input channels. It is to build a system in which vision, geometry, language, and interaction can contribute to a common computational process—and, eventually, teach one another what is worth remembering.

References

1. Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang, Jiatao Gu, Yian Ma, and Lianhui Qin. Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning. arXiv:2609.19878, 2026. Chart data.
2. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow Matching for Generative Modeling. ICLR, 2023.
3. Zeyuan Yang, Xueyang Yu, Delin Chen, Maohao Shen, and Chuang Gan. Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens. arXiv:2506.17218, 2025.
4. Moo Jin Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model. CoRL, 2024.
5. Zhuoyang Liu et al. LaST₀: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Model. arXiv:2601.05248, 2026.
6. Shuai Bai et al. Qwen2.5-VL Technical Report. arXiv:2502.13923, 2025.
7. Ang Li et al. Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning. arXiv:2507.16746, 2025.
8. Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. RLBench: The Robot Learning Benchmark & Learning Environment. IEEE Robotics and Automation Letters, 2020.
9. Bo Liu et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. NeurIPS, 2023.
10. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The Platonic Representation Hypothesis. ICML, 2024.
11. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training Large Language Models to Reason in a Continuous Latent Space. COLM, 2025.
12. Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang, Nicklas Majamaki, Navdeep Jaitly, Yi-An Ma, and Lianhui Qin. LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning. ICLR, 2026.
13. Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit Chain of Thought Reasoning via Knowledge Distillation. arXiv:2311.01460, 2023.

Citation

If you found this post helpful, please consider citing our paper:

@misc{kang2026uniladir,
  title = {{Uni-LaDiR}: Latent Diffusion Unifies Multimodal Reasoning},
  author = {Haoqiang Kang and Yizhe Zhang and Nikki Lijing Kuang
            and Jiatao Gu and Yian Ma and Lianhui Qin},
  year = {2026},
  eprint = {2609.19878},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  doi = {10.48550/arXiv.2609.19878},
  url = {https://arxiv.org/abs/2609.19878}
}