# Code-space vs token-space masked diffusion: how to compare them

Canonical HTML: https://aogavrilov.com/research-notes/code-space-vs-token-space-masked-diffusion/
Document language: en
Source publication: https://aogavrilov.com/publications/where-quality-breaks/
Source full-text HTML: https://aogavrilov.com/publications/where-quality-breaks/full-text/
DOI: https://doi.org/10.23919/FRUCT70069.2026.11506553

A stage-consistent comparison protocol for code-space and token-space masked diffusion language models when the discrete codec is lossy.

## Direct answer

### How should code-space and token-space masked diffusion language modeling be compared?

Evaluate source text, codec reconstructions, token-space outputs, and decoded code-space outputs with the same external scorer and preprocessing. Report the codec reconstruction gap separately: code-space generation cannot recover information that the codec already removed.

## Why the distinction matters

A token-space model generates text tokens directly. A code-space model first generates discrete latent codes and then relies on a decoder to recover text. Their final outputs can be compared, but the code-space pipeline has an extra failure point: reconstruction loss can lower the quality ceiling before latent generation starts.

A fair comparison therefore needs two questions, not one. First ask how much quality the codec preserves without any generation. Then ask how much additional loss each generator introduces under one unchanged decoded-text evaluation protocol.

## Practical procedure

1. **Establish the source baseline.** Score held-out source texts with the evaluator and preprocessing that will be used for every later stage.
2. **Measure reconstruction before generation.** Encode and decode the same examples without sampling new codes. This isolates the ceiling imposed by the codec.
3. **Evaluate both generation spaces after decoding.** Score token-space samples directly and decode code-space samples before scoring them. Do not compare a latent proxy with a decoded-text metric.
4. **Report distributions and uncertainty.** Show median and tail behavior, sample counts, preprocessing, and repeated-run uncertainty. One average can hide severe reconstruction failures.

## Evidence to require

- The same external decoded-text evaluator and preprocessing are used at every checkpoint.
- Source, reconstruction, token-space, and decoded code-space results are reported separately.
- The codec reconstruction gap is not attributed to the latent generator.
- Median and tail behavior accompany any mean comparison.
- Seeds or uncertainty estimates are reported before small differences are generalized.

## What the linked study reports

- In the reported 64-to-16 TinyStories setting, codec reconstruction alone raised median external perplexity from 15.17 to 27.36.
- Under the same scorer, code-space MDLM reached median perplexity 26.55, while the token-space baseline reached 38.42, a 30.9% reduction for code-space generation in that experiment.
- Geometry-aware regularization improved local latent diagnostics in the available matched runs but did not improve decoded-text metrics.

## Scope boundary

- These numbers describe one TinyStories compression regime, one hierarchical codec, and the reported evaluation setup; they do not establish a universal ordering of code-space and token-space models.
- Perplexity is informative but not a complete measure of semantic quality, diversity, factuality, or usefulness.
- The available comparisons should be read with their stated sample sizes and uncertainty limitations.

Complete research guide: https://aogavrilov.com/projects/codec-bottleneck-diagnosis/#code-space-vs-token-space

## Primary and nearby sources

- [Where Quality Breaks in Compressed Short-Text Generation](https://aogavrilov.com/publications/where-quality-breaks/): Primary paper and reported stage-wise measurements.
- [Simple and Effective Masked Diffusion Language Models](https://arxiv.org/abs/2406.07524): Masked discrete diffusion formulation used by the latent generator.
- [Neural Discrete Representation Learning](https://arxiv.org/abs/1711.00937): Foundational learned discrete-representation method.

This maintained note summarizes existing evidence and adds no experimental result beyond the cited sources.