RESEARCH NOTE
Code-space vs token-space masked diffusion: how to compare them
A stage-consistent comparison protocol for code-space and token-space masked diffusion language models when the discrete codec is lossy.
DIRECT ANSWER
How should code-space and token-space masked diffusion language modeling be compared?
Evaluate source text, codec reconstructions, token-space outputs, and decoded code-space outputs with the same external scorer and preprocessing. Report the codec reconstruction gap separately: code-space generation cannot recover information that the codec already removed.
Why the distinction matters
The method and the claimed guarantee need the same observable boundary.
A token-space model generates text tokens directly. A code-space model first generates discrete latent codes and then relies on a decoder to recover text. Their final outputs can be compared, but the code-space pipeline has an extra failure point: reconstruction loss can lower the quality ceiling before latent generation starts.
A fair comparison therefore needs two questions, not one. First ask how much quality the codec preserves without any generation. Then ask how much additional loss each generator introduces under one unchanged decoded-text evaluation protocol.
A practical procedure
Establish the source baseline
Score held-out source texts with the evaluator and preprocessing that will be used for every later stage.
Measure reconstruction before generation
Encode and decode the same examples without sampling new codes. This isolates the ceiling imposed by the codec.
Evaluate both generation spaces after decoding
Score token-space samples directly and decode code-space samples before scoring them. Do not compare a latent proxy with a decoded-text metric.
Report distributions and uncertainty
Show median and tail behavior, sample counts, preprocessing, and repeated-run uncertainty. One average can hide severe reconstruction failures.
Evidence to require
A claim is only as strong as the property measured after generation or decoding.
- The same external decoded-text evaluator and preprocessing are used at every checkpoint.
- Source, reconstruction, token-space, and decoded code-space results are reported separately.
- The codec reconstruction gap is not attributed to the latent generator.
- Median and tail behavior accompany any mean comparison.
- Seeds or uncertainty estimates are reported before small differences are generalized.
What the linked study reports
- In the reported 64-to-16 TinyStories setting, codec reconstruction alone raised median external perplexity from 15.17 to 27.36.
- Under the same scorer, code-space MDLM reached median perplexity 26.55, while the token-space baseline reached 38.42, a 30.9% reduction for code-space generation in that experiment.
- Geometry-aware regularization improved local latent diagnostics in the available matched runs but did not improve decoded-text metrics.
Scope boundary
- These numbers describe one TinyStories compression regime, one hierarchical codec, and the reported evaluation setup; they do not establish a universal ordering of code-space and token-space models.
- Perplexity is informative but not a complete measure of semantic quality, diversity, factuality, or usefulness.
- The available comparisons should be read with their stated sample sizes and uncertainty limitations.
Primary and nearby sources
Use the linked papers for the original methods, measurements, and stated limitations.
- Where Quality Breaks in Compressed Short-Text Generation
Primary paper and reported stage-wise measurements.
- Simple and Effective Masked Diffusion Language Models
Masked discrete diffusion formulation used by the latent generator.
- Neural Discrete Representation Learning
Foundational learned discrete-representation method.