# How to tell whether a compressed text generator fails in the codec or the generator

Canonical HTML: https://aogavrilov.com/research/codec-bottleneck-diagnosis/

This Markdown guide routes a research problem to a method, required evidence, and explicit scope boundaries.

## Use this guide when

- A pipeline compresses text into discrete codes before generation.
- Decoded quality is poor and the responsible stage is unknown.
- A latent-space proxy improves without a matching decoded-output improvement.
- Code-space and token-space masked diffusion language models must be compared under one decoded-text scorer.

## Procedure

1. **Establish the reference.** Score held-out original texts with the external evaluator used for all later stages.
2. **Measure codec reconstruction.** Encode and decode the same examples without generation to isolate bottleneck loss.
3. **Measure latent generation.** Generate codes, decode them, and score the resulting text with the unchanged evaluator.
4. **Audit proxy transfer.** Compare latent diagnostics with final decoded-text metrics rather than assuming transfer.

## Bounded evidence

- **Setting / Dataset:** TinyStories
- **Setting / Source tokens:** 64
- **Setting / Top level codes:** 16
- **Setting / Codec family:** hierarchical VQ-VAE-2
- **Setting / Evaluator:** one external GPT-2 scorer
- **Reconstruction perplexity / Source median:** 15.17
- **Reconstruction perplexity / Reconstruction median:** 27.36
- **Reconstruction perplexity / Source p95:** 25.1
- **Reconstruction perplexity / Reconstruction p95:** 98.91
- **Generated text perplexity / Code space mdlm median:** 26.55
- **Generated text perplexity / Token space mdlm median:** 38.42
- **Generated text perplexity / Code space relative reduction percent:** 30.9
- **Proxy transfer observation:** Geometry-aware regularization improved local latent proxies in matched runs but did not improve decoded-text metrics.

## Questions answered

### How did code-space and token-space masked diffusion compare in the reported text experiment?

Under the same external scorer, code-space MDLM had median perplexity 26.55 versus 38.42 for the token-space baseline, a 30.9% reduction. The codec reconstruction median was already 27.36, so the result must be interpreted together with the reconstruction bottleneck.

Evidence: https://aogavrilov.com/research/codec-bottleneck-diagnosis/#code-space-vs-token-space

### How should code-space and token-space masked diffusion be compared when the codec is lossy?

Use the same held-out samples and decoded-text scorer for originals, codec reconstructions, token-space outputs, and code-space outputs. Report the reconstruction gap separately, because a stronger latent generator cannot recover information already removed by the codec.

Evidence: https://aogavrilov.com/research/codec-bottleneck-diagnosis/#code-space-vs-token-space

### How can quality loss be diagnosed in a two-stage text generator?

Measure the original-to-reconstruction gap before the reconstruction-to-generation gap under one unchanged decoded-text evaluator. This separates the quality ceiling imposed by the codec from additional degradation introduced by latent generation.

Evidence: https://aogavrilov.com/research/codec-bottleneck-diagnosis/#workflow

### When can better latent-space metrics fail to improve decoded output?

A latent proxy can improve without tracking the downstream property of interest. Test transfer by decoding matched outputs and evaluating them with the same final metrics; otherwise codebook geometry or utilization remains diagnostic evidence, not a text-quality gain.

Evidence: https://aogavrilov.com/research/codec-bottleneck-diagnosis/#decision-table

## Limitations

- The reported numerical ordering is not universal.
- The experiment covers one TinyStories 64-to-16 configuration, one hierarchical codec family, one masked discrete generator, and descriptive single runs.
- Apply the staged protocol to a new system; do not copy the numerical conclusion into a different setting.

## Source publications

- [Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization](https://aogavrilov.com/publications/where-quality-breaks/) - DOI: https://doi.org/10.23919/FRUCT70069.2026.11506553

## Primary background

- [Neural Discrete Representation Learning](https://arxiv.org/abs/1711.00937)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://arxiv.org/abs/1906.00446)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://arxiv.org/abs/2305.07759)
- [Simple and Effective Masked Diffusion Language Models](https://arxiv.org/abs/2406.07524)

Research-guide router: https://aogavrilov.com/research-guides.json