# Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization > A staged diagnosis of compressed short-text generation that separates codec reconstruction loss from latent-generation loss. Canonical HTML: https://aogavrilov.com/publications/where-quality-breaks/ DOI: https://doi.org/10.23919/FRUCT70069.2026.11506553 PDF: https://aogavrilov.com/publications/where-quality-breaks/paper.pdf Full-text HTML: https://aogavrilov.com/publications/where-quality-breaks/full-text/ Full-text Markdown: https://aogavrilov.com/publications/where-quality-breaks/full-text/index.md Full-text section and source-block index: https://aogavrilov.com/publications/where-quality-breaks/full-text/sections.json W3C Web Annotation block collection: https://aogavrilov.com/publications/where-quality-breaks/full-text/annotations.jsonld Full-text NISO JATS 1.4 XML: https://aogavrilov.com/publications/where-quality-breaks/full-text/article.jats.xml Thematic research guide: https://aogavrilov.com/research/codec-bottleneck-diagnosis/ This is a plain Markdown reader representation of the canonical HTML publication guide. Use the DOI and original English metadata below for citation. ## Bibliographic record - Authors: Alexey Gavrilov; Alan-Barsag Gazzaev; Sergey Muravyov - Published: 2026-04-28 - Venue: 2026 39th Conference of Open Innovations Association (FRUCT) - Publication format: Main conference - Pages: 69-76 - Publisher: IEEE ## Paper in 30 seconds ### Problem In compressed short-text generation, poor decoded text may come from information lost by the codec or weak latent-space generation. End-to-end scores blur those failure modes and can send optimization effort to the wrong component. ### Approach The staged protocol scores originals, paired codec reconstructions, token-space MDLM outputs, and code-space diffusion outputs with one external GPT-2 evaluator. Codebook and geometry measures remain diagnostics rather than substitutes for decoded-text quality. ### Main result Codec reconstruction raises median external perplexity from 15.17 to 27.36 and p95 from 25.10 to 98.91. Code-space MDLM still reduces median perplexity by 30.9% versus token-space MDLM. ### Why it matters The result turns codec bottleneck diagnosis into an actionable order of work: prioritize codec improvements, then compare token-space and code-space generation, and trust latent proxy gains only when decoded text improves too. ## Abstract Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes. Without separating these failure modes, researchers can spend compute improving the wrong component. We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). We use a staged validation protocol that separates codec reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer, while reporting complementary semantic metrics for the geometry study. In the tested configuration, codec reconstruction alone raises median external perplexity from 15.17 to 27.36 (+80.4%) and p95 from 25.10 to 98.91 (+294.1%), showing that the dominant quality loss appears before latent generation begins. Under the same scorer, code-space MDLM remains materially stronger than token-space diffusion, reducing mean, median, and p95 by 32.9%, 30.9%, and 36.6%, respectively. Geometry-aware regularization improves local latent proxies but does not improve decoded-text metrics in the available runs. The contribution is methodological rather than algorithmic: the paper presents a reusable staged diagnosis for one concrete pipeline and shows that, in this setting, codec fidelity rather than latent denoising sets the practical quality ceiling. ## Key results | Evaluation point | n | Mean PPL | Median PPL | p95 PPL | | --- | --- | --- | --- | --- | | Original texts | 256 | 16.24 | 15.17 | 25.1 | | Codec reconstructions | 256 | 37.26 | 27.36 | 98.91 | | AR baseline | 251 | 30.98 | 23.27 | 56.11 | | Token-space MDLM | 256 | 44.74 | 38.42 | 93.6 | | Code-space MDLM | 256 | 30.01 | 26.55 | 59.36 | Most of the observed quality loss is introduced before generation; code-space diffusion still reduces median perplexity by 30.9% versus token-space diffusion. - Dataset: TinyStories - Sample size: 256 paired reconstruction samples; 251–256 generated samples per mode; four matched geometry settings. - Conditions: GPT-2 token sequences of length 64 compressed to 16 top-level codes with a hierarchical VQ-VAE-2; all generation modes use the shared external scorer. - Uncertainty: The reported comparisons are descriptive single runs; confidence intervals and multi-seed significance estimates were not computed. - External result-summary mirror: https://huggingface.co/datasets/aogavrilov/where-quality-breaks-results - External result-summary collection: https://huggingface.co/collections/aogavrilov/publication-result-summaries-6a6b733b3116a87befb37ec2 ## Questions this paper helps answer ### Where does quality break in compressed short-text generation? In the tested TinyStories 64-to-16 pipeline, the dominant degradation appears at codec reconstruction, before latent generation begins. Median external GPT-2 perplexity increases from 15.17 for original text to 27.36 after reconstruction, while p95 increases from 25.10 to 98.91. This is a result for one configuration rather than a universal codec ranking. ### How can codec loss be separated from latent-generation loss? The staged protocol evaluates originals, paired codec reconstructions, and final generated text with one shared external scorer. The original-to-reconstruction gap estimates the codec contribution, while the reconstruction-to-generated-output comparison helps localize the additional generation-stage loss. Latent health metrics are reported separately from decoded-text evidence. ### How should discrete-latent text generation be evaluated? The paper recommends checking reconstruction fidelity before comparing generators, applying the same decoded-text scorer at every stage, and reporting central and tail statistics. It also treats codebook usage, geometry, and other latent proxies as diagnostics rather than substitutes for the quality of decoded text. ### Does code-space diffusion outperform token-space diffusion? Under the shared scorer and tested setup, code-space MDLM reduces mean, median, and p95 perplexity by 32.9%, 30.9%, and 36.6% relative to token-space MDLM. The comparison is descriptive and configuration-specific: it does not establish a universal ranking across datasets, compression ratios, codecs, or diffusion architectures. ### Do better latent geometry metrics guarantee better generated text? No. In the available matched runs, geometry-aware regularization improved local latent-space proxies but did not improve decoded-text metrics. The result supports auditing every proxy gain at the final decoded output and treating the absence of transfer as a useful negative result rather than evidence of generation improvement. ## Limitations - The empirical study uses TinyStories only. - The main analysis covers one aggressive 64-to-16 compression regime. - The evaluated system combines one hierarchical VQ-VAE-2 codec family with one MDLM generator. - Comparisons are based on single runs and are descriptive rather than multi-seed statistical estimates. - External GPT-2 perplexity is a shared diagnostic, not a universal semantic-quality metric. - The conclusions should not be transferred directly to all datasets, codec architectures, or compression ratios. ## Availability - Public implementation: no implementation is linked in the maintained publication record. - Public data: no underlying experimental dataset is linked in the maintained publication record. ## References cited by the paper These entries correspond to the numbered References section in the paper PDF. - [1] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, Volodymyr Kuleshov. 2024. "Simple and Effective Masked Diffusion Language Models." Advances in Neural Information Processing Systems. https://arxiv.org/abs/2406.07524 arXiv: https://arxiv.org/abs/2406.07524 - [2] Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu. 2017. "Neural Discrete Representation Learning." Advances in Neural Information Processing Systems. https://arxiv.org/abs/1711.00937 arXiv: https://arxiv.org/abs/1711.00937 - [3] Ali Razavi, Aaron van den Oord, Oriol Vinyals. 2019. "Generating Diverse High-Fidelity Images with VQ-VAE-2." Advances in Neural Information Processing Systems. https://arxiv.org/abs/1906.00446 arXiv: https://arxiv.org/abs/1906.00446 - [4] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg. 2021. "Structured Denoising Diffusion Models in Discrete State-Spaces." Advances in Neural Information Processing Systems. https://arxiv.org/abs/2107.03006 arXiv: https://arxiv.org/abs/2107.03006 - [5] Aaron Lou, Chenlin Meng, Stefano Ermon. 2024. "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution." Proceedings of the 41st International Conference on Machine Learning. https://arxiv.org/abs/2310.16834 arXiv: https://arxiv.org/abs/2310.16834 - [6] Xiaochuang Han, Sachin Kumar, Yulia Tsvetkov. 2023. "SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.acl-long.647 DOI: https://doi.org/10.18653/v1/2023.acl-long.647 arXiv: https://arxiv.org/abs/2210.17432 - [7] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, Tatsunori B. Hashimoto. 2022. "Diffusion-LM Improves Controllable Text Generation." Advances in Neural Information Processing Systems. https://arxiv.org/abs/2205.14217 arXiv: https://arxiv.org/abs/2205.14217 - [8] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman. 2022. "MaskGIT: Masked Generative Image Transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content/CVPR2022/html/Chang_MaskGIT_Masked_Generative_Image_Transformer_CVPR_2022_paper.html arXiv: https://arxiv.org/abs/2202.04200 - [9] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer. 2019. "Mask-Predict: Parallel Decoding of Conditional Masked Language Models." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. https://doi.org/10.18653/v1/D19-1633 DOI: https://doi.org/10.18653/v1/D19-1633 arXiv: https://arxiv.org/abs/1904.09324 - [10] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, Max Welling. 2021. "Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions." Advances in Neural Information Processing Systems. https://arxiv.org/abs/2102.05379 arXiv: https://arxiv.org/abs/2102.05379 - [11] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. 2019. "Language Models are Unsupervised Multitask Learners." OpenAI Technical Report. https://cdn.openai.com/better-language-models/language-models.pdf - [12] Nils Reimers, Iryna Gurevych. 2019. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. https://doi.org/10.18653/v1/D19-1410 DOI: https://doi.org/10.18653/v1/D19-1410 arXiv: https://arxiv.org/abs/1908.10084 - [13] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi. 2020. "BERTScore: Evaluating Text Generation with BERT." International Conference on Learning Representations. https://arxiv.org/abs/1904.09675 arXiv: https://arxiv.org/abs/1904.09675 - [14] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, Zaid Harchaoui. 2021. "MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers." Advances in Neural Information Processing Systems. https://arxiv.org/abs/2102.01454 arXiv: https://arxiv.org/abs/2102.01454 - [15] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica. 2023. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Advances in Neural Information Processing Systems, Datasets and Benchmarks Track. https://arxiv.org/abs/2306.05685 arXiv: https://arxiv.org/abs/2306.05685 ## Citation and machine-readable resources - Canonical HTML: https://aogavrilov.com/publications/where-quality-breaks/ - DOI record: https://doi.org/10.23919/FRUCT70069.2026.11506553 - PDF: https://aogavrilov.com/publications/where-quality-breaks/paper.pdf - Full-text HTML: https://aogavrilov.com/publications/where-quality-breaks/full-text/ - Full-text Markdown: https://aogavrilov.com/publications/where-quality-breaks/full-text/index.md - W3C Web Annotation block collection: https://aogavrilov.com/publications/where-quality-breaks/full-text/annotations.jsonld - Full-text NISO JATS 1.4 XML: https://aogavrilov.com/publications/where-quality-breaks/full-text/article.jats.xml - BibTeX: https://aogavrilov.com/publications/where-quality-breaks/citation.bib - APA formatted citation: https://aogavrilov.com/publications/where-quality-breaks/citation-apa.txt - IEEE formatted citation: https://aogavrilov.com/publications/where-quality-breaks/citation-ieee.txt - RIS: https://aogavrilov.com/publications/where-quality-breaks/citation.ris - CSL-JSON: https://aogavrilov.com/publications/where-quality-breaks/citation.json - OAI Dublin Core XML: https://aogavrilov.com/publications/where-quality-breaks/oai-dc.xml - MODS XML: https://aogavrilov.com/publications/where-quality-breaks/mods.xml - NISO JATS 1.4 XML metadata: https://aogavrilov.com/publications/where-quality-breaks/metadata.jats.xml - RDF Turtle: https://aogavrilov.com/publications/where-quality-breaks/metadata.ttl - RO-Crate: https://aogavrilov.com/publications/where-quality-breaks/ro-crate-metadata.json