AG — Alexey Gavrilov home

Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization

A staged diagnosis of compressed short-text generation that separates codec reconstruction loss from latent-generation loss.

Alexey Gavrilov1Alan-Barsag Gazzaev1Sergey Muravyov1

  1. ITMO University, Saint Petersburg, Russia

Read the full paper in HTMLSearchable text with formulas, tables, figures, and references.

Final accepted manuscript (author-posted version with DOI). © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting or republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Posting and reuse terms.

Paper in 30 seconds

Research questionHow can quality loss in a compressed text-generation pipeline be attributed separately to the codec and to the latent generator?

Problem

In compressed short-text generation, poor decoded text may come from information lost by the codec or weak latent-space generation. End-to-end scores blur those failure modes and can send optimization effort to the wrong component.

Approach

The staged protocol scores originals, paired codec reconstructions, token-space MDLM outputs, and code-space diffusion outputs with one external GPT-2 evaluator. Codebook and geometry measures remain diagnostics rather than substitutes for decoded-text quality.

Main result

Codec reconstruction raises median external perplexity from 15.17 to 27.36 and p95 from 25.10 to 98.91. Code-space MDLM still reduces median perplexity by 30.9% versus token-space MDLM.

Why it matters

The result turns codec bottleneck diagnosis into an actionable order of work: prioritize codec improvements, then compare token-space and code-space generation, and trust latent proxy gains only when decoded text improves too.

Abstract

Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes. Without separating these failure modes, researchers can spend compute improving the wrong component. We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). We use a staged validation protocol that separates codec reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer, while reporting complementary semantic metrics for the geometry study. In the tested configuration, codec reconstruction alone raises median external perplexity from 15.17 to 27.36 (+80.4%) and p95 from 25.10 to 98.91 (+294.1%), showing that the dominant quality loss appears before latent generation begins. Under the same scorer, code-space MDLM remains materially stronger than token-space diffusion, reducing mean, median, and p95 by 32.9%, 30.9%, and 36.6%, respectively. Geometry-aware regularization improves local latent proxies but does not improve decoded-text metrics in the available runs. The contribution is methodological rather than algorithmic: the paper presents a reusable staged diagnosis for one concrete pipeline and shows that, in this setting, codec fidelity rather than latent denoising sets the practical quality ceiling.

Published at 2026 39th Conference of Open Innovations Association (FRUCT)

Contribution type Diagnostic methodology

issue 1pp. 69–76Main conference

DOI https://doi.org/10.23919/FRUCT70069.2026.11506553

Share this paperShare

Key results

Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization key results
Evaluation pointnMean PPLMedian PPLp95 PPL
Original texts25616.2415.1725.1
Codec reconstructions25637.2627.3698.91
AR baseline25130.9823.2756.11
Token-space MDLM25644.7438.4293.6
Code-space MDLM25630.0126.5559.36

Key result. Most of the observed quality loss is introduced before generation; code-space diffusion still reduces median perplexity by 30.9% versus token-space diffusion.

Dataset
TinyStories
Sample size
256 paired reconstruction samples; 251–256 generated samples per mode; four matched geometry settings.
Metrics
External GPT-2 perplexity: mean, median, p95, and maximum; Codebook usage and support size; SBERT, BERTScore, MAUVE, and an LLM-judge summary for geometry runs
Uncertainty
The reported comparisons are descriptive single runs; confidence intervals and multi-seed significance estimates were not computed.
Conditions
GPT-2 token sequences of length 64 compressed to 16 top-level codes with a hierarchical VQ-VAE-2; all generation modes use the shared external scorer.
Download results:

CSVJSONMarkdownExternal mirror:Hugging Face dataset card

PDF & citation

Cite this paper BibTeX is the recommended format. Every variant below is generated from the same publication record.

Open PDF
@inproceedings{Gavrilov2026WhereQuality,
  title      = {Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization},
  author     = {Gavrilov, Alexey and Gazzaev, Alan-Barsag and Muravyov, Sergey},
  booktitle  = {2026 39th Conference of Open Innovations Association (FRUCT)},
  publisher  = {IEEE},
  year       = {2026},
  pages      = {69--76},
  doi        = {10.23919/FRUCT70069.2026.11506553},
  url        = {https://doi.org/10.23919/FRUCT70069.2026.11506553},
  isbn       = {978-952-65246-5-8},
}
Download .bib
Citation files:

APA textIEEE textRISCSL-JSONSchema.org JSON-LDOAI-DC XMLOpenAIRE v4 XMLMODS XMLJATS 1.4 metadata XMLFull-text JATS 1.4 XMLRDF TurtleLink Set (JSON)Link Set (HTTP)RO-Crate

DOI:https://doi.org/10.23919/FRUCT70069.2026.11506553

Full guide

Full research guide

Method

The paper uses one scorer across three evaluation stages so that each additional transformation can be associated with a measurable quality gap.

  1. Reconstruct

    Encode each 64-token TinyStories sample into 16 top-level codes and decode it immediately to measure codec reconstruction loss.

  2. Generate

    Generate either text tokens or discrete latent codes with MDLM, then decode code-space samples through the same trained codec.

  3. Compare

    Score originals, reconstructions, and generated texts with the same external GPT-2 protocol, including median and tail statistics.

Staged compressed-text generation pipeline comparing original text, codec reconstruction, code-space MDLM, and decoded text to separate codec loss from generation loss.
Staged bottleneck localization separates codec reconstruction loss from latent-generation loss under one shared decoded-text evaluation protocol.Source: Author-created explanatory diagram based on the published method and results..Reuse terms: CC BY 4.0.Suggested attribution: Gavrilov, Gazzaev, and Muravyov (2026), Where Quality Breaks in Compressed Short-Text Generation. Download SVG.

Key idea

A downstream generator cannot recover information that the codec has already discarded. The reconstruction stage must therefore be audited before interpreting latent-generation results.

Difference from nearby approaches

Standard end-to-end comparisons report one final generation score. This protocol inserts a paired reconstruction checkpoint and separates health metrics in latent space from evidence on decoded text.

What is new

The contribution is a reusable staged diagnostic methodology for one concrete compressed-text pipeline, rather than a new denoising algorithm.

Questions this paper helps answer

Open a question for a concise answer grounded in the paper. Detailed evidence boundaries are listed in Limitations.

  1. Where does quality break in compressed short-text generation?

    In the tested TinyStories 64-to-16 pipeline, the dominant degradation appears at codec reconstruction, before latent generation begins. Median external GPT-2 perplexity increases from 15.17 for original text to 27.36 after reconstruction, while p95 increases from 25.10 to 98.91. This is a result for one configuration rather than a universal codec ranking.

  2. How can codec loss be separated from latent-generation loss?

    The staged protocol evaluates originals, paired codec reconstructions, and final generated text with one shared external scorer. The original-to-reconstruction gap estimates the codec contribution, while the reconstruction-to-generated-output comparison helps localize the additional generation-stage loss. Latent health metrics are reported separately from decoded-text evidence.

  3. How should discrete-latent text generation be evaluated?

    The paper recommends checking reconstruction fidelity before comparing generators, applying the same decoded-text scorer at every stage, and reporting central and tail statistics. It also treats codebook usage, geometry, and other latent proxies as diagnostics rather than substitutes for the quality of decoded text.

  4. Does code-space diffusion outperform token-space diffusion?

    Under the shared scorer and tested setup, code-space MDLM reduces mean, median, and p95 perplexity by 32.9%, 30.9%, and 36.6% relative to token-space MDLM. The comparison is descriptive and configuration-specific: it does not establish a universal ranking across datasets, compression ratios, codecs, or diffusion architectures.

  5. Do better latent geometry metrics guarantee better generated text?

    No. In the available matched runs, geometry-aware regularization improved local latent-space proxies but did not improve decoded-text metrics. The result supports auditing every proxy gain at the final decoded output and treating the absence of transfer as a useful negative result rather than evidence of generation improvement.

Comparison with nearby approaches

Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization factual comparison with nearby approaches
ApproachRepresentationControl / diagnosisWhat is preserved or measured
Token-space diffusionText tokensGeneration quality in token spaceFluency and scorer behavior of direct generation
Compressed latent generationDiscrete latent codes via a codecFinal end-to-end generation qualityCombined codec and latent-generator behavior
Staged bottleneck localizationText, codec reconstructions, and discrete latentsSeparate codec loss from generation lossWhere quality degrades under one shared scorer

The comparison distinguishes evaluation scope and evidence; it is not a universal ranking of the approaches.

Relevance & scope

The staged protocol is useful when a generative pipeline contains both a learned codec and a latent-space generator, but the source of degraded output quality is unclear.

  1. Compressed and discrete-latent text generation

  2. Codec reconstruction fidelity in generative pipelines

  3. Masked diffusion language modeling in code space

  4. Bottleneck localization and stage-consistent evaluation

  5. Auditing latent-space proxy improvements against decoded text

See limitations and evidence boundaries

Limitations

  • The empirical study uses TinyStories only.
  • The main analysis covers one aggressive 64-to-16 compression regime.
  • The evaluated system combines one hierarchical VQ-VAE-2 codec family with one MDLM generator.
  • Comparisons are based on single runs and are descriptive rather than multi-seed statistical estimates.
  • External GPT-2 perplexity is a shared diagnostic, not a universal semantic-quality metric.
  • The conclusions should not be transferred directly to all datasets, codec architectures, or compression ratios.

References cited by the paper

These entries correspond to the numbered References section in the paper PDF.

  1. Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, Volodymyr Kuleshov. . Simple and Effective Masked Diffusion Language Models. Advances in Neural Information Processing Systems.
  2. Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu. . Neural Discrete Representation Learning. Advances in Neural Information Processing Systems.
  3. Ali Razavi, Aaron van den Oord, Oriol Vinyals. . Generating Diverse High-Fidelity Images with VQ-VAE-2. Advances in Neural Information Processing Systems.
  4. Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg. . Structured Denoising Diffusion Models in Discrete State-Spaces. Advances in Neural Information Processing Systems.
  5. Aaron Lou, Chenlin Meng, Stefano Ermon. . Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. Proceedings of the 41st International Conference on Machine Learning.
  6. Xiaochuang Han, Sachin Kumar, Yulia Tsvetkov. . SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics.
  7. Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, Tatsunori B. Hashimoto. . Diffusion-LM Improves Controllable Text Generation. Advances in Neural Information Processing Systems.
  8. Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman. . MaskGIT: Masked Generative Image Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  9. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer. . Mask-Predict: Parallel Decoding of Conditional Masked Language Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
  10. Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, Max Welling. . Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions. Advances in Neural Information Processing Systems.
  11. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. . Language Models are Unsupervised Multitask Learners. OpenAI Technical Report.
  12. Nils Reimers, Iryna Gurevych. . Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
  13. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi. . BERTScore: Evaluating Text Generation with BERT. International Conference on Learning Representations.
  14. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, Zaid Harchaoui. . MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. Advances in Neural Information Processing Systems.
  15. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica. . Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, Datasets and Benchmarks Track.

Resources & reproducibility

Publisher
IEEE
Publication resources
The public manuscript, results tables, explanatory figure, and citation files are available here. Implementation code and checkpoints are not publicly released.

Data statement

Source
TinyStories, as described in the paper.
License
Use of TinyStories remains subject to the dataset's own terms; no dataset files are redistributed by this site.
Preprocessing
GPT-2 tokenization, fixed 64-token inputs, and hierarchical temporal compression from 64 to 32 to 16 positions.
Split
The publication reports 256 paired reconstruction samples and 251–256 generated samples per mode; it does not publish a reusable train/validation split manifest.
Format
Short text samples, GPT-2 token sequences, discrete code sequences, generation logs, and summary tables.
Version / checksum
A dataset checksum or immutable TinyStories snapshot identifier is not reported in the paper.
Acquisition
A public acquisition script is not released with the publication page.
Use limits
The evidence covers short synthetic stories and should not be treated as a benchmark for unrestricted natural-language generation.

Versions

  1. Published versionIEEE / FRUCT, 2026
  4. Author manuscriptText-accessible local PDF
  5. Conference talkFRUCT 39 presentation
  6. Conference proceedings indexOfficial FRUCT 39 volume
  7. Conference proceedings PDFFRUCT open full text
  8. Open scholarly recordOpenAlex
  9. Citation graph recordSemantic Scholar
  10. Author-shared full textResearchGate

The published DOI is the primary bibliographic identifier. This page remains the single canonical project URL across versions.

Read the Codec bottleneck diagnosis research guide.About the author.

Focused research notes

Intent-specific answers with evidence boundaries and links back to this paper.