Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization
A staged diagnosis of compressed short-text generation that separates codec reconstruction loss from latent-generation loss.
Published at 2026 39th Conference of Open Innovations Association (FRUCT)
2026issue 1pp. 69–76Main conference
Author manuscript matching the published text. Author-supplied manuscript for scholarly access; the IEEE/FRUCT publisher page remains the version of record.
Paper in 30 seconds
Problem
In compressed short-text generation, poor decoded text may come from information lost by the codec or weak latent-space generation. End-to-end scores blur those failure modes and can send optimization effort to the wrong component.
Approach
The staged protocol scores originals, paired codec reconstructions, token-space MDLM outputs, and code-space diffusion outputs with one external GPT-2 evaluator. Codebook and geometry measures remain diagnostics rather than substitutes for decoded-text quality.
Main result
Codec reconstruction raises median external perplexity from 15.17 to 27.36 and p95 from 25.10 to 98.91. Code-space MDLM still reduces median perplexity by 30.9% versus token-space MDLM.
Why it matters
The result turns codec bottleneck diagnosis into an actionable order of work: prioritize codec improvements, then compare token-space and code-space generation, and trust latent proxy gains only when decoded text improves too.
Method
The paper uses one scorer across three evaluation stages so that each additional transformation can be associated with a measurable quality gap.
Reconstruct
Encode each 64-token TinyStories sample into 16 top-level codes and decode it immediately to measure codec reconstruction loss.
Generate
Generate either text tokens or discrete latent codes with MDLM, then decode code-space samples through the same trained codec.
Compare
Score originals, reconstructions, and generated texts with the same external GPT-2 protocol, including median and tail statistics.
Key idea
A downstream generator cannot recover information that the codec has already discarded. The reconstruction stage must therefore be audited before interpreting latent-generation results.
Difference from nearby approaches
Standard end-to-end comparisons report one final generation score. This protocol inserts a paired reconstruction checkpoint and separates health metrics in latent space from evidence on decoded text.
What is new
The contribution is a reusable staged diagnostic methodology for one concrete compressed-text pipeline, rather than a new denoising algorithm.
Key results
| Evaluation point | n | Mean PPL | Median PPL | p95 PPL |
|---|---|---|---|---|
| Original texts | 256 | 16.24 | 15.17 | 25.1 |
| Codec reconstructions | 256 | 37.26 | 27.36 | 98.91 |
| AR baseline | 251 | 30.98 | 23.27 | 56.11 |
| Token-space MDLM | 256 | 44.74 | 38.42 | 93.6 |
| Code-space MDLM | 256 | 30.01 | 26.55 | 59.36 |
Key result. Most of the observed quality loss is introduced before generation; code-space diffusion still reduces median perplexity by 30.9% versus token-space diffusion.
- Dataset
- TinyStories
- Sample size
- 256 paired reconstruction samples; 251–256 generated samples per mode; four matched geometry settings.
- Metrics
- External GPT-2 perplexity: mean, median, p95, and maximum; Codebook usage and support size; SBERT, BERTScore, MAUVE, and an LLM-judge summary for geometry runs
- Uncertainty
- The reported comparisons are descriptive single runs; confidence intervals and multi-seed significance estimates were not computed.
- Conditions
- GPT-2 token sequences of length 64 compressed to 16 top-level codes with a hierarchical VQ-VAE-2; all generation modes use the shared external scorer.
Comparison with nearby approaches
| Approach | Representation | Control / diagnosis | What is preserved or measured |
|---|---|---|---|
| Token-space diffusion | Text tokens | Generation quality in token space | Fluency and scorer behavior of direct generation |
| Compressed latent generation | Discrete latent codes via a codec | Final end-to-end generation quality | Combined codec and latent-generator behavior |
| Staged bottleneck localization | Text, codec reconstructions, and discrete latents | Separate codec loss from generation loss | Where quality degrades under one shared scorer |
The comparison distinguishes evaluation scope and evidence; it is not a universal ranking of the approaches.
Abstract
Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes. Without separating these failure modes, researchers can spend compute improving the wrong component. We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). We use a staged validation protocol that separates codec reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer, while reporting complementary semantic metrics for the geometry study. In the tested configuration, codec reconstruction alone raises median external perplexity from 15.17 to 27.36 (+80.4%) and p95 from 25.10 to 98.91 (+294.1%), showing that the dominant quality loss appears before latent generation begins. Under the same scorer, code-space MDLM remains materially stronger than token-space diffusion, reducing mean, median, and p95 by 32.9%, 30.9%, and 36.6%, respectively. Geometry-aware regularization improves local latent proxies but does not improve decoded-text metrics in the available runs. The contribution is methodological rather than algorithmic: the paper presents a reusable staged diagnosis for one concrete pipeline and shows that, in this setting, codec fidelity rather than latent denoising sets the practical quality ceiling.
When to cite this paper
This paper may be relevant when discussing:
- compressed short-text generation and its quality–efficiency trade-offs
- discrete latent text generation or masked diffusion language modeling
- codec reconstruction fidelity in two-stage generative systems
- codec bottleneck diagnosis and staged bottleneck localization
- stage-consistent evaluation of compressed generation pipelines
- comparisons of token-space generation and code-space diffusion
- negative results where latent-space proxy gains do not transfer to decoded-text quality
Limitations
- The empirical study uses TinyStories only.
- The main analysis covers one aggressive 64-to-16 compression regime.
- The evaluated system combines one hierarchical VQ-VAE-2 codec family with one MDLM generator.
- Comparisons are based on single runs and are descriptive rather than multi-seed statistical estimates.
- External GPT-2 perplexity is a shared diagnostic, not a universal semantic-quality metric.
- The conclusions should not be transferred directly to all datasets, codec architectures, or compression ratios.
Related research areas
The staged protocol is useful when a generative pipeline contains both a learned codec and a latent-space generator, but the source of degraded output quality is unclear.
Compressed and discrete-latent text generation
Codec reconstruction fidelity in generative pipelines
Masked diffusion language modeling in code space
Bottleneck localization and stage-consistent evaluation
Auditing latent-space proxy improvements against decoded text
Scope. The evidence comes from one 64-to-16 TinyStories configuration with a hierarchical VQ-VAE-2 and MDLM, using descriptive single-run comparisons. It is not a universal ranking of codec, compression, token-space, or code-space generation methods.
Resources & reproducibility
- Publisher
- IEEE
- Local text PDF
- Author manuscript matching the published text
- Code
- Not publicly released; no inactive Code button is shown.
- Installation
- A public installation recipe is not yet available; no inactive Code button is shown.
- Minimal run
- The published protocol is reconstruct → score → generate → score → compare, but an executable public command is not yet released.
- Configurations
- The paper reports a hierarchical VQ-VAE-2 with 512-code top and lower codebooks and an MDLM using SUBS parameterization.
- Release / commit
- No public release identifier is available.
- Checkpoints
- The publication page does not distribute model checkpoints.
- Expected result
- Reconstruction median PPL 27.36; token-space MDLM 38.42; code-space MDLM 26.55 under the shared external GPT-2 scorer.
- Hardware
- Hardware requirements are not reported in the published paper.
Data statement
- Source
- TinyStories, as described in the paper.
- License
- Use of TinyStories remains subject to the dataset's own terms; no dataset files are redistributed by this site.
- Preprocessing
- GPT-2 tokenization, fixed 64-token inputs, and hierarchical temporal compression from 64 to 32 to 16 positions.
- Split
- The publication reports 256 paired reconstruction samples and 251–256 generated samples per mode; it does not publish a reusable train/validation split manifest.
- Format
- Short text samples, GPT-2 token sequences, discrete code sequences, generation logs, and summary tables.
- Version / checksum
- A dataset checksum or immutable TinyStories snapshot identifier is not reported in the paper.
- Acquisition
- A public acquisition script is not released with the publication page.
- Use limits
- The evidence covers short synthetic stories and should not be treated as a benchmark for unrestricted natural-language generation.
Versions
- Published versionIEEE / FRUCT, 2026
- Author manuscriptText-accessible local PDF
- Conference talkFRUCT 39 presentation
The published DOI is the primary bibliographic identifier. This page remains the single canonical project URL across versions.
Cite this paper
BibTeX is the recommended format. Every variant below is generated from the same publication record.
@inproceedings{Gavrilov2026WhereQuality,
title = {Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization},
author = {Gavrilov, Alexey and Gazzaev, Alan-Barsag and Muravyov, Sergey},
booktitle = {2026 39th Conference of Open Innovations Association (FRUCT)},
publisher = {IEEE},
year = {2026},
pages = {69--76},
doi = {10.23919/FRUCT70069.2026.11506553},
url = {https://doi.org/10.23919/FRUCT70069.2026.11506553},
isbn = {978-952-65246-5-8},
}
Gavrilov, A., Gazzaev, A.-B., and Muravyov, S. (2026). Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization. In 2026 39th Conference of Open Innovations Association (FRUCT) (pp. 69–76). IEEE. https://doi.org/10.23919/FRUCT70069.2026.11506553A. Gavrilov, A.-B. Gazzaev, and S. Muravyov, “Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization,” in 2026 39th Conference of Open Innovations Association (FRUCT), 2026, pp. 69–76, doi: 10.23919/FRUCT70069.2026.11506553.TY - CPAPER
TI - Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization
AU - Gavrilov, Alexey
AU - Gazzaev, Alan-Barsag
AU - Muravyov, Sergey
PY - 2026
DA - 2026-04-28
T2 - 2026 39th Conference of Open Innovations Association (FRUCT)
SP - 69
EP - 76
PB - IEEE
DO - 10.23919/FRUCT70069.2026.11506553
UR - https://doi.org/10.23919/FRUCT70069.2026.11506553
SN - 978-952-65246-5-8
SN - 2305-7254
ER -
Citation files:CITATION.cffAPA textIEEE textRIS