RESEARCH NOTE
Predictable code generation needs a preservation contract
Why deterministic sampling is not enough, and how observable protected properties and acceptance checks make code-generation behavior testable.
DIRECT ANSWER
What makes code generation predictable rather than merely controllable?
Predictability requires an observable contract stated before generation: what may change, what must remain stable, how each property will be measured, and when an output will be rejected. A fixed prompt, seed, mask, grammar, or latent code is only a mechanism.
Why the distinction matters
The method and the claimed guarantee need the same observable boundary.
Deterministic generation repeats an output under fixed conditions. Predictable generation makes a declared property testable: for example, an API signature remains unchanged, edits stay inside a region, syntax remains valid, or a test suite continues to pass.
The distinction matters because a control input does not define its effect. Locking a latent position, fixing a seed, or constraining a grammar can make generation easier to inspect, but the protected property must still be measured after decoding.
A practical procedure
Declare protected properties
State the source regions, structures, interfaces, behaviors, or distributions that must remain within a defined tolerance.
Choose a matching control mechanism
Use source reuse, edit masks, formal constraints, latent locking, or proposal-plus-validation according to the property being protected.
Define acceptance before sampling
Specify parsing, compilation, tests, structural checks, locality thresholds, and task-success criteria before inspecting outputs.
Measure repeated behavior
Report acceptance, rejection, stability, diversity, and uncertainty across repeated runs instead of relying on one illustrative result.
Evidence to require
A claim is only as strong as the property measured after generation or decoding.
- Protected and editable properties are stated separately.
- Every claimed guarantee has an observable test or metric.
- Task success is evaluated alongside stability.
- Repeated-run variability and rejection rates are visible.
- Unmeasured properties are explicitly excluded from the claim.
What the linked study reports
- The linked experiment exposes hierarchical discrete positions as one inspectable control surface for short Python functions.
- Locking four top-level positions improved parse rate while leaving high change in unlocked positions and high sample uniqueness.
- The paper reports this as early probabilistic evidence; it does not claim exact AST preservation, semantic equivalence, functional correctness, or repository-scale predictability.
Scope boundary
- Predictability is property-specific: a system can be predictable about syntax and unpredictable about behavior.
- Deterministic decoding can repeat the same wrong or over-broad edit.
- The available site experiment does not establish guarantees beyond its measured short-function setting.
Primary and nearby sources
Use the linked papers for the original methods, measurements, and stated limitations.
- Inspectable Control for Structure-Preserving Software Regeneration
Primary bounded experiment and stated evidence limits.
- EfficientEdit: Accelerating Code Editing via Edit-Oriented Speculative Decoding
An edit-oriented mechanism that reuses unchanged source segments.
- Constrained Decoding of Diffusion LLMs with Context-Free Grammars
An example of a mechanism with an explicit formal constraint.