Constrained Decoding (Grammar-Guided Generation)
Constrained decoding is an inference-time technique that enforces formal structural rules—such as JSON schemas, regular expressions, or context-free grammars—by compiling the target specification into a finite-state machine or pushdown automaton that masks invalid token logits at every generation step. Engines such as Outlines and XGrammar precompute token-level transition masks so that the model can only sample syntactically valid continuations with minimal runtime overhead. It guarantees syntactic and schema adherence, but it does not guarantee semantic accuracy: a model can still emit a schema-valid object populated with hallucinated or logically inconsistent values.
Origin and context
Early structured generation relied on prompt instructions and post-hoc retry loops, or naively evaluated regular expressions across the entire 128K-token vocabulary at every decoding step, adding severe latency. In July 2023, Brandon Willard and Rémi Louf introduced Outlines, demonstrating that indexing a vocabulary against a deterministic finite-state automaton (FSM) allows O(1) lookup of valid next-token bitmasks. In late 2024, researchers at CMU and OctoAI introduced XGrammar, splitting context-free grammar tokens into context-independent precomputed masks and a lightweight pushdown automaton for recursive structures, achieving near-zero overhead inside vLLM, SGLang, and TensorRT-LLM.
Why it matters
Agentic workflows, tool calling, and data-extraction pipelines break when an LLM wraps JSON in unexpected markdown prose, omits a required closing brace, or invents an enum value. Constrained decoding eliminates parsing exceptions and retry loops at the sampling layer, making structured outputs deterministic at the syntax level even for smaller open-weight models.
Example
An extraction service defines a Pydantic model requiring an ISO date, a three-letter currency code enum, and a list of line items. When the request hits vLLM with `response_format` set to that JSON schema, XGrammar compiles the schema into a byte-level automaton, overlaps mask computation on the CPU with the GPU forward pass, and sets the logits of all schema-violating vocabulary tokens to negative infinity before softmax sampling.
Maturity and evidence
Limits and open questions
Forcing a model onto a rigid syntactic track can degrade reasoning quality if the schema prevents the model from thinking before committing to a field value, or if tokenization boundaries split valid strings in unnatural ways. Furthermore, schema validity is not factual validity: constrained decoding cannot stop a model from placing a fabricated number inside a syntactically valid float field.
Related terms
References
- Efficient Guided Generation for Large Language ModelsWillard & Louf / Outlines (arXiv) · 2023-07-18 · class A
- XGrammar: Flexible and Efficient Structured Generation Engine for Large Language ModelsDong et al. / Carnegie Mellon University (arXiv) · 2024-11-22 · class A
- Tam et al.: Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language ModelsTam et al. / EMNLP 2024 Industry Track (arXiv) · 2024-08-05 · class A
Last updated: 2026-10-19
This term is also covered in the Skills Atlas as structured llm outputs skill.
This term is also covered in the Skills Atlas as llm inference serving skill.
This term is also covered in the Skills Atlas as llm decoding strategies skill.
This term is also covered in the Skills Atlas as llm function calling skill.
Referencing this analysis in documentation, research, or an engineering write-up? Copy a formatted link or citation below:
Show citation text & canonical URL
[Constrained Decoding (Grammar-Guided Generation) — AI Glossary — Skills Intelligence](https://www.skillsintelligence.tools/glossary/term/constrained-decoding)Skills Intelligence. (2026). Constrained Decoding (Grammar-Guided Generation) — AI Glossary. Skills Intelligence AI Glossary. Retrieved from https://www.skillsintelligence.tools/glossary/term/constrained-decoding