Structured Outputs: Getting Reliable JSON From Language Models
TL;DR
- Free-text model output breaks downstream code in ways that are hard to catch in testing and expensive to catch in production, so the fix is to constrain the output format, not to write smarter parsing.
- Native structured outputs that enforce a JSON Schema are the most reliable path when your provider supports them; function calling and prompt-and-parse are fallbacks with real trade-offs you should pick on purpose.
- Keep schemas small and strongly typed, validate every response against the schema, and retry with the validation error fed back in - that three-part loop is what makes the system dependable.
Why free-text output from an LLM keeps breaking your code
Here's the failure we see most often when a team first wires a language model into a real product. The prototype works. The demo works. Then it ships, and three days later something downstream throws an exception nobody can reproduce. The model returned a sentence before the JSON. Or it wrapped the JSON in a markdown code fence. Or it added a field you didn't ask for. Your parser, which was written against the happy path, falls over.
The root problem is that a chat-tuned model is built to produce helpful prose, not machine-readable data. When you ask it for JSON in the prompt, you're making a polite request, not setting a hard constraint. Most of the time it complies. The trouble is the small fraction of the time it doesn't - and at any real volume, a 2% failure rate is not rare, it's a steady stream of broken records.
What makes this class of bug nasty is the timing. You can't catch it reliably in testing because the model is non-deterministic, so the exact malformed shape that breaks you in production may never appear in your test runs. The cost lands later, in support tickets and silent data corruption, which is the worst place for it to land.
- Extra prose before or after the JSON - 'Sure, here is the data you requested:' followed by the object.
- Markdown fences wrapping the payload, so a naive JSON.parse on the raw string fails immediately.
- Trailing commas, single quotes, or unescaped characters that are valid in the model's mental model of text but not in the JSON spec.
- Fields that drift in name or type between calls, so the same prompt returns 'price' one time and 'cost' the next.
- Truncated output when the response hits a token limit mid-object, leaving you with half a JSON tree.
What structured output actually means for an LLM
Structured output is the practice of forcing the model's response into a predefined, machine-readable shape - almost always JSON - rather than letting it free-form. The key word is forcing. There's a real difference between asking for JSON and constraining the generation so that nothing but valid JSON can come out.
Under the hood, the strongest versions of this work at the decoding level. As the model generates each token, the runtime restricts the set of allowed next tokens to only those that keep the output valid against a grammar or schema. If the schema says the next thing must be a closing brace or a comma, the model literally cannot emit a word of prose there. That's why constrained decoding is categorically more reliable than any amount of prompt wording.
The practical upshot is that you move the guarantee from 'the model usually behaves' to 'the format is enforced by the system'. You still own correctness of the values - the model can put a wrong number in a valid field - but you stop fighting the format. That single shift removes the entire category of parse-failure bugs, which is usually the noisiest one.
- Structured output enforces the shape of the response - keys, types, nesting - so your code can rely on it.
- It does not guarantee the content is correct; a well-formed object can still hold a hallucinated value.
- Constrained decoding restricts allowable tokens during generation, which is stronger than instructions in the prompt.
- Most providers expose this as a response-format setting that takes a JSON Schema you define up front.
- Once the format is guaranteed, your validation layer can focus on business rules instead of syntax.
Native structured outputs vs JSON mode vs function calling vs prompt-and-parse
There are four common ways to get JSON out of a model, and they're not equivalent. Teams get into trouble by reaching for the easiest one and then discovering its limits at scale. Pick the method on purpose, based on what guarantee you need.
Native structured outputs are the strongest. You hand the provider a JSON Schema, and the runtime guarantees the response conforms to it. This is the default choice when your provider and model support it. JSON mode is a weaker cousin - it guarantees you get syntactically valid JSON, but not that the JSON matches your schema. It'll parse, but the fields might be wrong, so you still need full validation on top.
Function calling, or tool calling, was the original mechanism for getting typed arguments out of a model, and it's still excellent when the JSON represents a call to something - an action with named parameters. The model returns arguments shaped to your function signature. Many teams use it purely as a structured-output trick even when there's no real function, which works but is a slightly awkward fit. Prompt-and-parse - just asking for JSON in plain language and parsing the string - is the weakest and the one to avoid for anything that matters, though it's sometimes the only option with an older model or a constrained API.
- Native structured outputs: schema enforced by the runtime. Use this whenever it's available - it's the highest guarantee.
- JSON mode: valid JSON guaranteed, schema not enforced. Acceptable only with a strict validation layer behind it.
- Function calling: typed arguments against a function signature. Best when the output genuinely maps to an action.
- Prompt-and-parse: ask in the prompt, parse the string. Lowest reliability; treat as a last resort with heavy retries.
- Where you can, prefer the method that pushes the guarantee into the system rather than into your prompt wording.
How JSON Schema constraints make the output predictable
A JSON Schema is the contract. It describes the exact structure you expect - what keys exist, what type each value is, which fields are required, what values an enum is allowed to take. When you pass that schema to a model that supports native structured outputs, you're not suggesting a shape, you're defining the only shape the response can have.
The schema features that earn their keep in production are the strict ones. Mark every field you depend on as required, because optional fields invite the model to omit them and your code to assume they're there. Set additionalProperties to false so the model can't invent extra keys. Use enums for any field with a fixed set of valid values - status, category, priority - because an enum turns a free-text guess into a bounded choice. And type your numbers as numbers, not strings, so you're not parsing '42' downstream.
One thing worth saying plainly: not every schema feature is supported by every provider's structured-output implementation. Some support a useful subset of JSON Schema, and they'll reject or quietly ignore the parts they don't handle. So check what your provider supports before you build a schema that leans on a feature it'll drop. We've seen a regex pattern constraint silently do nothing because the runtime didn't enforce it - the schema looked safe and wasn't.
- Mark dependent fields as required so the model can't legally omit them.
- Set additionalProperties to false to block invented or drifting keys.
- Use enums for any field with a known, fixed set of valid values to stop enum drift.
- Type values precisely - numbers as numbers, booleans as booleans - so downstream code skips coercion.
- Confirm which JSON Schema features your provider's structured-output mode actually enforces before relying on them.
Validate every response and retry when it fails
Even with native structured outputs, you validate. This isn't belt-and-braces paranoia - it's the layer that catches the cases the format guarantee doesn't cover, and it's the layer you fall back on entirely when you're stuck with JSON mode or prompt-and-parse. Parse the response, then check it against your schema with a real validation library, the same kind you'd use for an external API payload. If it passes, proceed. If it fails, you have a clear, structured error.
The part most teams skip is the retry, and it's the part that turns a flaky pipeline into a dependable one. When validation fails, don't just log and drop the record. Send the request again - but include the previous bad output and the specific validation error in the new prompt. Telling the model exactly what was wrong ('the priority field returned high-priority but must be one of low, medium, high') gets a correct response on the second try far more often than a blind retry does.
Cap the retries. Two or three attempts is the sensible band - past that you're usually hitting a genuine problem with the prompt or the schema, not a transient blip, and looping forever just burns tokens and time. After the cap, route the record to a dead-letter queue or a human review step so it's visible, not silently lost. The highest-impact place to start hardening an AI pipeline is right here: wrap your model call in a validate-and-retry loop before you optimise anything else, because it removes the largest source of production failures for the least effort.
- Always parse and validate against the schema, even when the provider claims to enforce it.
- On failure, retry with the previous output and the exact validation error included in the prompt.
- Cap retries at two or three; beyond that the problem is structural, not transient.
- Send exhausted records to a dead-letter queue or human review - never silently discard them.
- Log validation failures with the raw output so you can see which schema fields cause the most trouble.
The common failure modes and how to design them out
Once you've run structured outputs at volume, the failures fall into a small set of recognisable patterns. Knowing them lets you design the schema and prompt to prevent them rather than patch them after the fact. None of these are exotic - they're the same handful, over and over, across every project.
Extra prose and code fences are the classic prompt-and-parse failures, and constrained decoding kills them outright. Trailing commas and quoting issues are syntax-level and also vanish with native structured outputs or JSON mode. Hallucinated fields - the model adding keys you didn't ask for - are stopped by additionalProperties set to false. Enum drift, where a status field returns a near-miss value like 'in-progress' instead of 'in_progress', is solved by defining the enum in the schema so the model can only pick from the real set.
The one that needs more than a schema is truncation. When the response runs out of token budget mid-object, you get valid-so-far JSON that's incomplete. The fix is partly operational - set a generous enough max-tokens for the expected output size - and partly structural, by keeping the output small so it can't balloon past the limit. If you genuinely need large structured output, paginate or chunk the work rather than asking for one giant object.
- Extra prose and markdown fences: eliminated by constrained decoding or native structured outputs.
- Trailing commas and bad quoting: gone once the format is enforced rather than requested.
- Hallucinated fields: blocked by additionalProperties false plus a required-field list.
- Enum drift: prevented by declaring enums in the schema so only valid values can be generated.
- Truncation: handled with an adequate token budget and by keeping each response small enough to finish.
Keep schemas small, flat, and strongly typed
Big schemas are where reliability quietly erodes. The instinct is to define one rich schema that captures everything you might ever want from a document in a single call. It feels efficient. In practice, the more fields and the deeper the nesting, the more places the model has to get something subtly wrong, and the harder your validation errors are to act on.
We push teams toward small, flat schemas - extract a focused set of fields per call, prefer a shallow object over five levels of nesting, and split a complex extraction into two or three narrow calls rather than one sprawling one. Narrow schemas also produce sharper retries, because when something fails you know exactly which small thing to correct. A flat shape is easier to validate, easier to log, and easier to evolve when requirements change.
Strong typing is the other half. Every field should have a precise type and, where it applies, a tight constraint - an enum, a number range, a required marker. Loose typing pushes the cleanup work downstream into your code and reintroduces the very fragility you adopted structured outputs to remove. The goal is that by the time a validated object reaches your business logic, there's nothing left to guess about its shape.
- Extract a focused field set per call instead of one schema that tries to capture everything.
- Prefer shallow, flat objects over deep nesting - fewer places for the model to go subtly wrong.
- Split complex extractions into two or three narrow calls for sharper, more correctable failures.
- Give every field a precise type and a tight constraint so cleanup doesn't leak into your code.
- Treat the schema as a living contract you can version as requirements change.
A practical path: where to start with structured outputs
If you're adding structured outputs to a system that's currently parsing free text, you don't need to rebuild everything at once. The order of operations matters more than the tooling. Start by writing down the schema you actually need - the smallest set of typed fields the downstream code depends on. That single artifact clarifies most of the rest.
Then switch your model call to native structured outputs if your provider supports them, falling back to function calling or JSON mode if not. Wrap the call in the validate-and-retry loop from earlier, with the validation error fed back on retry and a hard cap. Add logging on every failure so you can see, within a week of real traffic, which fields cause trouble. That feedback is what tells you where to tighten the schema or reword the prompt.
The pattern that holds up: enforce the format at the system level, validate every response, retry intelligently when it fails, and keep the schema small enough that failures are rare and easy to fix. Get those four things in place and the steady stream of broken records stops being a thing you firefight. Start with the schema and the retry loop - they're the cheapest changes with the biggest drop in production failures, and everything else builds on them.
- Write the smallest typed schema your downstream code genuinely needs before changing any model code.
- Switch to native structured outputs where supported; fall back to function calling or JSON mode otherwise.
- Wrap the call in validate-and-retry with error feedback and a hard cap on attempts.
- Log every validation failure with raw output so the worst fields surface within a week of real traffic.
- Iterate on the schema and prompt from that data rather than guessing at what to harden.
Want this built for your business?
We map the highest-leverage place to start and ship a first live system within two weeks.
Book a strategy callCommon questions
What is structured output from a language model?
It's forcing a model's response into a predefined machine-readable shape, almost always JSON, instead of free-form text. The strongest versions constrain generation at the decoding level so the output is guaranteed to match a schema, which removes the entire class of parse-failure bugs that plague prompt-and-parse pipelines.
What's the difference between JSON mode and structured outputs?
JSON mode guarantees you get syntactically valid JSON but not that it matches your schema - the fields and types can still be wrong. Native structured outputs enforce a JSON Schema you define, so both the syntax and the shape are guaranteed. Prefer structured outputs when available, and always validate behind JSON mode.
Should I use function calling or structured outputs for JSON?
Use function calling when the JSON genuinely represents a call to something - an action with named parameters mapped to a function signature. Use native structured outputs when you just need typed data back. Many teams use function calling as a structured-output trick, which works, but native structured outputs are a cleaner fit for pure data extraction.
How do I handle a model returning invalid JSON?
Validate every response against your schema, and on failure retry with the previous bad output and the exact validation error included in the prompt. Cap retries at two or three, then route the record to a dead-letter queue or human review. Constrained decoding with native structured outputs prevents most invalid JSON in the first place.
Why does my LLM keep adding extra fields or wrong enum values?
Extra fields come from not setting additionalProperties to false in your schema, which leaves the model free to invent keys. Wrong enum values, like 'in-progress' instead of 'in_progress', come from not declaring the enum in the schema. Define both constraints explicitly and the model can only return the values you allow.
It's forcing a model's response into a predefined machine-readable shape, almost always JSON, instead of free-form text. The strongest versions constrain generation at the decoding level so the output is guaranteed to match a schema, which removes the entire class of parse-failure bugs that plague prompt-and-parse pipelines.
Ask AI about X18 Global
“What does X18 Global (x18global.com) do for enterprise AI and automation - and can you summarise their guide "Structured Outputs: Getting Reliable JSON From Language Models"?”