Developers: Check finish_reason Before Validating JSON Against OpenAPI

Before parsing any LLM response as JSON, always read the provider's finish_reason and handle accordingly to avoid unsafe repairs. If the model was truncated, re-run it with more budget instead of patching the JSON, since attempting to repair truncated data can produce fabricated values. If it was refused, log it and route to a fallback. Only after confirming a clean finish should you validate against your OpenAPI schema; on non-strict or streaming paths, only then should a tolerant parser be applied to the output. Provider quirks (nullable fields, oneOf support, depth limits) need their own schema transforms so the same contract doesn't break twice.

TL;DR:

• Only re-try or split requests when finish_reason indicates model truncation; repairing truncated responses leads to fabricated data and schema inconsistencies.
• Use tolerant parsers solely for responses confirmed to have finished cleanly, never for responses cut off mid-object, to prevent masking truncation errors.
• Log and tag every repaired field with a _repaired flag, ensuring auditability and preventing silent inaccuracies in downstream data.
• Adjust JSON Schema with provider-specific fixes, like replacing nullable: true with anyOf and preferring anyOf over oneOf for broader support across validation tools.
• Prioritize schema validation immediately after parsing, then perform semantic validation, and monitor key metrics such as finish_reason distribution and retry rates to detect drift early.

Table of Contents

• How Do You Check Finish_Reason Before Validating JSON?
• Tolerant Parsing and Safe Repair Patterns
• What OpenAPI and Provider Quirks Break Validation?
• How Do You Build a Validate, Retry, and Monitor Loop?
• What Datatool Runs in Production, and Why
• Why Repair-Last Beats Repair-First
• Where Datatool Fits in Your Validate and Repair Pipeline
• Sources
• FAQ

How Do You Check Finish_Reason Before Validating JSON?

Most "broken JSON" bugs aren't JSON bugs. They're truncation bugs wearing a parser error as a disguise. Testing on malformed LLM output keeps landing on the same root cause: teams parse first and check finish_reason never, so a cut-off response looks like corrupt data instead of what it actually is, a request that ran out of room.

Here's the recovery ladder, in order:

1. Check the terminating signal first. OpenAI calls it finish_reason; Anthropic calls it stop_reason. Read it before you touch JSON.parse.
2. If it's length or max_tokens, the model got cut off mid-object. Raise the token budget or split the request into smaller chunks. Never try to repair a truncated payload, because you have no way to know what was supposed to come next, and guessing means inventing data your schema will happily validate as true.
3. If it's a refusal, don't retry blindly. Surface the refusal as an error, route it to a fallback prompt or a human queue, and move on.
4. If it's a clean stop but the JSON still won't parse, that's your actual repair candidate. Re-ask the model with the parse error embedded in the prompt. If that fails too, only then reach for a tolerant parser.

Here's what that looks like in code:

def handle_response(resp):
    reason = resp.finish_reason
    if reason == "length":
        raise RetryWithMoreBudget(resp)
    if reason == "content_filter":
        return route_to_fallback(resp)
    if reason == "stop":
        return validate_and_repair(resp.text)
    raise UnknownFinishReason(reason)

Skip step one, and your repair engine starts "fixing" outputs that were never malformed. They were just cut off. That's the fastest way to ship a JSON blob that's syntactically valid and factually wrong, a problem structural fidelity research shows gets worse, not better, the deeper the nesting goes.

Tolerant Parsing and Safe Repair Patterns

Tolerant parsing should only be used to recover usable objects from JSON outputs that have a confirmed clean finish but contain minor errors; it is not appropriate to use tolerant parsing to fix truncated outputs. A repair-and-recovery playbook for LLM JSON makes the same distinction: reserve tolerant parsers for non-strict paths and streaming, and skip them entirely when a provider SDK already exposes incremental parsed snapshots. Duplicating that work adds a failure mode you don't need.

When you do need it, use it narrowly:

• Use partial-JSON parsing for streaming UIs that render incomplete objects as they arrive, not for finished responses.
• Avoid tolerant parsing on any response that has not been confirmed to have finished cleanly, because it may mask truncation errors and produce incorrect objects.
• When a field is missing after a clean finish, don't invent a value. Surface exactly what's missing, re-ask the model with a minimal, targeted prompt, and tag the field you patched.
• Log the original raw output next to the repaired version, always. Audits need both.

A concrete flow: malformed JSON comes back with a trailing comma and a missing status field. A tolerant parser like jiter or a Python json-repair step fixes the syntax and extracts what parsed cleanly. You diff that against your schema, find status missing, and re-ask with a template like "Return only the status field for this record, using the enum defined in the schema." Once that comes back, you merge it in, tag it repaired: true, and run full validation again before it goes anywhere near production data.

Pro Tip: Never let a repaired field look identical to a model-native one downstream. A _repaired flag in your logs costs nothing and saves hours the day a repair goes wrong.

What OpenAPI and Provider Quirks Break Validation?

A schema that validates fine against your own OpenAPI spec can still get rejected by a provider's structured output mode. JSON Schema is the structural contract of the AI stack, but each provider implements a different subset of it, and strict modes vary enough that a canonical schema built for one provider routinely fails on another.

The fixes are mechanical once you know what to look for:

• Nullable fields: don't write "type": "string", "nullable": true and assume every provider honors it. Rewrite to "type": ["string", "null"], or if the field uses $ref, wrap it in anyOf with an explicit null branch, since a bare nullable flag next to $ref gets dropped by some runtime validators.
• oneOf support: prefer anyOf over oneOf for cross-provider schemas. Several strict modes reject or silently mishandle oneOf, while anyOf is more broadly supported.
• Depth and enum limits: test your real schema against target provider limits (recursion depth, enum size, total properties) before rollout, not after a production 400 tells you.
• Shared source, per-provider output: generate provider-specific JSON Schema variants from one canonical OpenAPI definition rather than hand-maintaining three schemas that drift apart.

A minimal transform for the nullable case:

def make_nullable(schema_node):
    if "$ref" in schema_node:
        return {"anyOf": [schema_node, {"type": "null"}]}
    schema_node["type"] = [schema_node["type"], "null"]
    return schema_node

Learn more about handling schema complexity across providers in Datatool's guide to why complex JSON schemas break AI structured output.

How Do You Build a Validate, Retry, and Monitor Loop?

Schema validation catches structural correctness but cannot detect semantic errors such as mismatched totals or invalid status values; therefore, semantic validation layers (e.g., with Pydantic or Zod) should be applied after schema validation to enforce business rules.

The operational loop:

1. Run an assertValidstyle runtime check at the call site, narrowing the parsed type only after validation passes.
2. On failure, embed the validator's actual error message into a retry prompt. Cap retries at one to three attempts. Beyond that you're burning latency and cost for diminishing returns.
3. After schema validation passes, run semantic checks: does total equal the sum of line items, does end_date come after start_date, does a foreign key actually exist.
4. Track finish_reason distribution, parse failure rate, schema validation failure rate, and retry rate. Alert on spikes in any of them, since a jump in retries almost always means a prompt or schema drifted upstream.

async function assertValidWithRetry(raw, schema, maxRetries = 2) {
  let attempt = 0;
  while (attempt <= maxRetries) {
    const result = ajv.validate(schema, raw);
    if (result) return raw;
    raw = await reask(ajv.errors, raw);
    attempt++;
  }
  throw new ValidationExhausted(ajv.errors);
}

Teams that skip the monitoring layer tend to find out about a schema drift from a customer ticket instead of a dashboard, which is the expensive way to learn.

What Datatool Runs in Production, and Why

Datatool's role in AI pipelines is deterministic repair plus schema validation for structured data that comes out of an LLM broken, wrapped, truncated, or half-escaped. It's built to sit downstream of the finish_reason check, not to replace it.

The rules that hold up under repeated testing:

• Check finish_reason first. Repair is the last step, never the first.
• Tag every repaired field so downstream systems and audits know what the model produced versus what a repair pass filled in.
• Run semantic validators after schema validation passes. A syntactically valid object can still be wrong.
• Re-validate against the OpenAPI schema after every repair, not just once at the end of the pipeline.

Gregory D. Collins covers JSON repair, schema validation, and LLM output reliability for Datatool, focused on the failure patterns that show up once teams move past prototype prompts into production traffic.

Why Repair-Last Beats Repair-First

The instinct to reach for a tolerant parser the moment JSON fails to parse is understandable and usually wrong. It treats every malformed response as the same problem, when truncation, refusal, and genuine syntax errors need three different responses, and only one of them is a repair job.

Conventional advice on this topic leans too hard on "just use a JSON repair library" as a universal fix. That advice skips the branch that matters most: reading finish_reason before you decide anything else. A repair library applied to a truncated response doesn't recover data, it fabricates a plausible ending for an object that was never finished, and your schema validator will approve it without complaint because syntactic validity says nothing about truth.

What the evidence actually supports is a strict ordering. Check the finish signal first. Re-run or chunk on truncation. Route refusals to a fallback path. Reserve tolerant parsing and repair for the narrow case of a clean finish that still produced malformed JSON, and tag every repair so it's auditable later. Structure errors are a distinct failure class from content errors, and they need distinct handling, not a single repair step bolted onto the end of every pipeline. Get the ordering right and repair becomes a rare, well logged event instead of a constant background process masking real upstream problems.

— Gregory

Where Datatool Fits in Your Validate and Repair Pipeline

Datatool is the layer you drop in after finish_reason handling, right where most teams currently have a brittle try/catch around JSON.parse. It runs deterministic repair on malformed, wrapped, or partial JSON, validates the result against your OpenAPI or JSON Schema contract, and gives you a test harness to check that repair logic against real failure samples instead of hoping it works in production.

It plugs in at two points: right after parse failure detection, so a clean-finish response that fails to parse gets repaired instead of thrown away, and inside your test suite, so schema drift shows up in CI instead of a customer bug report. For teams standardizing on Pydantic or Zod for semantic checks, Datatool handles the structural repair and validation layer underneath, so your semantic validators only ever see JSON that's already structurally sound. Start with the Datatool product page to try the repair engine against your own malformed samples and see what it catches before you build that logic yourself.

Sources

• When Structured Output Breaks: Structure-Content Decomposition and SA-RLVR (arXiv)
• Structured Outputs Are Not a Solved Problem: JSON Mode Failure Modes in Production
• The 200 that lies: runtime schema validation with Ajv on top of generated types – Cosmin Pop

FAQ

What Does It Mean to Validate JSON Against OpenAPI?

For LLM-generated data, it means confirming the model's output matches your OpenAPI schema's structure and types after a clean finish, then layering semantic checks like Pydantic or Zod on top for business rules the schema can't express.

Should I Repair Truncated JSON or Re-Run the Request?

Re-run it. Truncation, signaled by a finish_reason of length or max_tokens, means data is missing, and repairing a truncated object means inventing values instead of recovering them.

When Is a Tolerant JSON Parser Actually Safe to Use?

Only after confirming a clean finish. Tolerant parsers belong on non-strict output paths and streaming UIs, not on responses that never finished generating.

How Do I Handle Nullable Fields Across Different Providers?

Rewrite nullable: true next to a $ref into an explicit anyOf with a null branch, since some validators drop a bare nullable flag on referenced schemas.

What Should I Monitor After Deploying a Validate and Repair Pipeline?

Track finish_reason distribution, parse failure rate, schema validation failure rate, and retry rate, and alert when any of them spikes, since that usually signals prompt or schema drift.

Recommended

• JSON Schema Validation: A Practical Guide for Developers
• JSON Input: Parse, Validate, and Repair It Right
• Ship Reliable LLM Output Validation: 5 Developer First Controls
• Why AI Returns Extra Text in JSON: A Dev Guide