How to validate LLM output before it reaches your database

Validating LLM output means checking every model response against a strict schema in code, at the point where it enters your system, and writing only the responses that pass while setting the rest aside with the reason they failed.

OWASP lists the opposite, passing model output on to other components without enough validation, as one of its top risks for LLM applications (LLM05:2025, Improper Output Handling), and recommends treating model output like untrusted user input.

Anyone who turns model output into data: reading invoices or receipts, sorting support tickets, filling a CRM from emails, writing product listings, or letting an agent save records through a tool. If a model response ends up in a database column, this applies to you.

Model output looks right most of the time, and the failures are small: a currency symbol where you expected a code, an amount as the text “1,250.00” instead of a number, a date in another format, a field the model made up, a sentence of preamble before the JSON. Each one either crashes the code that reads it or, worse, gets saved and corrupts the data downstream.

Structured output features narrow the gap without closing it. OpenAI’s docs say Structured Outputs keeps responses to your JSON Schema, while JSON mode only promises valid JSON, and that the values can still contain mistakes. Anthropic’s docs list cases where output may not match the schema at all, a refusal or a response cut off at the max_tokens limit, and say the capitalization of enum values isn’t guaranteed.

Put one gate between the model and everything that stores or acts on its output:

1. Parse. Turn the text into JSON. If that fails, the response is rejected.
2. Validate against a schema. The schema says exactly what is allowed: required fields, types, fixed sets of values, formats. The example uses zod; any validator that returns errors instead of guessing will do.
3. Write only what passed. The parsed, typed value goes to the database, never the raw text.
4. Quarantine the rest with a reason. Keep the raw response and the list of problems, so someone can see what went wrong and retry the record after a fix.

import { z } from 'zod'

/** The only shape allowed past the boundary. Unknown keys are rejected. */
export const InvoiceSchema = z.strictObject({
  vendor: z.string().min(1),
  currency: z.enum(['USD', 'EUR', 'GBP']),
  amountCents: z.number().int().positive(),
  dueDate: z.iso.date(), // YYYY-MM-DD
})
export type Invoice = z.infer<typeof InvoiceSchema>

export type Quarantined = { raw: string; reason: string }
export type Checked =
  | { ok: true; invoice: Invoice }
  | { ok: false; quarantined: Quarantined }

/** Check one model response before anything is written. */
export function checkModelOutput(raw: string): Checked {
  let data: unknown
  try {
    data = JSON.parse(raw)
  } catch {
    return { ok: false, quarantined: { raw, reason: 'not valid JSON' } }
  }

  const result = InvoiceSchema.safeParse(data)
  if (!result.success) {
    const reason = result.error.issues
      .map((issue) => `${issue.path.join('.') || '(root)'}: ${issue.message}`)
      .join('; ')
    return { ok: false, quarantined: { raw, reason } }
  }
  return { ok: true, invoice: result.data }
}

/** Split a batch: valid rows go to the database, the rest to quarantine. */
export function splitBatch(raws: string[]) {
  const valid: Invoice[] = []
  const quarantined: Quarantined[] = []
  for (const raw of raws) {
    const checked = checkModelOutput(raw)
    if (checked.ok) valid.push(checked.invoice)
    else quarantined.push(checked.quarantined)
  }
  return { valid, quarantined }
}

Two choices in the schema do most of the work. z.strictObject rejects keys the schema doesn’t name, where zod’s plain z.object would quietly drop them. And nothing is coerced: an amount sent as text fails instead of being converted, because a converted guess is exactly the kind of quiet error you’re trying to stop.

A schema checks shape, not truth. A well-formed invoice can still carry the wrong amount. Where you can, add checks against data you already trust, such as the vendor’s known currency or the order total, in the same gate.

Extraction code written quickly, by a person or an agent, often has three lines: call the model, JSON.parse the text, insert the result. It passes on the examples you try during development, because those are the inputs the model handles well. The inputs that break it show up later, in production data.

A written schema also gives a coding agent something concrete to test against. Tests can feed known-bad responses through the gate and assert they’re quarantined, which is how the tests for the example above work.

• Prompting harder. Asking for “valid JSON only” lowers the error rate but guarantees nothing, and you still need code for the failures.
• Provider structured outputs. OpenAI’s and Anthropic’s structured output features constrain the response to your schema, which removes most shape errors. Use them, and keep the gate: both providers document cases where output can miss the schema, and neither can check that the values are true.
• Database constraints. NOT NULL and CHECK constraints are a useful last line: Postgres raises an error when a value would violate one. But the write simply fails; you don’t keep the raw response or a readable reason. Note that a CHECK constraint lets a null through unless the column is also NOT NULL.
• Repair and retry. Send the validation errors back to the model and ask again. Fine for a small, fixed number of attempts; past that, quarantine instead of looping (see how to stop an AI agent loop from running up your bill).
• Hand-written checks. A few if statements work for one small object. A schema library gives you types, nested objects and error messages that name the failing field.

1. Find every place a model response is parsed, saved, shown to users or used to call another system.
2. Write one schema per kind of output, as strict as the data allows: enums for fixed sets, whole numbers in the currency’s smallest unit (such as cents) for money, ISO formats for dates.
3. Send every response through one function like checkModelOutput before it reaches the database.
4. Add a quarantine table with the raw text, the reason, the source record and a timestamp, and look at it regularly.
5. Add tests with bad responses you’ve actually seen: a symbol for a currency, text for a number, extra keys, preamble before the JSON.

What to tell your coding agent. Paste this into Claude Code, Codex or Antigravity:

Find every place in this project where model output is parsed or saved. Add one validation gate per kind of output: parse the JSON, validate it with a strict zod schema (z.strictObject, enums for fixed values, integer amounts in the currency’s smallest unit for money, ISO dates, no coercion), and save only the parsed value. Anything that fails goes to a quarantine table with the raw response, the list of validation errors and a timestamp. Never write raw model text to the main tables. Add tests that feed known-bad responses through the gate and assert they are quarantined.

To make it a standing rule instead of a one-off request, put the same text in the instructions file your agent reads at the start of a session: CLAUDE.md for Claude Code, AGENTS.md for Codex, and AGENTS.md, GEMINI.md or a file in .agents/rules/ for Antigravity. ArcLab doesn’t connect to any of these tools; you paste the text yourself.

The Deterministic Validation Gates & Idempotency module in the Systems Foundations track goes further: fail-closed validation, gates between the steps of a pipeline and safe replays, ending with a gate spec you write for your own app. The track’s first module, System Fundamentals, has a scenario built on this exact failure: an invoice reader that sometimes outputs the wrong currency symbol.

The official documentation this guide was checked against:

• LLM05:2025 Improper Output Handling, OWASP Gen AI Security Project: definition; treat model output as untrusted input
• Structured Outputs, OpenAI API docs: schema adherence vs JSON mode; values may still contain mistakes
• Structured outputs, Claude API docs, Anthropic: refusals, max_tokens and enum capitalization
• Basic usage, Zod docs: safeParse and error issues
• Defining schemas, Zod docs: z.object strips unknown keys, z.strictObject rejects them; enums; ISO dates
• Constraints, PostgreSQL 18 docs: violations raise an error; CHECK and null
• How Claude remembers your project, Claude Code docs, Anthropic: CLAUDE.md is read at the start of every session
• AGENTS.md, Codex docs, OpenAI: Codex reads AGENTS.md before doing any work
• Rules, Google Antigravity docs: workspace rules in AGENTS.md, GEMINI.md or .agents/rules/