Stop Parsing Raw JSON: Type-Safe LLM Output Pipelines with BAML

Picture the failure that actually costs you a customer. Your product extracts purchase orders from inbound email and writes them into the customer's ERP. For three weeks it works. Then a model update, or just an unlucky sampling run, wraps the JSON in a Markdown code fence, or adds a chatty preamble, or renames total_amount to totalAmount. Your json.loads() call throws, or worse, it succeeds and a hallucinated key slides into a database mutation. The feature did not get dumber. Your parser was always the weakest link, and it finally broke where a buyer could see it.

This guide is about removing that failure class structurally rather than patching it per incident. The tool we examine is BAML by BoundaryML: a small domain-specific language that treats every LLM call as a typed function with a schema contract, then generates native client code for your application language. We verified every claim below against BoundaryML's own repository and documentation; where a number is BoundaryML's benchmark rather than ours, we say so.

The Real Business Bottleneck: Reliability You Cannot Price#

For a founder putting an LLM inside a transactional workflow (CRM updates, invoice extraction, SQL generation, order routing), the constraint is rarely model quality. It is that the pipe between the model and your database is held together with string parsing.

Reliability. A model that produces valid, schema-conforming output 97% of the time sounds fine until you multiply. At 10,000 extractions a day, 3% is 300 daily failures that either page your team or silently corrupt records. And the failures are not random noise you can average away: they cluster around exactly the inputs your customers care about most, the long messy documents.

Cost. The standard remedy is retry-on-parse-failure. Every retry doubles the token bill for that request, and because failures cluster on the longest inputs, retries are systematically your most expensive calls. Teams also reach for a bigger model purely to get cleaner JSON, paying frontier prices to solve a formatting problem.

Latency. Retries also double tail latency, and strict provider-side JSON modes add their own constraint: you wait for the full response before you can parse anything. For a user watching a form populate, "typed and streaming" versus "blank screen for twelve seconds" is the difference between a demo that closes and one that doesn't.

The pattern to notice: all three costs come from the same root, which is that the output contract lives in prose inside your prompt instead of in code your compiler can check.

Why the Naive Fixes Fail#

Regex and json.loads() with a try/except. This treats malformed output as an exception. In production it is not exceptional, it is a steady-state percentage. Each new failure shape (code fences, trailing commas, single quotes, "Here is your JSON:") gets its own patch, and the patches live in application code where the next engineer will not find them.

Pasting a JSON Schema into the prompt. Better, but the schema and your application types now live in two places. When a field changes, someone must update the Pydantic model, the TypeScript interface, and the prompt text in lockstep. Nothing enforces this. The drift is invisible until it is a runtime error, which is the exact class of bug type systems were invented to end.

Relying only on provider JSON modes. Constrained decoding from the provider is real progress, and for many single-provider workloads it is enough. But it binds your output contract to one vendor's implementation and feature set. The moment you add a fallback provider or an open-weights model for cost reasons, you are back to hand-parsing for the second path, and your reliability story is only as good as your least reliable branch.

Retry loops as a strategy. Retrying does convert some failures into successes. It also converts your error rate into a cost and latency multiplier while leaving the underlying fragility untouched. A retry loop is a tax, not a fix.

What all four have in common: they defend against malformed output after the fact, in scattered application code, instead of defining the contract once and making everything (prompt, parser, client types) derive from it.

Production Architecture: The Schema as the Contract#

BAML inverts the usual setup. Instead of a prompt string that mentions a schema, you write a typed function, and the prompt, the output-format instructions, the parser, and the native client library are all generated from it. BoundaryML's repository describes the design goal plainly: "Types persist at runtime. There is no any nor casting dangerously to any type." The same function is callable from multiple languages; the README states you can "call a BAML function from TS, Py, Go, C#, Java, etc."

1. Define the contract once#

// invoice.baml
class LineItem {
  description string
  quantity int
  unit_price float
}

class Invoice {
  vendor_name string
  invoice_number string
  currency string @description("ISO 4217 code, e.g. USD")
  total float
  due_date string @description("ISO 8601 date")
  line_items LineItem[]
}

function ExtractInvoice(email_body: string) -> Invoice {
  client ExtractionClient
  prompt #"
    Extract the invoice from this email.

    {{ ctx.output_format }}

    {{ _.role("user") }}
    {{ email_body }}
  "#
}

{{ ctx.output_format }} is where BAML injects the schema instructions into the prompt, so the prompt can never drift from the type. The BAML compiler turns this file into generated client code, and your application call site is one typed line:

from baml_client import b

invoice = b.ExtractInvoice(email_body)
# invoice is a typed Invoice object. invoice.total is a float.
# A missing or malformed field fails HERE, loudly - not in your DB write.

There is no json.loads() anywhere in your codebase for this path. That is the point: the parsing risk is centralized in a component built for it, instead of distributed across every call site.

2. Let the parser absorb model mistakes#

The component doing that work is BoundaryML's Schema-Aligned Parsing (SAP). Their engineering write-up describes the philosophy as "assume that the model will make mistakes, and build a parser that is robust enough to handle them": it strips prefix and suffix chatter, repairs missing quotes, commas, and brackets, converts single values to arrays where the schema expects a list, and resolves the result against your declared types, in the spirit of an edit-distance computation toward the nearest schema-valid object.

On BoundaryML's own runs of the Berkeley Function Calling Leaderboard (n=1,000 per model), SAP scored 92% with GPT-3.5-turbo, 93% with GPT-4o, 91.7% with Claude 3 Haiku, and 94.4% with Claude 3.5 Sonnet, outperforming native function calling in their comparison. Two honest caveats belong next to that: these are the vendor's benchmark numbers, not an Effloow measurement, and the tested models are past-generation. What the numbers demonstrate is the mechanism, not a guarantee for your workload: a schema-aware parser lets mid-tier models hit reliability levels teams normally buy frontier models for. That is a pricing lever, and it composes with routing: cheap model plus robust parser first, escalate only on genuine failure. If you already route traffic through a gateway, this slots in front of the gateway layer we covered previously.

3. Make failover a config line, not an incident#

Provider outages and rate limits are the other reliability plane. In BAML, resilience policy is declared next to the client, out of application code:

retry_policy TwoAttempts {
  max_retries 2
  strategy { type exponential_backoff }
}

client<llm> ExtractionClient {
  provider fallback
  options {
    strategy ["openai/gpt-5", "anthropic/claude-opus-4-1-20250805"]
  }
}

The fallback provider tries the strategy list in order; a round-robin provider exists for load balancing. Because SAP sits under every branch, both providers flow through the same schema contract, which is what makes multi-provider setups practical without maintaining two parsers.

4. Stream typed partials instead of waiting#

BAML generates partial_types for every class: while tokens arrive, "BAML will convert all Class fields into nullable fields, and fill those fields with non-null values as much as possible given the tokens received so far" (their streaming docs). Annotations give you semantic control, and this is where business rules meet streaming: @stream.done holds a field back until it is complete, and @stream.not_null holds the whole object until a critical field exists.

class Invoice {
  vendor_name string
  total float @stream.done   // never show a half-streamed money amount
  // ...
}

Your UI can populate line items live while the total renders only when final. That converts the latency cost of structured output from "wait for everything" into perceived responsiveness, without giving up type safety mid-stream.

5. Test prompts like code#

BAML ships a VS Code playground and a built-in test framework, so schema changes are exercised against fixture inputs locally, before deployment, with no live API call. The compile-test loop runs entirely on your machine; there is nothing to mock because the contract is code. If your team currently builds provider schemas by hand, our free function calling schema builder and JSON-to-types generator cover the adjacent chores, and the BAML approach is the natural next step once those files multiply.

What This Is Worth: The Founder's Math#

You do not need our benchmark to compute your own ROI; you need four numbers you already have. Daily structured-output calls, current parse-failure rate, average cost per call, and the loaded cost of an engineer-hour spent on extraction bugs.

Work one plausible line of it: at 10,000 calls/day, a 3% failure rate handled by one retry each adds ~300 duplicate calls daily. At even $0.01 per call, that is ~$1,100/year in pure retry spend, which is the small line. The large lines are the engineer-hours spent per malformed-output incident, and every bad record that reached a customer system. If failures currently force you onto a frontier model for formatting reasons alone, the delta between frontier and mid-tier pricing across your entire structured-output volume is the biggest number on the page, and it is exactly the number a schema-robust parser puts in play (it stacks with the prompt-side savings from our token optimization guide). Plug in your own rates; if the result is under an engineer-day per quarter, the naive approach is genuinely fine for you.

Can this survive your workflow? Ask three questions before adopting. Does structured output sit on a revenue or data-integrity path, so a parse failure is a customer-visible defect rather than a logline? Do you have (or want) more than one model behind the feature? Does at least one schema change per month ripple across prompt, types, and parser today? Two or more yeses and the contract-first pattern pays for its learning curve. Zero and it will not.

When to Use, When to Skip#

Use BAML (or the contract-first pattern generally) when: LLM output mutates databases or downstream systems; you run or plan multi-provider fallback; multiple services in different languages consume the same extraction; you stream structured data to a UI; schema churn is regular.

Skip it when: your LLM output is prose for humans, not data for machines; you have one provider, one language, and its native structured-output mode already holds at your volume; your team will not absorb a DSL and code-generation step for a single low-stakes endpoint; or you are in an exploratory phase where the schema changes hourly and any contract is premature.

Known limitations, stated plainly: BAML is a new language surface your team must learn, with its own compiler in your build chain. Schema-aligned parsing repairs format, not truth; a wrong-but-well-formed value passes the parser, so validation rules and human review still own semantic correctness. And the SAP benchmark figures above are BoundaryML's own; treat them as the vendor's evidence until you reproduce them on your traffic.

For Your Engineers#

Verified source notes: BAML is Apache-2.0, at 9.1k GitHub stars as of this writing (2026-08). Generated clients cover TypeScript, Python, Go, C#, Java "etc." per the README; check the docs for your exact target before committing. Runtime client swapping is available via ClientRegistry (add_llm_client / set_primary) for per-tenant or per-environment model selection without redeploying .baml files. Retry policies are named declarations referenced from clients; fallback and round-robin are composable client providers. Streaming emits partial_types with @stream.done, @stream.not_null, and @stream.with_state for completion metadata. Start with one endpoint: pick your highest-failure extraction, port its schema to a .baml class, and run your last month of failed raw outputs through the generated parser as fixtures. That single test tells you what SAP would have saved you, on your data, before you adopt anything.

What Effloow added: we verified BAML's claims against five primary sources (the repository, docs, the SAP engineering post, the client-registry and streaming references), separated vendor benchmark numbers from reproducible mechanisms, and framed the adoption decision as a four-number ROI calculation with explicit skip criteria, rather than a feature tour.

If structured output sits between your model and your customers' data, this is exactly the class of architecture Effloow builds and hardens for B2B teams: contract-first pipelines, provider failover, and the evidence to show your buyers it holds. See what we do at /services, or tell us about your pipeline and we'll tell you where it will break first.