How to Choose an LLM for Production AI Workflows in 2026

A new flagship model lands every few weeks. Each one tops a leaderboard, and each one triggers the same Slack thread: should we switch?

For teams running agent systems in production, the answer is almost always: not yet, and maybe not ever. The model is one input into a system whose behaviour is determined mostly by everything else around it — the prompt, the tools, the retrieval layer, the review path, the eval suite. Switching the model changes one of those inputs. It does not, on its own, fix a system that is unreliable, expensive, or both.

The model landscape in 2026 is mature enough that the frontier labs are near-parity on the things that matter most for operational work. The interesting decisions are no longer "which lab is winning" but "which model class fits this workflow, and at what cost."

The four things that actually matter in production

Marketing decks lead with benchmark scores. Production teams should ignore them.

What matters is what the model does inside a closed workflow, on the cases the system will actually see, at a cost and latency the business can sustain. That collapses to four criteria.

Tool use reliability. Can the model emit the right tool call, in the right format, with the right arguments, on the cases that matter? Most agent systems spend the majority of their tokens on tool use, not on chat. A model that scores 5% higher on a reasoning benchmark but fails 10% more often on the structured output the system depends on is a worse production model for that system. Measure this on your cases, not on a public eval.

Latency under load. A model that responds in 800ms on a single request can take 6 seconds when the system is running 50 concurrent workflows with long context. Latency compounds in agent systems: a single slow step turns a 3-step workflow into a 20-second experience. Test under load, not in isolation.

Cost per resolved case. Cost per token is a useless number for an agent system. What matters is the cost of a workflow that actually completes correctly. A more expensive model that finishes in 2 steps is cheaper than a cheap model that loops, retries, or fails and has to be re-run. Track dollars-per-resolved-case, not dollars-per-call.

Behaviour under distribution shift. Models that work well on the cases you tested often degrade on cases that look slightly different — unusual input formats, ambiguous language, edge cases the system has not seen. The question is not whether the model is good in the lab, but whether it is graceful when it is uncertain. Graceful here means: it asks for clarification, it returns a structured error, it routes to a human, it does not hallucinate plausible-looking nonsense.

A model that wins on three of these and loses badly on the fourth is the wrong model for production. Most of the time the fourth one is the trap.

The frontier-vs-open-source question

The framing of "open-source versus closed" is mostly irrelevant for operational work. The real question is which model class — small and fast, mid-size and capable, or large and expensive — fits the workflow.

Frontier closed models are justified when the workflow involves long context, complex multi-step reasoning, ambiguous inputs that require real interpretation, or cases where a wrong output is expensive. They are not justified as the default for every workflow in the system.

Open and open-weight models — both the mid-tier and the distilled smaller ones — are now good enough for a large class of production tasks. Extraction from structured documents, classification, routing, simple tool calls, summarisation, short-form generation: these are solved problems at the small-to-mid model tier, and they are not worth paying frontier prices for.

The hybrid pattern is the one we use most often. A router in front of the system sends easy cases to a smaller, cheaper model, and only escalates hard cases to the frontier. This is not theoretical. For document-heavy intake workflows, we've seen cost reductions of 60–80% with no measurable drop in quality, because the easy cases really are easy and the frontier only gets the cases that need it.

The trap with this pattern is treating the router itself as trivial. It is not. A bad router that over-escalates is just a more expensive frontier call. A bad router that under-escalates lets hard cases through to a model that is not equipped for them. The router needs its own eval suite, its own monitoring, and its own review path.

Context window is a trap

Larger context windows are sold as a feature. For production agent work, they are often a liability.

Models with very long context windows are slower and more expensive per call. They also have a well-documented tendency to attend less reliably to information buried in the middle of a long context, which means the system ends up re-implementing retrieval inside the prompt anyway. The work that retrieval was supposed to replace gets done less reliably and at higher cost.

The right pattern is still: small, well-curated context, plus a retrieval layer that puts the right information in front of the model at the right time. A model that can handle 8k tokens well is more useful in production than a model that can handle 1M tokens poorly, because the well-curated case is faster, cheaper, and more predictable.

The exception is genuinely long-document workflows — full contract review, multi-thousand-line code analysis, large corpus search. For those, long context is the right tool, and the cost is the cost.

Chasing the new model

Every model launch triggers a wave of "should we switch?" conversations. Most of those conversations are a distraction.

Switching a model in a production agent system is not free. It requires re-running the eval suite, re-running the regression tests, often re-writing parts of the prompt that were tuned to the previous model's style, and re-running the calibration of the human review thresholds. It also produces a period of increased uncertainty, during which the system's behaviour is different from what the monitoring dashboards expect. For a system that is operating in production, that period of uncertainty is a cost.

The teams that switch models frequently tend to be the teams whose systems are not yet load-bearing. Once a system is running 10,000 cases a day and producing output that customers and operators act on, the cost of switching is much higher than the cost of running a 6-month-old model.

The right time to switch is when the new model produces a measurable improvement on the eval suite that includes the system's actual cases — not the public benchmark — and when the improvement is large enough to justify the migration cost. That happens, but not every quarter.

Model selection is a system decision

The model is approximately 10% of a production agent system. The other 90% is prompt design, tool definitions, retrieval, the orchestration layer, the eval suite, the review interface, and the monitoring.

Teams that obsess over model choice and neglect the rest tend to produce systems that are fragile in production: they work on the cases the model was tested on, fail on the cases it was not, and accumulate technical debt that makes them harder to evolve.

Teams that treat model choice as a system decision tend to produce systems that hold up in production: they work on a wide range of cases, fail gracefully when they do fail, and can be evolved as the model landscape shifts. The model inside the system is replaceable. The system around it is the asset.

The position we hold

The frontier models in 2026 are good. They are not so much better than the previous generation that they justify a migration on their own. The interesting work is in the system around the model — the routing, the retrieval, the review path, the eval suite. The model is a component. The system is the product.

If you are choosing a model for a production workflow, the right question is not "which model is best." It is "which model is best for the four criteria that matter, on the cases this workflow actually sees, at a cost the business can sustain, in a system I am prepared to operate." Most of the time, the answer is a hybrid: a small, fast model for the easy cases, a frontier model for the hard ones, and a router between them that is treated as a first-class part of the architecture.

That answer will not change much over the next 12 months. The model landscape will. The framework for choosing will not.