Evaluating LLM providers for production

Every production AI project we've shipped has started with the same uncomfortable question: which model? The answer changes every few months as the landscape shifts, so we've built a scorecard that abstracts away the specific names and focuses on the axes that actually matter for a given task.

The first axis is task fit. Frontier models are the default for complex reasoning, but a task that boils down to structured extraction from a narrow domain is often better served by a fine-tuned smaller model. We run both on a representative sample of real inputs before making a decision — marketing copy and eval benchmarks tell you surprisingly little about production behaviour.

Latency matters more than most people account for in early evaluations. A model that takes four seconds per call might be acceptable in a batch job but will destroy the UX of a real-time feature. We measure p95 latency, not mean latency — your p95 is what users experience on a slow day, and slow days correlate with high traffic.

Cost projections need to account for context growth. If your average call starts at 500 tokens but grows to 4,000 as users work through a session, your per-request cost has grown eightfold. Build token budget discipline into the architecture early: summarise rather than append, and prune context aggressively.

Reliability and rate limits are often deal-breakers that don't show up in demos. We run load tests against provider APIs before committing to a production architecture. We also evaluate fallback options — if the primary provider has an outage, can we route to a secondary without degrading the user experience beyond acceptable thresholds?