Which Model is Best?

This is the first in our Demystifying AI series, taking the moving parts of AI one at a time and explaining them the way we wish someone had explained them to us: the model, the prompt, the context and the harness around them.

Which model is best? I get asked this constantly, and I understand why. Every new release comes with a slew of contradictory benchmark charts, comparison threads and a fresh wave of hype. So before the next launch pulls us all back in, it’s worth stepping back and asking what any of it actually means for your day-to-day.

Before answering it, two different things need pulling apart, because the question usually runs them together. 1) There are families of models: the general-purpose ones, the small fast ones, the advanced reasoning ones, etc, and 2) there are providers (i.e. OpenAI and Anthropic), each selling their own version of most of those families. 

From our point of view, the models provided across the major platforms (as well as several open-source models) are now all mostly good enough to do what many people need. There are minor gains here and there and each generation is still demonstrably better than the last, and (thankfully) that will keep happening. But for the things most organisations need doing, the leading models all do them well, and swapping one for another is often a single line of configuration. If your AI results are disappointing, the model is almost never the reason. So ‘which model is best’ is mostly settled. The questions I keep hearing underneath it are the interesting ones, and they don’t get talked about as much. Which family of model does the work actually call for? When are you supposed to use each one? What do those low, medium, high and extra-high settings actually do? Why do some queries cost almost nothing and others noticeably more?

When do you use each family?

Every provider now sells families or tiers of model: small, fast and cheap alongside large, slow and deep. Think of it like staffing and resourcing. You wouldn’t put your most senior person on formatting meeting notes, but that’s exactly what’s happening when the expensive model writes them. The opposite is just as problematic. Hand an advanced specialist task to a cheap model and it won’t refuse the work; it’ll do it, confidently, to the standard of a junior employee.

What does the “reasoning” level actually do?

This is a setting that’s sometimes visible in a platform but other times needs to be selected when implementing models and agents in code. Higher levels means the model spends longer working the problem through before it answers. That’s all it is. It’s the difference between asking a colleague a question at their desk and asking them to go away and come back with a worked answer. On a hard, multi-step problem, the added time and cost is typically worth the better response. On a simple one, you’ve paid extra for thinking nobody needed. 

What does it cost?

The cost of AI is starting to become a broader point of conversation. There was a moment in the industry where employees would hammer through the models and were even encouraged to engage in token maxxing: effectively using and consuming tokens to the highest possible extreme as an indicator of how much an organisation is using AI. However, tokens and costs are directly linked. The more tokens you use, the higher the cost. Deep reasoning on a large model uses many more tokens and thus can cost many times that of a quick answer on a small one. 

So treat the families like a team: cheap, fast defaults for routine work, and escalation when the output needs more thought. The organisations we see benefiting from AI without disproportionately feeling the pain of the cost aren’t the ones using the “best” model or “best” platform. They’re the ones matching the model, and the thinking time, to the work.

When the model really is the problem

Sometimes it is. Sometimes, no matter how you prompt, no matter what context you provide, no matter how long you let the best model think, the output keeps coming out wrong enough to be a problem. A clear example would be in the earlier days of 2024 when OpenAI’s Whisper transcription tool was used by hospitals to transcribe medical consultations. The model was known to invent text and medical treatments  that were never mentioned. ++OpenAI itself warned++ against using it in high-risk areas because the consequences of inaccuracies could be significant. It wasn’t a bad model, and in many ways, the best in its class at the time, but it just wasn’t good enough for the purpose it was used for. 

As I said earlier, the models are much better these days, and they’re the one lever that we don’t need to do much about as they will continue to get better with or without our input. So, if your AI outputs are repeatedly coming back generic, inconsistent, or just wrong, I’d advise against assuming it’s the model’s fault. We’ll cover the other levers in the next articles in the series.

*Part 2: What Makes A Good AI Prompt?*