Every New AI Model Is a Job Candidate: Vet It Like a New Hire

In Life as an AI-Native SMB Owner, I wrote about running a team of AI employees. This piece is about hiring for that team. In the second half of 2026, we learned the hard way that a newer model isn’t automatically a better hire.

Where we started

At the start of 2026, our daily workhorses were Claude Sonnet 4.6, Claude Opus 4.5, and GPT-5.2 Codex. On the Artificial Analysis Intelligence Index, they score 25, 29, and 29.

In March, OpenAI released GPT-5.4, and it became our main engine. Through June, nearly all of our agentic engineering ran on it, and every month we got faster.

Hitting a wall

The first big shake-up came on July 1, when Anthropic brought Claude Fable 5 back online. Fable was extremely powerful and extremely expensive. We used it sparingly, saving it for our hardest problems, and it was a huge help every time we did. A week later, on July 9, OpenAI released the GPT-5.6 family: Sol, Terra, and Luna. Newer and cheaper per token, it looked like an easy upgrade. By the week of July 13, GPT-5.6 Terra was doing 92% of our work, and going back wasn’t an option.

By the numbers:

Our AI model mix, week by week, from March to October 2026: weekly tokens by model and weekly spend, with model launch dates and the wall shaded from mid-July through August

Daily averages before and during the wall. GPT-5.4 era, May 18 to July 5, 2026: 78 million tokens, $61 spend, $0.78 per million tokens, 12% of commits were bug fixes. GPT-5.6 Terra, July 13 to August 30, 2026: 165 million tokens, $69 spend, $0.42 per million tokens, 25% of commits were bug fixes.

We were burning more than twice the tokens, but each token cost about half as much, so the bill barely moved. Nothing in our spending looked wrong. That’s a big part of why it took us so long to figure out what was happening.

What had actually changed was the work itself: it had gotten harder. In our workload, Terra was slower than GPT-5.4. It introduced bugs, and finding those bugs took even more time and tokens, which led to more of the same. Across our repositories, the share of commits that were bug fixes doubled. For the entire GPT-5.6 period, nearly two months, we kept tuning prompts and harness rules, assuming the problem was on our side.

The light bulb came on September 5, 2026, our first day working with GPT-6 Astra. Work that had been grinding for weeks started moving again, and it finally clicked: the GPT-5.6 models were causing us serious problems. From that day on, each new model was better than the last. It took until mid-October for us to have enough distance to write about it.

On launch day, the GPT-5.6 family looked like a clear upgrade: newer, cheaper per token, and backed by strong benchmark numbers. The resume said one thing. Our real work said another.

Getting past it

Then a new wave of candidates arrived:

Sept. 3, 2026: OpenAI’s GPT-6 Astra got us over the wall. At a list price of $10 per million input tokens and $50 per million output tokens, it wasn’t cheap.
Sept. 22, 2026: Anthropic’s Claude Opus 5.5 arrived with top-tier performance at a lower price, followed by Claude Sonnet 5.5 on Sept. 28.
Sept. 29, 2026: OpenAI’s GPT-6.1 Sol claimed near-Astra results at one-fifth of Astra’s token price.
Oct. 13, 2026: Google came back with Gemini 4 Argon. Our early testing was enough to mark it as a strong new candidate.

Artificial Analysis plots each model’s index score against what it costs, at list price, to run one index task. Here’s how that looked on September 23, 2026:

Artificial Analysis Intelligence Index vs. Cost per Task, September 23, 2026

Claude Opus 5 (high), one of our summer workhorses, sits at about 48 for roughly $3.60 a task. Opus 5.5 had already pushed the top of the chart to about 57.

Three weeks later, on October 14, 2026:

Artificial Analysis Intelligence Index vs. Cost per Task, October 14, 2026

GPT-6.1 Sol (medium) now scores about 48 for roughly 22 cents a task. That’s the score Opus 5 (high) earned over the summer, at less than a tenth of the cost, in about three months. Epoch AI has tracked this pattern for years: the price of a given level of AI performance falls somewhere between 9x and 900x a year, depending on the task.

A great resume isn’t a hire

Our wall came from treating a new model as a drop-in upgrade. Now we treat each release like a job applicant. No small business owner hires someone on a resume alone. You interview them, check references, and give them a trial period. We do the same with models:

Read the resume. Independent benchmarks like Artificial Analysis tell you whether a candidate is worth an interview. Compare against the model you already have, not just the launch-day claims.
Run a working interview. Give the model a handful of your own real tasks, the ones your current team already handles well, and compare results, speed, and token counts side by side.
Check references. Look at what other builders report after the first week, when launch hype has settled and real-world quirks show up.
Talk pay. Look at cost per finished task, not price per token. Terra was half the price per token and still cost us more, in time and in tokens. The reverse can be true, too: Fable was our most expensive model, and well worth it for the few jobs that needed it.
Onboard, then probation. Rewrite its instructions for how it works, start it on lower-risk work, and watch the numbers that warn you early: tokens per day and how much of the work is bug fixes.

Onboarding matters on its own, too. A new model thinks and behaves differently from the ones you already have. Hand it the same instructions and it can do worse. One study found that reusing a prompt built for one model on another often performs substantially worse than a prompt written for the new model. Not Diamond saw the same thing: better models, worse results, when nobody updated the instructions.

What it looked like after the wall

GPT-6 Astra got us moving again, but it was an expensive hire. From September 7 to 21, 2026, while Astra did most of our work, we averaged about 119 million tokens and $165 a day. That’s $1.39 for every million tokens.

So we went back to interviewing. Once Opus 5.5 and GPT-6.1 Sol arrived, we put them through the same process and refactored our Agentic Memory Bank harness to fit how they work. That meant rewriting instructions, trimming what they didn’t need, and routing each task to the model that handles it best for the money.

From October 3 to 14, 2026, we ran about 350 million tokens a day, nearly three times the Astra weeks. Daily spend fell 26% to about $121, and cost per million tokens fell 75% to $0.35.

Our token burn vs. cost across three model line-ups: tokens per day, spend per day, and cost per million tokens for August, September 7–21, and October 3–14, 2026, split by OpenAI and Anthropic

These models burn a lot of tokens, too. Our output tokens went from about 385,000 a day to 1.17 million. The difference is that, unlike during the wall, the work is moving. We’re landing nearly twice as many commits a day as we did during the wall.

Set a pace people can keep

No single person can keep up with this many releases. BCG researchers have a name for what happens when you try: “AI brain fry,” the mental fatigue that comes from overseeing more AI than your brain can handle. They found less of it where managers were deliberate about how AI was used.

That’s why we don’t chase every launch. Our JS Agentic Framework relies on methods that have been around for decades. The OODA loop helps us decide which releases deserve an interview. PDCA turns each trial into a small, measured experiment, and the lessons go back into the harness. Because we stay AI-agnostic, we can move work to a different model when one isn’t working out.

We make changes in deliberate steps, at a pace people can handle, and we still stay near the front.

Start your own hiring process

Write down which model does which job in your business, what it costs, and how many tokens it uses in a normal week. The next time a big release lands, don’t switch everything over. Interview it on three real tasks first, and if your token use suddenly doubles, find out why before you blame your prompts.

Your AI team keeps getting smarter and cheaper. That’s great news, as long as you’re hiring on purpose.

Want help vetting your next AI hire? Let’s talk.