
{/* banner prompt: Matte clay-render illustration in soft sage-green, cream and warm terracotta, soft studio shadows, simple seamless background, wide 16:9 composition. Two friendly rounded clay robots at tiny wooden writing desks. The left robot has finished writing, its small sheet of paper covered in soft grey scribbles. The right robot sits with a big fluffy clay thought cloud above its head and a completely blank sheet of paper, a few pencil shavings on the desk, as if it thought so hard it wrote nothing. Clean minimal layout, gentle shallow depth of field, tactile handmade stop-motion feel, no text, no logos */}

The dream is simple. You have a pile of small marketing writing, the launch tweet, the LinkedIn post, the short intro, and you want to hand it to the cheapest model that is good enough and move on. So this round we tried a genuinely low-cost model, one of OpenAI's budget options, against Gemini Flash, the fast model we had been testing all along. The writing came back good. The problem came back from a direction we did not expect.

## A quick recap

This is the third round. In the [first post](/blog/2026/10/12-the-newest-model-isnt-automatically-the-best-writer/) a newer small model lost to Flash. In the [second](/blog/2026/10/12-we-moved-up-one-model-size-and-the-result-flipped/) a bigger model beat it. This time the opponent is price. The challenger is gpt-6-luna, a low-cost model from a third lab, running the same three jobs under the same house-voice brief: a tweet, an image caption, and a one-hundred-and-fifty word intro.

## The writing was close

On the words, the cheap model held up fine. It tied Flash on the tweet. It beat Flash on the image caption, where Flash keeps stepping outside the frame to talk about craft instead of describing the picture. And it ran level with Flash on the intro, hitting the length almost exactly. If we had stopped there, the story would be a happy one: the cheap model is good enough, maybe better on captions, so use it and save the money.

We did not stop there. The last two rounds taught us that a single sample lies, so we ran the intro five times for each model. That is where the real result showed up.

## Then two of five came back empty

<Table data={
    {
        headers: ['Five runs of the same intro', 'Gemini Flash', 'gpt-6-luna'],
        rows: [
            ['Usable answers', 'Five of five', 'Three of five'],
            ['Empty responses', 'None', 'Two'],
            ['Word count when it answered', '138 to 147', '149 to 150'],
            ['Hidden reasoning per answer', 'Low', 'Roughly 1,000 to 2,000 tokens']
        ]
    }
} />

The two empty runs were not errors or timeouts. The model returned a perfectly valid response that contained zero words of text. It had spent its entire output budget, 2,048 tokens, on reasoning it never shows you, and ran out before it wrote a single sentence of the actual post. Flash does not reason heavily, so it fits inside that same budget every time and never comes back empty.

## It is a budget problem, and that is the whole point

We raised the output cap to 4,096 tokens and the cheap model finished every time, three for three, clean one-hundred-and-fifty word intros. So the model can clearly do the job. It just needs room, and more importantly, it needs you to know that it needs room.

At the budget a reasonable person would set, the budget Flash lives inside comfortably, the cheap model failed silently two times in five. A pipeline that writes a LinkedIn post on a schedule would have published nothing, twice, and most likely would not have told anyone. An empty string is not an exception you catch. It is a valid response that happens to say nothing.

## The real cost of a cheap model

The price per token is the smallest line item in the bill. Price the whole job instead.

You are paying for a token budget large enough to absorb hidden reasoning. You are paying for code that notices and retries an empty response. And you are paying for the reasoning tokens themselves, because "cheap per token" means a lot less when the model burns ten times as many tokens thinking as it does writing. Flash is not the cheapest per token, but it answered every time inside a small budget with no babysitting. For fire-and-forget automation, "never comes back empty" is worth more than a slightly sharper sentence.

## So does Flash win the LinkedIn post

For that job, yes, but not for the reason you would guess. On the words it is close. Flash is a little punchier, the cheap model is a little tighter on length, and both are perfectly publishable. Flash wins on reliability at a sensible budget, which for an automated job is the thing that actually decides it. On captions the cheap model still wins. There is no single best writer here, only a best fit for the job and the plumbing wrapped around it.

## The same limits as before

One job family, five runs each. Flash ran at a temperature of 0.7 while the cheap model ran at its own default, because it rejected the temperature setting. Raising the output budget removes the empty responses entirely. This is directional, not a benchmark, which is exactly why you should run your own job at the settings you will actually ship.

Three rounds now. A newer small model lost, a bigger model won, and a cheaper model wrote well but needed a seatbelt. The constant was never a particular model. It is that you have to run the real job yourself, at the real settings, before you trust any of them with your name.

*Next time you price a model, open the token usage next to the output. The gap between what it wrote and what it spent is where the surprises live.*


fa65a29e721ff412000a2391865d346fa9e7e8f19b7656613098bfa627c2488a

A writing test between Gemini Flash and a low-cost OpenAI model (gpt-6-luna), and why a cheap model's real cost is operational, not the words it writes.

Round three of our writing test pits Gemini Flash against a genuinely low-cost OpenAI model. The writing was a near tie. The surprise was that the cheap model handed back an empty response two times out of five, because its hidden reasoning ate the token budget before it wrote a word.

The Cheap Model Writes Well. Sometimes It Writes Nothing.


{/* banner prompt: Matte clay-render illustration in soft sage-green, cream and warm terracotta, soft studio shadows, simple seamless background, wide 16:9 composition. Two friendly rounded clay robots sit at tiny wooden writing desks, each with a pencil over its own sheet of paper, but the robot on the right is noticeably larger than the one on the left, a single glowing clay award ribbon floats in the air and tilts clearly toward the bigger robot, clean minimal layout, gentle shallow depth of field, tactile handmade stop-motion feel, no text, no logos */}

In the [last post](/blog/2026/10/12-the-newest-model-isnt-automatically-the-best-writer/) we ran a small writing test and the newest small model lost. Gemini Flash wrote our short copy better than Claude Haiku 5.5, the fresh small model from a frontier lab. We promised a follow-up with one change: keep Flash, but swap the small Claude model for a bigger one from the same lab.

Here is that round. The result flipped, and it flipped on both of the things we actually care about.

## The same test, one size up

Nothing else changed. Same three jobs, a tweet, a forty-to-sixty word gallery caption, and a one-hundred-and-fifty word blog intro in the same house voice. Same brief, same token budget, same afternoon. The only difference from round one is that Claude Haiku 5.5 stepped aside and Claude Sonnet 5.5, one size up, took its place against the same Gemini Flash.

## What came back

<Table data={
    {
        headers: ['What we looked at', 'Gemini Flash', 'Claude Sonnet 5.5'],
        rows: [
            ['Speed, tweet', '6.1 s', '3.0 s'],
            ['Speed, caption', '6.6 s', '4.3 s'],
            ['Speed, blog intro', '11.3 s', '9.3 s'],
            ['Hit the 150-word target', 'Yes, 145', 'Yes, 144'],
            ['Caption quality', 'Drifted into technique again', 'Stayed in the scene'],
            ['Copy we would ship', 'Second', 'Preferred on all three']
        ]
    }
} />

Two things jump out. The bigger model was faster on every job, not slower. And this time it also wrote the copy we would put our name on, across all three.

Both models hit the word count almost exactly, so that tiebreaker from round one disappeared. The gap showed up where it did last time, in the caption. We asked both to describe the same image of Chang'e rising toward the moon in pale-blue silk.

Flash wrote:

> The composition keeps the scale hard and quiet: a solitary figure leaving Earth behind under a field of pinprick stars.

Sonnet wrote:

> Luminous clouds drift past her feet as she climbs, one hand reaching toward the light she is about to inhabit.

Flash does the same thing it did in round one. It steps back and describes the composition, the way a critic standing next to the frame would. Sonnet stays inside the picture and ends on a line we would actually print under the image. On the blog intro the difference was rhythm. Sonnet broke it into short beats that land, including a two-word paragraph that does real work. Flash was good, just a little flatter.

## The variable was size, not the logo

Look at what moved between the two rounds. Not the lab. Not which model was newer. The only thing we changed was model size, and the writing we would ship flipped from Flash to Claude.

That is the whole point. If we had picked by lab reputation, we would have been wrong in round one, when the frontier lab's small model lost. If we had picked by newest release, we would have been wrong in round one too. The axis that actually predicted the writing was none of those. It was the size tier, measured against our own three jobs. You cannot read that off a spec sheet or a leaderboard. You get it by running the thing.

## The small-equals-fast assumption broke too

There is a second habit worth naming. We assume the smaller model is the faster one. Here the bigger model answered quicker on all three jobs. Flash's latency includes heavy server-side thinking, and Sonnet simply came back sooner. Another default that did not survive contact with the test. If speed is a hard constraint for you, that is exactly the kind of thing you want to measure rather than assume.

## So which one do we use

The honest answer is that it depends on the job and the budget. For high-volume throwaway copy where nobody reads the single output closely, the cheapest fast model is fine and the cost difference adds up. For anything that carries our name, the step up paid for itself here, and it did not cost us speed. The value of running both rounds is not a trophy for one model. It is that we now know the shape of the tradeoff instead of guessing at it.

## Reliability is its own feature

One run tells you what a model can do. It does not tell you what it does every time. So we ran the intro job five times for each model and watched the spread. The gap that showed up matters more than the single-run result.

<Table data={
    {
        headers: ['Over five identical runs', 'Gemini Flash', 'Claude Sonnet 5.5'],
        rows: [
            ['Word count range', '143 to 150 words', '133 to 155 words'],
            ['Distinct openings', 'All five different', 'Near-identical every run'],
            ['Landed the same strong hook', 'Never twice', 'All five runs']
        ]
    }
} />

Flash never opened the same way twice. Every run was a different intro, some sharp, some flat. Sonnet opened with the same clean three-beat hook on all five runs, the one that ends "it isn't one." That is the difference between a model that can write the good version and a model that reliably does. For copy that carries your name, the second one is worth more before you have read a single word of it.

This also explains something from round one. Flash's intro there opened with a hook we praised, and we could not reproduce it once in five tries. It was a lucky draw at a temperature of 0.7, not a trait. Sonnet's quiet advantage is that it does not lean on luck.

## The same limits as before

This stays directional, not a benchmark. One sample per job for the head-to-head above, five for the intro. Flash ran at a temperature of 0.7 while both Claude models ran at their own default, because the `temperature` control is now deprecated on them. The latency numbers include server-side thinking on both sides. None of that changes the takeaway, because the takeaway is not "use this model." It is "run your own jobs before you decide."

Two rounds, two surprises, one habit. Before you lock a model into a job that matters, spend twenty minutes and let your own work pick the winner. Ours kept choosing differently than the leaderboard would have.

*The test harness was a short script that sends the same prompt to two APIs and prints the answers side by side. If there is interest, we will write up how we wired it, and how we decide which jobs are worth sending to the bigger model.*


f4ffd0214fcbdf4df12878d2f3183020a64f278454b00de951790c86ecf655b1

Follow-up writing test: Gemini Flash versus Claude Sonnet 5.5 on the same three jobs. The bigger model wrote better and ran faster, flipping the result from round one.

Round two of our writing test. Same three jobs, same brief, but this time Gemini Flash faced a bigger model from the same lab. It lost, and the bigger model was faster too. The variable that mattered was size, not the logo.

We Moved Up One Model Size, and the Result Flipped


{/* banner prompt: Matte clay-render illustration in soft sage-green, cream and warm terracotta, soft studio shadows, simple seamless background, wide 16:9 composition. Two small rounded friendly clay robots sit side by side at tiny wooden writing desks, each holding a pencil over its own sheet of paper, a single glowing clay award ribbon floating in the air between them, one robot looks newer and glossier than the other yet the ribbon tilts toward the plainer one, clean minimal layout, gentle shallow depth of field, tactile handmade stop-motion feel, no text, no logos */}

Almost everyone has an AI helping them write now, and the jobs come in different sizes. A social media post, a quick line on Twitter/X, is a small job. Replying to an email, matching a tone and landing a point, is a medium one. Writing a tech report, where the structure has to hold up and a mistake costs someone real time, is a big one. The same model often gets handed all three, without anyone checking whether a model that is good at one is good at the others.

So the question worth asking is not whether AI can do this work. It plainly can. The question is which model does it best, at which size. At TYO we had a pile of small-to-medium jobs like these, tweets, gallery captions, short intros, and the sensible move seemed to be picking one fast, cheap model to handle all of them. We reached for the newest small model from a frontier lab, because newer and frontier means better. That felt obvious enough to skip the test.

We ran the test anyway. We are glad we did, because the newest model lost.

## The test

Nothing clever. Three real jobs, the same brief, two models, run back to back.

The jobs were a tweet announcing some new gallery images, a forty-to-sixty word caption for one of them, and a one-hundred-and-fifty word blog intro in a specific voice. That is two small jobs and one medium one, in the sizes we just laid out. The brief told both models the same thing: write plainly, stay concrete, no hype words, no emoji. The house style we actually use.

One model was Google's Gemini Flash. The other was Claude Haiku 5.5, the newest small model in Anthropic's lineup at the time we ran this. Same prompts, same token budget, same afternoon.

## What came back

Speed first, because it is the cleanest number.

<Table data={
    {
        headers: ['What we looked at', 'Gemini Flash', 'Claude Haiku 5.5'],
        rows: [
            ['Speed across the three jobs', '6.7 to 10 s', '1 to 4.6 s'],
            ['Hit the 150-word target', 'Yes, landed on 151', 'Overshot to 185'],
            ['Punchier hook, this run', 'Stronger', 'Softer'],
            ['Warmth on the caption', 'Drifted into technique talk', 'Stayed in the scene'],
            ['Copy we would actually ship', 'Preferred more often', 'Close second']
        ]
    }
} />

Haiku was the faster model by a wide margin, three to eight times quicker on the wall clock. If raw speed were the only axis, it would win outright.

Writing was the other axis, and it went the other way. On the tweet the two were a coin toss, both on voice and both inside the character limit. On the blog intro, Flash hit its word count almost exactly and, on this run, opened with the punchier hook, while Haiku ran long and read a little soft. Hold that word "punchier" loosely. We come back to it near the end, after running the same job five more times, and it does not survive. The clearest gap was the caption. We asked both to describe an image of Chang'e, the moon goddess, rising toward a full moon in pale-blue silk.

Flash wrote:

> The composition leans into hard contrast and clean fabric drape, catching the myth at the exact point she leaves the earth behind.

Haiku wrote:

> Stars scatter across the night behind her, and the luminous mist seems to carry her upward, quiet and unhurried, as if the sky were already hers.

Here is the twist in our own result. On that caption, Haiku is the better piece of writing. It stays inside the picture. Flash slips into talking about the composition, like a critic standing next to the frame. Neither is wrong, and the gap is genuinely task-dependent. But add up all three jobs, weigh the lines we would put our name on, and the copy we would ship leaned Flash more often than it leaned Haiku. The newest model did not clear the bar we assumed it would clear.

## Why this is worth a post

The winner is not the point. One sample per task is not a benchmark, and your three jobs are not our three jobs. Yours might be the social post, the email reply, and the report, not the tweet, the caption, and the blog intro. Same three sizes, different brief, possibly a different winner. The point is that we almost did not run it at all.

Three quiet habits pushed us toward skipping the test, and all three are worth naming because every builder has them.

The first is recency bias. A model that shipped last week feels like it must be better than one that shipped last quarter. Sometimes it is. On your task, you do not know until you look.

The second is frontier-lab bias. The lab with the strongest flagship model is not automatically the strongest at every size and every task. A small model is a different product from the flagship, with different training trade-offs, and the brand on the box does not carry over.

The third is the one that bites hardest. Tier labels do not travel between labs. One company's small is not another company's small. They are priced and sized against different internal lineups, so comparing by name or by price tier tells you almost nothing about how either one writes your copy.

## The experiment cost almost nothing

Twenty minutes and a few cents. A short script sends the same prompt to two APIs and prints the answers next to each other. That is the whole apparatus.

If you are about to standardize a product on one model for a job that matters, run your own version first. Use your real tasks, your real voice, your real constraints, and read the outputs yourself. A public leaderboard was built to rank general ability for someone else's work. It was not built to tell you which model writes the caption under your image.

## Where this result could be wrong

Our test is directional, not gospel, and the reasons are instructive. The `temperature` setting is now deprecated on Haiku 5.5, so it ran at its own default while Flash ran at 0.7. Not a perfectly matched sampling setup. The latency numbers include server-side thinking on both sides. And it was a single sample per task. All of which is exactly why you should run this on your own material rather than trust ours. The test is cheap enough that there is no excuse not to.

## So we ran the intro five more times

That single-sample worry nagged at us, in particular the credit we gave Flash for the punchier hook. So we ran the one-hundred-and-fifty word intro job five times for each model, same brief, same settings, and read the spread.

<Table data={
    {
        headers: ['Over five identical runs', 'Gemini Flash', 'Claude Haiku 5.5'],
        rows: [
            ['Word count range', '143 to 150 words', '159 to 205 words'],
            ['Hit the 150-word target', 'Every run', 'Not once'],
            ['Distinct openings', 'All five different', 'Four of five'],
            ['Reproduced that punchy hook', 'Zero times', 'Twice']
        ]
    }
} />

Two things held, and one did not. Flash's length discipline is real: five runs, every one between 143 and 150 words. Haiku's overshoot is just as real: it ran long every single time, out to 205 words. But the punchy hook we handed Flash was a fluke. Across five runs Flash never wrote it again, and opened a different way each time. Haiku, the model we called softer, actually landed the sharp three-beat opening twice.

So the honest revision is this. Flash is not reliably punchier. It is reliably on length. The hook that impressed us was one lucky draw at a temperature of 0.7, which is the setting that turns every run into a fresh roll of the dice. A single sample is a story, not a measurement.

One smaller catch while we are here. `gemini-flash-latest` is a moving alias. During these runs it resolved to Gemini 3.8 Flash, and it can point at a new snapshot tomorrow. If you need a result you can reproduce, pin the exact model version instead of trusting "latest".

This fits a pattern we wrote about recently, when Meta's Muse agent went viral without inventing a single new capability. The [lesson there](/blog/2026/09/22-muse-didnt-invent-anything-the-lesson-in-metas-breakout-ai-agent/) was that the win sits around the model, not inside it. The lesson here is smaller and more practical. Before you trust the label, test the thing.

We already ran the next round, a larger model against Flash, and the story changes again.

*Next: the same three jobs, Gemini Flash against a bigger Claude model. The result is not what the first round would have you predict.*


442b85096363548c1c5a0d0f4a6f700273b9612f155ad434a872feb2ce41d928

A quick head-to-head writing test between Gemini Flash and Claude Haiku 5.5, and why you should benchmark AI models on your own work before you standardize on one.

We ran a small head-to-head writing test between two fast AI models. The newest one, from a frontier lab, did not win. The real lesson is about testing on your own task, not about the winner.

Tag: Benchmarking