The Newest Model Isn't Automatically the Best Writereric

Almost everyone has an AI helping them write now, and the jobs come in different sizes. A social media post, a quick line on Twitter/X, is a small job. Replying to an email, matching a tone and landing a point, is a medium one. Writing a tech report, where the structure has to hold up and a mistake costs someone real time, is a big one. The same model often gets handed all three, without anyone checking whether a model that is good at one is good at the others.

So the question worth asking is not whether AI can do this work. It plainly can. The question is which model does it best, at which size. At TYO we had a pile of small-to-medium jobs like these, tweets, gallery captions, short intros, and the sensible move seemed to be picking one fast, cheap model to handle all of them. We reached for the newest small model from a frontier lab, because newer and frontier means better. That felt obvious enough to skip the test.

We ran the test anyway. We are glad we did, because the newest model lost.

The test

Nothing clever. Three real jobs, the same brief, two models, run back to back.

The jobs were a tweet announcing some new gallery images, a forty-to-sixty word caption for one of them, and a one-hundred-and-fifty word blog intro in a specific voice. That is two small jobs and one medium one, in the sizes we just laid out. The brief told both models the same thing: write plainly, stay concrete, no hype words, no emoji. The house style we actually use.

One model was Google's Gemini Flash. The other was Claude Haiku 5.5, the newest small model in Anthropic's lineup at the time we ran this. Same prompts, same token budget, same afternoon.

What came back

Speed first, because it is the cleanest number.

Haiku was the faster model by a wide margin, three to eight times quicker on the wall clock. If raw speed were the only axis, it would win outright.

Writing was the other axis, and it went the other way. On the tweet the two were a coin toss, both on voice and both inside the character limit. On the blog intro, Flash hit its word count almost exactly and, on this run, opened with the punchier hook, while Haiku ran long and read a little soft. Hold that word "punchier" loosely. We come back to it near the end, after running the same job five more times, and it does not survive. The clearest gap was the caption. We asked both to describe an image of Chang'e, the moon goddess, rising toward a full moon in pale-blue silk.

Flash wrote:

The composition leans into hard contrast and clean fabric drape, catching the myth at the exact point she leaves the earth behind.

Haiku wrote:

Stars scatter across the night behind her, and the luminous mist seems to carry her upward, quiet and unhurried, as if the sky were already hers.

Here is the twist in our own result. On that caption, Haiku is the better piece of writing. It stays inside the picture. Flash slips into talking about the composition, like a critic standing next to the frame. Neither is wrong, and the gap is genuinely task-dependent. But add up all three jobs, weigh the lines we would put our name on, and the copy we would ship leaned Flash more often than it leaned Haiku. The newest model did not clear the bar we assumed it would clear.

Why this is worth a post

The winner is not the point. One sample per task is not a benchmark, and your three jobs are not our three jobs. Yours might be the social post, the email reply, and the report, not the tweet, the caption, and the blog intro. Same three sizes, different brief, possibly a different winner. The point is that we almost did not run it at all.

Three quiet habits pushed us toward skipping the test, and all three are worth naming because every builder has them.

The first is recency bias. A model that shipped last week feels like it must be better than one that shipped last quarter. Sometimes it is. On your task, you do not know until you look.

The second is frontier-lab bias. The lab with the strongest flagship model is not automatically the strongest at every size and every task. A small model is a different product from the flagship, with different training trade-offs, and the brand on the box does not carry over.

The third is the one that bites hardest. Tier labels do not travel between labs. One company's small is not another company's small. They are priced and sized against different internal lineups, so comparing by name or by price tier tells you almost nothing about how either one writes your copy.

The experiment cost almost nothing

Twenty minutes and a few cents. A short script sends the same prompt to two APIs and prints the answers next to each other. That is the whole apparatus.

If you are about to standardize a product on one model for a job that matters, run your own version first. Use your real tasks, your real voice, your real constraints, and read the outputs yourself. A public leaderboard was built to rank general ability for someone else's work. It was not built to tell you which model writes the caption under your image.

Where this result could be wrong

Our test is directional, not gospel, and the reasons are instructive. The temperature setting is now deprecated on Haiku 5.5, so it ran at its own default while Flash ran at 0.7. Not a perfectly matched sampling setup. The latency numbers include server-side thinking on both sides. And it was a single sample per task. All of which is exactly why you should run this on your own material rather than trust ours. The test is cheap enough that there is no excuse not to.

So we ran the intro five more times

That single-sample worry nagged at us, in particular the credit we gave Flash for the punchier hook. So we ran the one-hundred-and-fifty word intro job five times for each model, same brief, same settings, and read the spread.

Two things held, and one did not. Flash's length discipline is real: five runs, every one between 143 and 150 words. Haiku's overshoot is just as real: it ran long every single time, out to 205 words. But the punchy hook we handed Flash was a fluke. Across five runs Flash never wrote it again, and opened a different way each time. Haiku, the model we called softer, actually landed the sharp three-beat opening twice.

So the honest revision is this. Flash is not reliably punchier. It is reliably on length. The hook that impressed us was one lucky draw at a temperature of 0.7, which is the setting that turns every run into a fresh roll of the dice. A single sample is a story, not a measurement.

One smaller catch while we are here. gemini-flash-latest is a moving alias. During these runs it resolved to Gemini 3.8 Flash, and it can point at a new snapshot tomorrow. If you need a result you can reproduce, pin the exact model version instead of trusting "latest".

This fits a pattern we wrote about recently, when Meta's Muse agent went viral without inventing a single new capability. The lesson there was that the win sits around the model, not inside it. The lesson here is smaller and more practical. Before you trust the label, test the thing.

We already ran the next round, a larger model against Flash, and the story changes again.

Next: the same three jobs, Gemini Flash against a bigger Claude model. The result is not what the first round would have you predict.