Changing the model or changing the brief

I often give the same brief to several models. When they agree, I want to know whether that tells me something useful or whether they are just making the same familiar choices.

I was reading Fred Brooks's The Design of Design and thinking about how his ideas apply to designing with AI. That led to two small tests on October 25: do models from the same lab make similar choices, and does rewording a brief change those choices?

In the earlier landing-page test, the models built complete pages. Here I asked for colors, fonts, a corner radius and a headline. Claude Code turned those values into the cards below. The models didn't design the card layouts.

Do models from the same lab choose similar things?

Claude Code wrote the brief and the script that sent it to ten models:

"Design the hero section of a landing page for a neighbourhood tool library (people borrow drills, ladders and saws from a shared shed)."

The requested values were background, text and accent colors, heading and body fonts, a corner radius and a headline. This kept the comparison manageable, but left out most of what goes into designing a page.

I used two models from each of five labs: Claude Sonnet 5 and Claude Opus 5, GPT-5.4 mini and GPT-5.6 Luna, Gemini 2.5 Flash and Gemini 3.1 Pro preview, Qwen 3.7 Flash and Qwen 3.8 Max, and Kimi K3 and Kimi K2.7 Code. Each model answered four times. All 40 answers were usable. This first run cost about 8 cents.

How close the answers were

Claude Code measured how far apart two answers were, on a scale from 0 to 1. The measure combines seven parts with equal weight: the accent's hue, saturation and lightness, the background's lightness, the corner radius (on a log scale), and whether the two answers used the same heading font and the same body font.

Two answers from the same model were 0.23 apart on average. Two models from the same lab were 0.34 apart. Two models from different labs were 0.39 apart.

To see whether the lab difference could be luck, Claude Code's script tried every possible way of splitting the ten models into five pairs. There are 945 ways. The real lab pairing came 31st, which gives p of about 0.03.

That result depended a lot on which pair I looked at. The two Claude models were the closest of the five same-lab pairs, at 0.25, and they used the same heading font, Archivo Black, in seven of their eight answers. The two OpenAI models were 0.44 apart, further apart than the average pair from different labs. So in this test, "models from the same lab agree" was true mostly because of one pair.

Almost every accent was orange

34 of the 40 accent colors were orange, with a hue between 15 and 45 degrees. One more, from Gemini 2.5 Flash, was a red orange at 14 degrees, just below that band. Every one of the ten models chose orange at least once. The other five were green or teal, three of them from GPT-5.4 mini.

The brief may explain some of this. It is about drills, ladders and saws, so the orange may come from the subject more than from any lab's habits. On another brief that day, for a calm data app, none of the four models in that test chose orange.

What if I change the wording?

For the second test, I wanted to know whether changing a few words would make as much difference as changing the model.

Claude Code used the same tool-library brief and wrote four variations. They all describe the same thing:

• "Design the hero section of a landing page for a neighbourhood tool library (people borrow drills, ladders and saws from a shared shed)."
• "Design the top section of a homepage for a neighbourhood tool library (people borrow drills, ladders and saws from a shared shed)."
• "Design the hero section of a landing page for a community tool library where neighbours borrow drills, ladders and saws from a shared shed."
• "Design the hero section of a landing page for a local tool-lending library (people borrow drills, ladders and saws from a shared shed)."
• "Design the hero section of a website for a neighbourhood tool library: people borrow drills, ladders and saws from one shared shed."

Four models got every version twice: Claude Sonnet 5, GPT-5.6 Luna, Gemini 3.1 Pro preview and Qwen 3.8 Max, one from each of four labs. They returned the same kinds of values as in the first test. All 40 answers were usable. This run cost about 13 cents.

Changing the model made more difference

The answers were compared with the same 0 to 1 distance as the earlier test. Asking the same model the same wording twice already gave some spread: 0.27 apart on average. Changing the wording with the same model gave 0.30. Changing the model with the same wording gave 0.42.

Compared with simply asking again, rewording added 0.03 to the distance and switching the model added 0.14. Claude Code also gave a rough idea of how sure we can be of each, by repeating the comparison on about 2,000 random subsets of the four models. For rewording, the rough 90% interval ran from slightly below zero to 0.065, so in this test I can't tell it apart from no effect. For switching the model it ran from 0.046 to 0.215, which stays above zero.

The fonts show it most clearly. Across all ten of its answers and all five wordings, Claude Sonnet used only two heading fonts, Archivo Black and Poppins. No other model used either of them, though Qwen once used plain Archivo.

All 40 accents were orange, with every wording and every model. As in the earlier test, that may come from the subject (drills and saws) rather than from the models.

What I will do with this

I won't count two models from the same lab as two independent votes in my design rounds. If they agree, I still want to see what models from other labs make of the choice and decide for myself.

In the wording test, asking a different model changed the colors and fonts more than swapping a few words. That gives me a reason to try another model when I want a different direction. It doesn't tell me which result is better.

What I don't know yet

The same-lab test used one brief, ten models and four tries each. The lab effect is small. When Claude Code left the two Claude models out and ran the test again, the difference between same-lab and different-lab pairs mostly went away (p of about 0.27). The models gave values, not pages, at a temperature of 1 and a low reasoning setting. Claude Code chose the seven parts of the distance measure and gave them equal weight; I didn't test other weightings. Claude Code also wrote the brief, the script and the scoring. I reran the earlier landing-page test with other models setting it up. I haven't done that for either of these tests.

These five wordings were deliberately close. They kept the meaning and all the objects, so this tests small rewording, not a different brief. I haven't tested what happens when I change the mood, audience or requirements. It was four models, two tries per wording, and one brief. Claude Code wrote the four new wordings, the script and the scoring.