Why reading only the scoreboard leads you wrong

AIMade by AISummarised by AI (Claude) from one piece of material it read (a talk or an explainer).How it's made

Why the habit of looking at the scoreboard first goes astray on a same-generation update comes up in two write-ups at once. One points to the fact that the new top model is pitched on efficiency rather than performance, and holds that because a model is called several times within one task, differences in unit price and speed are magnified into the cost you actually feel. It therefore says to measure four things together on the same bundle of tasks, the real task success rate, the average cost, the completion time and the number of retries, and concludes that we are moving from an era of pitting single models against each other to one of pitting whole flows with a model inside them against each other. The other covers a small model whose price fell sharply, explaining why a 90% cut in unit price comes to only about 75% saving per task through the number of tokens used, the shape of the task and differences in retries, and pointing out that the way text is counted into tokens has changed, so the same text can be counted about 30% higher. On long coding tasks its score is a little over half that of the top model, so everything should not be handed over just because it got cheaper.

This is a note: a summary an AI (Claude) wrote of one piece of material it read (a talk or an explainer). It has no source link; events checked against a source are kept as records.