A model with a better leaderboard spot can lose to a cheaper model on your specific task. That is not a paradox, it is just how benchmarks are built.
Why the public leaderboard is not about you
Public tests measure general ability: math, code on generic problems, factual recall. Your task is usually much narrower and more specific: parsing a particular document format, classifying into your own categories, writing in your product's tone.
There is a second reason to distrust the table as-is. Popular benchmark datasets have been public for a long time, and there is a real suspicion that parts of them leak into the training data of newer models. A model can be "remembering" the answer rather than solving the problem fresh.
The third reason is the simplest: a leaderboard averages across thousands of unrelated queries. Your task is one specific type of request that repeats thousands of times in production. An average score on someone else's mix says nothing about that one case.
What an eval on your own tasks actually is
The idea is simple: pull twenty or thirty real examples from your product, fix what counts as a correct or acceptable answer for each one, run them through a few models, and compare on one metric.
This is not a research lab. It is a fast check that answers one question: which model and which prompt give the best result on my data for my money.
Twenty minutes to a first set
Step one, five minutes: collect twenty real queries your product has already seen, or write close approximations. Do not invent synthetic ones, use the actual phrasing users type.
Step two, five minutes: for each query, write down what a good answer looks like. Plain language is fine: "must state the amount and the date", "must not invent a product that doesn't exist". Formal metrics can come later, a short criterion is enough to start.
Step three, five minutes: write a small script that runs all twenty queries through the API with one fixed prompt and saves the answers. Pseudocode:
for item in dataset:
response = call_model(model_id, item.prompt)
save(item.id, model_id, response, tokens_used, latency)
Step four, five minutes: run the same set against a second model, a cheaper one, and a third candidate if you have one. Line the answers up side by side and mark by hand which ones pass your criterion.
How to score it instead of eyeballing it
Even a rough metric beats an impression. The simplest one: the share of examples where the answer passed your criterion. Twenty examples, fifteen passed, that is 75%.
Next add a second number, cost. Divide the total run cost by the number of examples to get a price per answer. Now you can compare "which model gives the quality I need for less money" instead of "which model feels smarter". The winner is often not the flagship but a tier below it paired with a slightly longer prompt.
If the task is error-sensitive, add a third metric: the share of cases where the model did not just miss slightly but confidently made something up. Those cases cost more than an ordinary inaccuracy.
Common mistakes in a homemade eval
A tiny set, three to five examples, gives you a random result: a model can get lucky or unlucky on specific phrasing. Twenty is a reasonable floor, thirty to fifty is safer.
Testing on invented examples instead of real user queries is a common trap. Synthetic prompts tend to be simpler than real ones, and the score comes out inflated.
Comparing models on different prompts is another trap. If each model gets its own hand-tuned prompt, it is unclear what you actually compared, the models or the quality of the prompt engineering.
When to refresh the eval
The set is not static. Update it when the input data format changes, when a new class of user query shows up, and when a new version of a model you already use ships. An old eval run against a new model version can lie in either direction.
Keep the example set and the response logs next to your project code, not in personal notes. Six months from now that is the only way to explain why this particular model got picked.
In short
A public leaderboard describes an average task in an average world. Your own eval on twenty real examples, priced per answer, describes your task and your budget. Twenty minutes of work gives you an answer a leaderboard never could.