A scorecard can put the model names in order while leaving the conditions of the contest off the page. Change the wording, interface, or judge and that order may become less stable. To use the comparison for a real decision, keep the conditions attached to the scores. They are part of what was measured.
One new study of tool-augmented comparisons examines six open models across five domains and five wordings. It finds settings where tools improve accuracy while making pairwise comparisons more sensitive to wording. A second study of multilingual LLM judges reports that prompting effects depend on the judge, task, and language, with some effects reversing across configurations.
Neither paper establishes that evaluation is futile. They give you a practical question to put beside a result: would the decision survive reasonable variations in how the same work is presented and judged? Keep the configuration with the score so you can test that question instead of asking a model name to explain the whole result.
Preserve the comparison contract
Record the task set, input representation, tool interface, prompt, model identifier, scoring rules, and judge configuration. Where a provider does not offer an immutable version, record the evaluation date and the resulting limitation. A model name alone is not enough to reconstruct what happened.
Keep changes to that contract visible. If one candidate requires a different prompt or tool description, evaluate and label the resulting configuration rather than presenting the result as a property of the model in isolation. The difference may be entirely reasonable, but it belongs in the record.
When the score moves later, return to that record. Did the model change, did the prompt improve, did the judge interpret the answer differently, or was an integration repaired? Each possibility can teach the team something useful. Keeping the changes distinguishable gives the movement an explanation the next person can investigate.
Test reasonable alternative wordings
Choose a small set of wordings that preserve the task and the available evidence. Avoid quietly changing the requirements while calling the result a prompt-sensitivity test. Have a reviewer confirm that the variants still ask the same substantive question.
Measure whether the model ordering changes as well as how much individual scores move. If the procurement or deployment decision depends on a narrow ranking that reverses easily, the uncertainty deserves a place in the decision record.
Do not automatically search for the wording that makes the preferred candidate win. The purpose is to understand the stability of the comparison under the ways people may actually express the task. Preserve unfavorable variants and investigate them alongside the favorable ones.
Inspect how tools enter the result
Tools can change the information available to a model and the decisions required to obtain it. A successful tool-supported answer therefore reflects more than reasoning over a fixed passage. Record whether the model called a tool, what it received, and what it did with the result.
The tool-comparison paper makes this separation particularly useful. Its findings vary across interfaces and instructions rather than supporting one universal claim that tools increase or decrease stability. An enterprise evaluation should retain those distinctions instead of flattening them into a slogan.
Use matched cases where possible. Compare a result supplied directly with the same result obtained through a tool, while keeping the substantive evidence constant. Differences can identify an interface or decision-to-call effect that a single accuracy number would conceal.
Validate the judge in the intended setting
Take the judge into the setting where its decisions will be used. A prompt that works acceptably on a task or language may behave differently elsewhere. Ask people qualified to assess the work to review a sample, then examine the disagreements in the languages and categories the application actually serves. That gives the judging process a local reference.
Keep subgroup results available when they affect an operating decision. A favorable aggregate can conceal a reversal or a weak category. Where sample sizes are small, label the uncertainty rather than treating every observed difference as a settled ranking.
Also distinguish changes in the system being evaluated from changes in the judge. If both move together, run an overlap comparison that helps identify which change explains the result. Otherwise an apparent quality improvement may partly reflect a different scoring instrument.
Set aside a small decision set that is not used to select prompts or tune the judge. Run the chosen configuration there after development ends, preserving inconvenient cases. If the ranking changes, investigate whether the difference comes from the sample, the wording, or the evaluator. This separate check can reveal a configuration that fits the development exercise better than the work it is meant to represent.
Report what the evidence supports
A useful scorecard can include the central result, its important variation, and the conditions under which the team would revisit the choice. It does not need to compress every question into one winner. Some applications may justify a bounded pilot or different configurations for different kinds of work.
Choose a comparison that currently drives a real decision. Keep its configuration, try a few equivalent wordings, and review meaningful disagreements with the judge. Then put the scorecard back in front of the decision maker, with the conditions and variation visible. You can explain how much weight the ranking deserves and which questions still need work before a choice is made.