An engineer can abandon a good patch because the project changed. An AI judge that sees only the patch may treat that outcome as evidence of bad code. What would it take to learn from the team’s decisions without learning the wrong lesson?
In the AutoRefine preprint, Meta researchers give a second model the patch information the judge will see, plus the recorded outcome. That second model explains the outcome when it can ground a reason in the judge’s inputs. It can set an example aside when the evidence is missing. The researchers then revise the judge’s instructions using disagreements on the retained training examples, and select the revision on separate data. A hidden test or a decision to drop a feature can leave a training label the judge has little chance of explaining. AutoRefine preprint
On 17 changes from the current version of an internal performance-optimization agent, chance-adjusted agreement with merge and abandonment rose from 0.16 to 0.51 after recalibration. The new judge passed all eight changes that eventually landed, up from six. It marked five of the nine abandoned changes as failures, up from four. The older judge did better on a separate set of 16 changes from earlier workflows. The score measures agreement with one team’s workflow outcomes, with considerable uncertainty across those 17 changes. The study’s results and limits
When a patch stalls, give reviewers a way to distinguish a technical rejection from a scope change or a pipeline failure. Keep those notes with the evaluation set and check what evidence would have been available when the judge scored the change. If the judge could learn the reason only after the decision, set that example aside when tuning.