AutoRefine tests what code judges can learn from merges
Meta researchers improved an AI judge’s agreement with recent merge and abandonment outcomes in a small test, while the older judge still did better on earlier work.
As a leader: Measure both missed useful changes and passed weak changes before making a recalibrated judge a release gate.
As a practitioner: Record why a change was abandoned, and compare that reason with the information the judge actually receives.
arXiv / Shervin Shahidi, James Poon and colleagues ↗Research preprint