The problem
The awards program needed to evaluate applications with limited human review time. Historical judgments gave me a way to check the AI before using it to support award decisions.
My role
I built the judging panel and designed its evaluation against human experts. I investigated disagreements and tested whether the evaluation could distinguish sound judgments from convincing prose.
Checking the human reference
The human scores were concentrated at the upper end of the scale. Classical reliability estimates made the judges look less consistent than they were, so I examined the effect of that restricted range before using their judgments as a reference.
I used chance-corrected agreement to assess human reliability and compared the AI’s rankings with human consensus. Agreement on category winners was a separate measure.
The referees were rewarding how the answer was written
The initial evaluation favored the AI in 16 of 16 comparisons. I tested the referee models with fabricated pairs and found that they were rewarding fluent writing over sound judgment.
I changed the evaluation so the referees judged each side independently. The revised setup passed 24 of 24 revalidation checks before I ran the real study.
What the results showed
The production panel matched the human winner in 7 of 7 categories on 29 held-out applications. Human grading effort fell from 330 hours to 10 hours with people involved in the review.
The grading-hour comparison is based on my first-hand account of the workflow. The winner comparison comes from retrospective validation against historical human judgments.
Where the results stop
The overall ranking correlation with human consensus was 0.39. Matching category winners did not mean the panel agreed with humans on every score or the complete ranking.
The validation describes this panel on a held-out historical set. It does not establish that the same performance will hold for every future set of applications.