Project collection

LLM evaluation

I use research methods to check how well AI systems work. That includes testing the evaluation itself when the results look too favorable.

AI ENGINEERING / RESEARCH

Evaluating an
AI judging panel

I built an AI judging panel for an industry awards program and compared it with human experts. The evaluation favored the AI so consistently that I wanted to check whether the evaluator was working.

Read the case study

Claude Code · LLM evaluation · Python · Study design

WINNER AGREEMENT WITH HUMAN EXPERTS

7/7

category winners matched
on held-out applications

✓

Tested retrospectively on 29 held-out applications.
The winners matched; individual scores still differed.

330 → 10human grading hours

↗

How the workflow connects

Inputs

APIClickUp API
Application attachments and award category
FilesApplication documents
Text and rendered page images

Processing

CodePython
Ingest and check application content
AIOpenAI API
Judge panel produces scores and feedback

Outputs

DocumentsReviewer briefs
Evidence and scores for review
DocumentsPersonalized feedback
Applicant reports and winner case studies

People review the evidence and make the final award decisions. Human scores and referee models are used in the separate validation study.

LLM evaluation

Evaluating anAI judging panel

Evaluating an
AI judging panel