Evaluation harness for our LLM features

Project Description

We ship three LLM-backed features and we find out we have made them worse when a customer tells us. We want a proper evaluation harness: a golden set, automated scoring where it can be automated, human review where it cannot, and it running in CI so a prompt change shows its effect before merge. We are more interested in someone who has learned what does not work here — LLM-as-judge without calibration, for instance — than in someone who has read about it.

Skills Required