Model Evaluation & RLHFIntermediateTier 2 · $99
RLHF Preference Ranking
Ranking model responses on a rubric, including ties, honesty and rater agreement.
3 modules130 minutes of material10 exam questions · 80% to pass
Syllabus
01Ranking on a rubric
45 min- Apply criteria in a fixed order
- Justify a tie
02Honesty and helpfulness
45 min- Separate a confident wrong answer from a hedged right one
- Control for length
03Agreement and bias
40 min- Measure agreement between raters
- Document a disagreement
Assessed concepts
The exam tests these directly. Definitions are summarised here; the exact answer key stays on the server.
Preference ranking
Tie handling
Helpfulness versus honesty
Rater agreement
Rater bias control