5.1.6 · Lesson 6 of 14
Compare Model and Prompt Versions Fairly
Compare candidate models and prompt templates fairly using managed evaluation and bias-mitigated judging.
WHAT
Compare candidate foundation models and prompt versions without evaluation bias.
HOW
- Use Amazon Bedrock Model Evaluation (Automated or Human).
- For LLM-as-a-Judge pairwise compares, apply position swapping and average scores.
EXAM TRAP
LLM-as-a-Judge is not immune to scoring bias. Position swapping is the high-yield mitigation.
Key points
- Bedrock Model Evaluation
- Automated vs Human Evaluation
- Pairwise position swapping
- Average A/B and B/A scores
- Fair version comparison
Exam focus
LLM judges have position bias — neutralize by evaluating A vs B and B vs A, then averaging.