5.1.2 · Lesson 2 of 14
Build Representative Evaluation Datasets
Build curated evaluation datasets that represent production queries, edge cases, and safety scenarios for GenAI quality gates.
WHAT
Build representative evaluation datasets for comparing prompts, models, and retrieval configurations.
HOW
- Include typical production questions, multi-part queries, and adversarial/red-team cases.
- Use Bedrock Model Evaluation with built-in or custom datasets for correctness, toxicity, and knowledge.
- Keep a fixed regression set so prompt changes can be compared fairly over time.
Key points
- Curated test datasets
- Edge cases and red-team prompts
- Automated vs human evaluation datasets
- Bedrock Model Evaluation datasets
- Representative production coverage
Exam focus
Evaluation datasets must represent real workloads — happy-path demos alone do not gate production prompt or model changes.