← AIP-C01 Study Guide
Domain 5 · 11% of the exam · 14 lessons
Testing, Validation, and Troubleshooting
Evaluate foundation model outputs and GenAI systems, implement quality assurance and deployment validation, test retrieval and agent performance, and troubleshoot content, FM integration, prompting, retrieval, and prompt-maintenance issues.
Tasks & skills
5.1
9 skillsImplement evaluation systems for GenAI
- 5.1.1Evaluation Frameworks & Automated Metrics
- 5.1.2Build Representative Evaluation Datasets
- 5.1.3Score Grounding, Usefulness, and Safety
- 5.1.4Validate Retrieval and Agent Task Success
- 5.1.5Gate Deployments with Quality Thresholds
- 5.1.6Compare Model and Prompt Versions Fairly
- 5.1.7Detect Regressions After Content or Prompt Changes
- 5.1.8Incorporate Human Review into Evaluation
- 5.1.9Report Evaluation Results to Stakeholders
5.2
5 skillsTroubleshoot GenAI applications