5.1.7 · Lesson 7 of 14
Detect Regressions After Content or Prompt Changes
Detect quality regressions after prompt, model, or knowledge-base changes using fixed eval sets and canaries.
WHAT
Detect regressions after content, prompt, or model changes before full production exposure.
HOW
- Canary & A/B via API Gateway or AppConfig (5–10% traffic).
- Monitor latency, thumbs-up/down, error rates, faithfulness, truncation.
- Re-run fixed offline eval sets after Knowledge Base or prompt updates.
Key points
- Fixed regression datasets
- Canary / A/B 5–10% traffic
- Monitor latency and feedback
- TruncationCount metrics
- Block regressions before full rollout
Exam focus
Route a small percentage of live traffic to new prompt/model versions and watch metrics before full promotion.