4.3.6 · Lesson 16 of 16
Feed Production Signals into Evaluation Loops
Use production token, failure, and latency signals to drive continuous optimization of prompts, models, and caches.
WHAT
Diagnose GenAI-specific failure modes and feed production signals into ongoing optimization.
HOW
- CloudWatch Logs Insights on prompt-to-token ratios and max-token truncation.
- X-Ray for multi-service agent bottlenecks.
- Update cascading thresholds, cache keys, and prompt compression based on findings.
Key points
- Logs Insights token analysis
- Truncation diagnosis
- Production → eval feedback
- Cost driver identification
- Continuous optimization loop
Exam focus
Query invocation logs to find cost drivers and truncations, then feed those findings back into prompt, cascade, and cache design.