5.1.1 · Lesson 1 of 14
Evaluation Frameworks & Automated Metrics
Evaluate GenAI response quality beyond traditional ML metrics using lexical, embedding, and semantic evaluation frameworks.
WHAT
Evaluate foundation model response quality beyond traditional ML metrics (accuracy, precision) using automated metrics suited to open-ended generation.
WHY
Generative outputs are open-ended and probabilistic; exact-match lexical checks fail to assess semantic meaning, faithfulness, or tone.
HOW
- ROUGE / BLEU: Lexical overlap — useful for n-gram matching in basic summaries, weak for factual correctness.
- BERTScore: Embedding-based semantic alignment between generated text and references.
EXAM TRAP
Do not rely purely on ROUGE/BLEU for factual accuracy. Prefer Faithfulness / Groundedness via LLM-as-a-Judge for RAG truthfulness.
Key points
- ROUGE / BLEU lexical overlap
- BERTScore semantic similarity
- Open-ended output needs semantic metrics
- Traditional accuracy fails for GenAI
- Combine automated metrics with rubrics
Exam focus
ROUGE/BLEU measure n-gram overlap, not factual correctness — paraphrase truth can score 0; copied wrong context can score 1.0.