4.2.4 · Lesson 8 of 16
Balance Quality Against Speed and Cost
Tune decoding and routing so quality floors are met without paying for unnecessary creativity or oversized models.
WHAT
Balance response quality floors against speed and cost using decoding parameters and model choice.
Parameters
- Temperature (0.0–1.0): Low for deterministic tasks; higher for creative writing.
- Top-P / Top-K: Restrict to a nucleus of high-probability tokens to keep coherence.
EXAM TRAP
Increasing temperature does not fix arithmetic hallucinations — lower it toward 0.0.
Key points
- Temperature for determinism vs creativity
- Top-P / Top-K pruning
- Quality floors vs cost
- Right-size models
- Task-appropriate decoding
Exam focus
For math/code/extraction use low temperature (0.0–0.2); raising temperature increases randomness and can worsen numeric hallucinations.