4.2.2 · Lesson 6 of 16
Throughput Under Concurrency
Improve GenAI throughput under concurrent load through parallel orchestration and capacity-aware design.
WHAT
Improve throughput when many users or jobs hit the GenAI path concurrently.
HOW
- Execute independent vector searches and tool calls in parallel.
- Use Step Functions parallel states for fan-out work.
- Combine with cascading/caching so not every concurrent request hits the largest FM.
Key points
- Parallel tool/retrieval calls
- Step Functions parallel branches
- Avoid serial bottlenecks
- Capacity vs concurrency
- Async where waiting is wasteful
Exam focus
Serialize only when dependencies require it — parallelize independent retrieval and tool calls to raise throughput under concurrency.