Design resilient AI systems

Design for model/service disruption.

Possible failures:

• temporary model failure
• regional unavailability
• throttling
• network failure
• provider outage

GenAI applications can become unavailable if they depend on a single model endpoint.

Possible resilience patterns:

Request
   ↓
Primary Model
   ↓ failure
Fallback Model / Region
   ↓ failure
Graceful Degradation

Use:

• retries where appropriate
• circuit breakers
• cross-Region inference
• alternate deployment regions
• fallback models
• cached responses
• non-GenAI fallback workflows

• Amazon Bedrock Cross-Region Inference
• AWS Step Functions
• AWS Lambda
• Amazon API Gateway
• Amazon CloudWatch

Healthy
  ↓
Failures increase
  ↓
Open circuit
  ↓
Stop sending traffic to failing dependency
  ↓
Recovery check
  ↓
Close circuit

If the FM is unavailable:

AI unavailable
     ↓
Use deterministic FAQ
     OR
Return cached answer
     OR
Route to human
     OR
Use fallback model

Retries alone are not a complete resilience strategy.

Repeatedly retrying a failing dependency can make an outage worse.

---