Scenario Library

A fully serverless API on AWS: an Application Load Balancer triggers Lambda functions on every request — no servers to manage and you only pay per invocation. DynamoDB handles storage at any scale. Watch for cold-start latency spikes when Lambda containers are not yet warm after a period of low traffic.

An idle OCI compartment still billing ~$90/month: reserved public IPs, block volume backups, a 10 Mbps load balancer minimum, and boot volumes under stopped compute and database instances. Notably cheaper than the other clouds — OCI's NAT Gateway and Service Gateway are free. Remove idle resources to watch the residual bill drop.

Simulate an event burst that overwhelms a GCP Pub/Sub subscription: the unacknowledged message backlog surges, subscriber VMs can't keep pace, and message delivery latency climbs well past acknowledgement deadlines triggering redelivery storms. Explore how adding more GCE subscriber instances and tuning flow-control settings restore normal throughput.

Simulate a burst of inbound orders that overwhelms an Azure Service Bus queue: the active message count surges past the queue's throughput unit capacity, consumer VMs fall behind, and end-to-end processing latency explodes as messages approach their lock expiry and are abandoned back to the queue. Explore how scaling out consumer instances, enabling auto-forwarding to a dead-letter queue, and upgrading to a Premium namespace restore normal throughput.

Learn how autoscaling groups respond to traffic spikes and CPU thresholds

On July 24, 2026, the us-west-2 (Oregon) regional outage took down hundreds of single-region deployments for ~80 minutes — disrupting Apple Pay, DoorDash, Reddit, Hulu, and PlayStation Network. This companion scenario shows the resilience pattern that kept multi-region architectures standing: Route 53 Latency Routing with health checks. The scenario opens mid-failover — us-west-2 ALB is already degraded and Route 53 has begun draining DNS toward us-east-1 (N. Virginia). Watch how a 30-second health-check TTL shapes the failover window: every DNS resolver that hasn't refreshed yet still sends users west, where they see errors. Understand why a 60-second TTL (not the default 300 s) and aggressive health-check intervals (10 s, 3 failures) are the difference between a 2-minute and a 12-minute outage. The us-east-1 EC2 fleet is autoscaling — observe it absorb the redirected load as west drains.

Watch a live Multi-AZ failover unfold step by step. Traffic ramps across two app servers behind an ALB, pushing connection pressure on RDS Primary until it overloads at step 15. The replica auto-promotes, latency spikes, error rates climb — then watch the system recover. A real AWS RDS Multi-AZ failure arc in under 60 steps.

Simulate a Product Hunt-style launch surge on a single server. Find the exact upload concurrency where your transcoding queue backs up — before it happens in production.

On July 4, 2023 at 18:47 US/Pacific, Cloud Spanner went down globally for 2 hours and 16 minutes. This was not a zone failure. All three replicas — the leader in us-central1, the read-write replica in us-east1, and the witness in us-east4 — failed simultaneously because of an internal software bug in Spanner's serving layer. Paxos was intact and working correctly. The Paxos replication logs were consistent. No data was lost. But none of that mattered, because the bug was in the code that accepts client RPCs and routes them to the correct Paxos group — and that code ran identically on all three replicas. There was no healthy replica to elect as a new leader. Adding more replicas would not have helped. Switching regions would not have helped. The only fix was a rollback of the bad binary deployment, which Google engineers completed after 2h16m. This scenario opens at the moment the outage began: all three Spanner replicas are returning UNAVAILABLE, GKE pods are alive but every database call is failing with 503s, and the Global LB is routing correctly — it just has nowhere healthy to route writes to. Your learning goals: (1) Distinguish 'zone/hardware failure that Paxos handles automatically' from 'software bug that runs on all replicas identically and Paxos cannot fix'; (2) Learn the correct incident response for this class of failure — circuit breakers to stop retry storms, local caching to preserve read availability, write-queue to replay mutations on recovery, SLO alerting on 5xx rate not just p99 latency; (3) Understand why multi-region Spanner reduces but does not eliminate the need for application-level resilience patterns.

A network degradation event has hit your us-ashburn-1 region, taking down your primary Load Balancer and putting your Autonomous Database primary under stress. OCI Traffic Management has already activated its FAILOVER steering policy and is routing new connections to your us-phoenix-1 backend — but there are three OCI-specific lessons worth understanding before you accept the recovery as complete. First: how OCI Traffic Management steering policies differ from Route 53 health checks. Traffic Management uses an ordered answer pool: it tries each pool in priority order and skips pools whose health-check endpoint is returning failures. The health monitor polls on a configurable interval (minimum 10 seconds for HTTP, 30 seconds for HTTPS); compare this to AWS Route 53 which polls every 10 seconds (standard health checks) or 30 seconds (basic). Both products are DNS-based, so clients must wait for TTL expiry before they resolve the failover IP — you can lower the DNS TTL on your OCI Traffic Management FQDN to reduce this window. Second: Autonomous Data Guard sync and how to promote the standby. The standby Autonomous Database in us-phoenix-1 continuously receives redo log shipments from the primary. You have two promotion paths: (1) SWITCHOVER — use this when the primary is reachable; it drains all in-flight transactions, promotes the standby to primary, and demotes the old primary to standby in a single consistent operation with zero data loss; invoke it from the OCI Console (Autonomous Database → More Actions → Switchover) or via CLI (`oci db autonomous-database switchover`). (2) FAILOVER — use this when the primary is completely unreachable; the standby applies all received redo and opens read-write; any redo not yet shipped from the primary (the replication lag at the moment of failure) represents your RPO; after failover the old primary is a 'former primary' that must be reinstated as a new standby before you can switchover back. Third: the application connection string problem. Unlike AWS RDS Multi-AZ (single DNS endpoint that transparently updates) or Azure SQL (automatic listener), OCI Autonomous Database standby has its own wallet and connection string. Your application must be pre-configured to use the standby endpoint, or you must update the connection string in your application configuration after promotion. Fast-Start Failover (FSFO) can automate the promotion decision when the primary is unreachable for a configurable observer timeout, but the application connection string update is still a manual step unless you build it into your deployment configuration.

Your NYC3 datacenter is experiencing a network degradation event. The Load Balancer health check is failing, and your primary Managed PostgreSQL cluster is under stress. You need to reroute traffic to your SFO3 standby Droplet — but this is where DigitalOcean's HA model diverges from AWS, GCP, and Azure in ways that matter for your architecture decisions. This scenario has three lessons. Lesson 1: DigitalOcean Floating IPs are region-scoped, not global-anycast. A Floating IP is a public IP tied to a single datacenter (nyc3). You can reassign it instantly to any Droplet in the same datacenter via API (`doctl compute floating-ip-action assign <floating-ip> <droplet-id>`) — but you cannot point a nyc3 Floating IP at an sfo3 Droplet. Cross-region compute failover on DigitalOcean therefore requires a DNS change: update your domain to point from the nyc3 Floating IP (or Load Balancer IP) to the sfo3 Droplet IP. DNS propagation is TTL-bounded — if your TTL was 300 seconds before the incident, clients can be unreachable for up to 5 minutes. Compare this to AWS Global Accelerator or Azure Front Door (both anycast — no DNS change, sub-second failover) or GCP Global Load Balancer (also anycast). If you need instant cross-region compute failover on DigitalOcean, the workaround is to set a very low DNS TTL (60 s or lower) before an incident and use a third-party DNS provider like Cloudflare with sub-second propagation. Lesson 2: DigitalOcean Managed PostgreSQL has excellent within-region HA — automatic primary/standby failover, promoted in under 60 seconds with zero manual intervention. But there is no built-in cross-region standby. The 'SFO3 standby' PostgreSQL cluster in this scenario is maintained via manual pg_logical replication that you configured yourself: a publication on the nyc3 primary and a subscription on the sfo3 cluster. pg_logical is asynchronous, so the sfo3 cluster lags the nyc3 primary by the replication delay. To promote the sfo3 cluster to writable primary you must: (1) confirm replication lag is acceptable, (2) disable the subscription on sfo3 (`ALTER SUBSCRIPTION sub_name DISABLE`), (3) update your application connection string to point to the sfo3 endpoint, (4) optionally set up reverse replication from sfo3 back to nyc3 for when you want to fail back. Compare this to AWS RDS Multi-AZ (automatic DNS failover, same endpoint), Azure SQL Active Geo-Replication (`az sql db replica set-primary`, ~30 s), and OCI Autonomous Data Guard (SWITCHOVER in the console). Lesson 3: When to choose DigitalOcean despite these limitations. DO's pricing is genuinely competitive — a 2 vCPU / 4 GB Droplet costs $26/mo vs $60–80/mo for equivalent AWS/GCP/Azure instances. DO's developer experience is simpler, its App Platform handles single-region PaaS deployments with built-in horizontal scaling, and for applications where a single region is sufficient (most internal tools, early-stage startups, read-heavy workloads with acceptable RPO), DO's within-region HA is robust. The cross-region gap only matters when you need sub-minute global failover with automatic database promotion — and that is a legitimate requirement that pushes you toward the big three.

On July 24, 2026, a regional failure in us-west-2 (Oregon) degraded routing for dozens of popular services for ~80 minutes — ALBs began dropping packets, EC2 fleets lost internet connectivity, and retrying application servers flooded RDS with connection attempts. Apple Pay, DoorDash, Reddit, Hulu, and PlayStation Network were among the consumer services disrupted. This scenario opens mid-crisis: three EC2 web servers are already critical, ElastiCache has lost coherence, and RDS is absorbing a connection flood. Watch the error rate climb as EC2 loses egress, see the cache miss storm after ElastiCache degrades, and observe how the RDS read replica auto-promotes under primary pressure. The crisis is structural — not traffic-driven — so adding more servers cannot help. Around step 45, AWS restores the regional routing fabric; watch the error rate drop and servers work through their recovery cooldowns in the final stretch.

Experience a hybrid architecture spanning AWS and OCI with cross-cloud connectivity

CloudFront sits in front of your application and caches static content at edge locations worldwide. Only cache misses and dynamic API requests reach the Application Load Balancer, reducing backend load by 60–80% during traffic spikes. EC2 handles the dynamic compute tier while RDS stores persistent data.

Deploy a simple web application on Oracle Cloud Infrastructure with load balancing

Observe what happens when ElastiCache for Redis crashes mid-traffic: the cache eviction storm forces every request through to the database, connection pools saturate, and latency spikes until the cache warms back up. Explore resilience patterns like circuit breakers and staggered cache warm-up.

See how a Redis cache and SQS message queue reduce database load and smooth out traffic spikes in a microservices architecture

A simple, budget-friendly web app on DigitalOcean: a Load Balancer routes visitors to a single Droplet, which stores data in a Managed PostgreSQL database. A great starting point for developers new to cloud infrastructure who want predictable, low monthly bills.

Run a containerized app on GKE Autopilot, accelerated by a Memorystore cache and buffered by Cloud Pub/Sub for async processing

Your React SPA is hosted on S3, served through a single CloudFront distribution. You added a second CloudFront behavior to route /api/* requests to an ALB — clean and simple, until your API started returning 200 OK for every request that should have been a 404 or 403. The culprit is CloudFront's global SPA error-page rule: when you set 4xx → index.html with 200 status to make client-side routing work, that rule applies to every behavior in the distribution, not just the SPA. API 404s and 403s from the ALB are silently rewritten to 200 before the browser ever sees them. The crisis is already live — CloudFront and the ALB are flagged warning not because of load but because of this routing logic conflict. The fix is a subdomain split (a separate API subdomain that bypasses CloudFront entirely) or per-behavior CloudFront error pages, but the latter is not yet supported for ALB origins. Explore the resources to understand the exact failure path.

The clean fix for the CloudFront 404-trap: stop routing API traffic through CloudFront at all. The React SPA is still on S3 behind CloudFront — its error-page rewrite rule (4xx → index.html 200) now only applies to SPA routes, which is exactly what you want. API traffic goes to api.example.com, a separate ALB subdomain with no CloudFront in front of it. Real 404s and 403s from the application travel directly from the ALB to the browser with their status codes intact. Explore this architecture to understand why the subdomain split is the most reliable solution, and note what the alternative — per-behavior CloudFront error pages — requires if you ever need to keep everything on a single distribution.

A two-pool AKS cluster — general (2–8 nodes) and api (1–6 nodes) — runs in East US Zone 1 behind an Azure App Gateway, backed by Azure SQL in Zone 2 and Azure Cache for Redis (85% hit rate). All resources start healthy under a steady wave baseline of ≈900–2100 RPS. Nothing self-triggers: simulate a multi-fault cascade yourself by injecting faults via the chaos API or the /failures endpoint. Note the naming difference — the chaos API uses `zone_outage` while /failures uses `az_outage` for the same event type. Injecting a Zone 1 outage simultaneously impacts the App Gateway, both AKS pools, and the Redis cache while the SQL database in Zone 2 survives — observe how zone failures dominate cascade grades and how connection pressure spikes as all cache reads fall through to the database.

On July 16 2026, AWS CloudFront began returning errors globally — every edge location stopped serving cached responses. This scenario reproduces the origin overload that followed: with a 75% cache-hit rate suddenly gone, ALB and EC2 web servers absorb 4× their normal request volume and climb into the warning zone. RDS connection pressure rises as every user request now requires a full database round-trip. The crisis is already live — explore what protections (rate limiting, static failover pages, graceful degradation) would have kept your origin standing.

On June 2, 2019, a misconfigured traffic-engineering update caused extreme congestion across Google's eastern US backbone — and it took down both Cloud Load Balancing AND Cloud DNS for ~4h. Health-check probes could not reach any backends, DNS propagation stalled, and customers behind a single regional ALB were completely unreachable for the full outage window. Multi-region deployments using the Global HTTP Load Balancer fared better for a specific reason: because the Global LB uses anycast (not DNS), its edge PoPs began finding alternate backbone paths as congestion partially cleared — the same VIP address started routing again through less-congested paths before DNS could recover. This scenario opens mid-incident: the Global LB and Cloud DNS are offline, us-central1 is completely unreachable, and us-east1 is only getting a trickle of traffic through partially-recovered anycast paths. Your job: understand why the Global LB's anycast architecture gave multi-region setups an earlier recovery window even when the LB itself was degraded — and what additional protections (Cloud Armor failover policy, multi-CDN, pre-warmed external endpoints) would give you a reliable cold-start path when the entire routing layer goes down.

A container-native stack on DigitalOcean: a Load Balancer fronts a DOKS cluster that runs your app pods. A Managed Redis cache absorbs read traffic, and a Managed Kafka queue decouples background jobs from the web tier — all with zero infrastructure to patch.

Explore an AKS-hosted microservices app that offloads read traffic to Azure Cache for Redis and fans out async work through Azure Service Bus

Simulate a burst of inbound events that overwhelms the worker fleet: the SQS queue depth climbs into the thousands, workers fall behind, and end-to-end processing latency skyrockets. Watch how adding consumers and applying dead-letter queues restore stability.

On October 9, 2025, a crash loop in Azure Front Door's data plane took its global edge network offline for ~8h10m (tracking ID: QNBQ-5W8). Unlike a backend failure — where Front Door detects unhealthy health probes and routes around the problem — this crash loop made Front Door itself the failure. Both the East US and West US App Service origins were completely healthy and ready to serve traffic, but the global proxy layer connecting users to them was gone. This is the failure mode that Front Door does not protect you from, because Front Door cannot route around its own crash. This scenario opens at peak impact: Front Door is offline globally, both App Service backends have near-zero incoming traffic (not because they're broken, but because the proxy layer isn't delivering requests), and Azure SQL is under minimal load. Your two learning goals: (1) Understand the difference between 'backend failure that Front Door routes around' vs 'Front Door itself is the single point of failure,' and (2) Learn why Traffic Manager pointing directly at App Service origins — bypassing Front Door entirely — is the resilience layer you need when the proxy itself goes down. Traffic Manager is DNS-based (TTL-bounded propagation) but is a different service from Front Door and would have remained available during the QNBQ-5W8 event.

Everything is stopped, yet you're still paying ~$310/month. Explore what keeps billing on AWS when traffic is zero — a NAT Gateway, interface VPC endpoints, detached Elastic IPs, an idle ALB, EBS snapshots, and the disks under stopped EC2/RDS instances. Remove idle resources one by one and watch the residual bill drop.

A decommissioned GCP project that still bills ~$120/month at zero traffic: Cloud NAT's fixed gateway fee, Private Service Connect endpoints, reserved-but-unused static IPs, persistent disk snapshots, an idle load balancer forwarding rule, and the disks under stopped VMs and Cloud SQL. Remove each idle resource to watch the residual bill drop.

An Azure resource group after 'shutting everything down' — still ~$175/month: a NAT Gateway, private endpoints, unassociated static public IPs, managed disk snapshots, a Standard Load Balancer's fixed fee, and the disks under deallocated VMs and stopped Azure SQL. Remove idle resources to watch the residual bill drop.

A wound-down DigitalOcean project still billing ~$90/month: reserved IPs not assigned to Droplets, volume snapshots, an idle $10/mo load balancer, and volumes under powered-off Droplets and a stopped managed database. DigitalOcean has no NAT Gateway or private endpoint product — its zombie surface is smaller but real. Remove idle resources to watch the residual bill drop.