AWS Outage Caused by DynamoDB DNS Race Condition and Cascading Failures

Amazon Web Services suffered a major outage on October 19-20, 2025, starting in the US-EAST-1 region due to a race condition in DynamoDB’s automated DNS management system. The system includes a DNS Planner that monitors load balancers and generates DNS plans, and a DNS Enactor that applies changes via Route 53. A latent defect caused one Enactor to experience high delays while the Planner continued creating new plans. A second Enactor applied a newer plan and initiated cleanup, deleting the delayed plan’s DNS records just as the first Enactor finished, resulting in an empty DNS record for the DynamoDB regional endpoint.

This DNS failure prevented customer and internal AWS traffic from reaching DynamoDB, triggering widespread disruptions. The DropletWorkflow Manager (DWFM), responsible for EC2 instance leases, failed state checks, preventing new leases and instance state changes. After DynamoDB recovered at 2:25 AM PDT, DWFM attempted to re-establish leases across the massive EC2 fleet, causing congestive collapse that required manual intervention until 5:28 AM PDT.

Subsequent network configuration backlogs delayed EC2 instance networking, affecting Network Load Balancers and causing health check failures. This impaired dependent services including Lambda, ECS, EKS, and Fargate. The outage lasted hours, impacting major websites, smart home devices, and government services, with potential economic damage in the hundreds of billions of dollars. AWS has disabled the DNS automation globally until safeguards are implemented and is reviewing ways to prevent recurrence and improve recovery times.