It is always us-east-1

Late on Sunday, Oct 19, 2025 (11:48 PM PDT), a disruption in AWS N. Virginia (us-east-1) triggered a chain reaction across core services. For users, this kind of event shows up as the stuff you feel immediately: logins failing, requests timing out, apps acting “half-up,” and brand-new capacity refusing to come online.

The “shape” of the incident

Three systems matter most to understand the blast radius:

  • DynamoDB: a DNS automation race condition broke endpoint resolution.
  • EC2: a dependency cascade led to lease timeouts, launch failures, and recovery congestion.
  • Network Load Balancer (NLB): health check failures increased connection errors for some load balancers.

Timeline (PDT)

  • 11:48 PM (Oct 19) → 2:40 AM (Oct 20)
    DynamoDB elevated API errors; new connections failed due to endpoint resolution issues.

  • 11:48 PM (Oct 19) → 1:50 PM (Oct 20)
    EC2 APIs saw errors/latency; new instance launches failed for a long window. Existing instances stayed healthy.

  • 5:30 AM → 2:09 PM (Oct 20)
    NLB connection errors increased for a subset of load balancers.

1) DynamoDB: when DNS automation goes wrong

What users saw

From 11:48 PM to 2:40 AM, DynamoDB customers (and AWS services that depend on DynamoDB) had trouble establishing new connections and saw elevated API error rates.

What actually broke

DynamoDB depends heavily on DNS as part of how it scales, isolates faults, and routes traffic efficiently. That means a lot of DNS records and automation to keep them correct.

The root cause was a latent race condition in DynamoDB’s automated DNS management that produced an incorrect empty DNS record for the regional endpoint:

  • dynamodb.us-east-1.amazonaws.com

Why the system didn’t self-heal

DynamoDB’s DNS automation is split for availability:

  • DNS Planner generates DNS “plans” (which load balancers should serve an endpoint, with what weights).
  • DNS Enactor applies those plans to Route 53, running redundantly across three AZs.

The failure chain (condensed):

  1. One Enactor became unusually delayed while applying updates.
  2. The Planner kept generating newer plans.
  3. Another Enactor applied a new plan quickly and initiated cleanup (deleting much older plans).
  4. The delayed Enactor eventually applied an older plan over the newer one because its “is this plan newer?” check was stale by the time it got to the endpoint.
  5. Cleanup then deleted that old plan while it was active, removing IPs and leaving the system in a state that blocked subsequent automated fixes—requiring intervention.

The key lesson: cleanup logic + concurrency + stale checks can create a “delete the thing that’s currently live” scenario.

2) EC2: dependency cascades and recovery congestion

What users saw

From late Oct 19 into Oct 20, customers experienced increased EC2 API errors/latencies and new instance launch failures, returning errors like:

  • “request limit exceeded”
  • “insufficient capacity”

Importantly: existing instances launched before the event remained healthy. The big pain was new capacity.

What actually broke

To launch instances, EC2 relies on internal control-plane subsystems that depend on DynamoDB-backed state. A key one is:

  • DropletWorkflow Manager (DWFM): manages the underlying physical servers (“droplets”) and maintains leases to track and coordinate state.

When DynamoDB was unhealthy, DWFM state checks failed and leases began timing out. After DynamoDB recovered, DWFM attempted to re-establish leases across a huge fleet. The recovery effort itself became the problem: with enough timeouts and enough queued work, DWFM entered a congestive-collapse condition—too much backlog, too slow to catch up, causing repeated timeouts.

How it was mitigated

Engineers throttled work and performed selective restarts to clear queues and reduce processing time, then gradually relaxed throttles as the system stabilized—ultimately restoring normal operation.

The key lesson: recovery is its own failure mode. Systems need explicit design for “coming back,” not just “staying up.”

3) Network Load Balancer: health checks as a single point of pain

What users saw

From 5:30 AM to 2:09 PM, some customers saw increased connection errors on NLBs.

What actually broke

NLB depends on continuous health checks and fleet coordination to route connections only to healthy nodes. During a complex regional recovery, health check failures and instability in the NLB environment caused a subset of load balancers to behave as if healthy capacity was missing—leading to connection errors.

The key lesson: health systems are production systems. If health evaluation degrades, routing degrades.

What to take away (engineering lens)

  • DNS is a control plane, not “just plumbing.” If endpoint resolution breaks, everything above it inherits the failure.
  • Latent race conditions don’t show up until timing gets weird. High delays + concurrency + cleanup can unlock rare bugs.
  • Dependency depth matters. A DynamoDB control-plane issue cascaded into EC2 launch mechanisms and beyond.
  • Design for recovery: throttles, backpressure, bounded queues, and “safe mode” operational paths often decide whether recovery is minutes or hours.