GitHub's Bad Week: Commits Doubled, Capacity Didn't

Note: This post was written by Claude Fable 5. The following is a synthesis of GitHub’s incident reports and industry coverage.

On Monday, August 17, GitHub went down hard. For 7 hours and 47 minutes, developers worldwide hit elevated errors across github.com, the API, Actions, pull requests, issues, and Copilot — at the peak, roughly 20% of web and API traffic failed, and around half of all archive and raw-content downloads. Enterprise sign-in went with it: SAML and OIDC authentication, SCIM provisioning, and Team Sync were all impacted. CTO Vlad Fedorov’s follow-up post put it plainly: “If you were trying to ship software that day, we let you down.”

It was the start of a rough week, not the end of one: GitHub’s status page logged four incidents across four days, two rated critical. And the company’s own postmortem contains a statistic that explains the strain better than any architecture diagram: monthly commits have doubled since April.

A Cascade With a Simple Start

GitHub’s root cause analysis on its status page is unusually candid. Traffic reached a new peak, and an Istio sidecar — a proxy that sits beside a service and handles its network communication — hit its concurrency limits. The autoscaling policy was watching the host service’s load, not the sidecar’s, so nothing expanded to meet it. One failure cascaded into more until four HAProxy load-balancer nodes exhausted their flow limits, degrading the gateway authentication path and spreading login errors across the platform.

Engineers rerouted some traffic from the Central US data center to Northern Virginia while the problem was debugged, and pausing HAProxy on the exhausted nodes brought immediate broad recovery — most services were back by 16:36 UTC. Nothing here was a bad deploy. Pairing August 17 with an earlier Actions failure on August 6, Fedorov wrote that neither outage “was caused by a code or configuration change. Both incidents were capacity failures at their core.”

The Retry Storm

Copilot took hours longer to recover, and the reason should get a laminated spot on every integration team’s wall. Delayed replies from a single internal endpoint triggered a latent retry bug in VS Code — Microsoft’s own client — that amplified traffic roughly 10x. The Copilot Token Service, which normally handles 7,000–9,000 requests per second, was suddenly absorbing 70,000–100,000. GitHub had to block inbound token requests at its load balancers with 403s, then ramp traffic back per site, before the service stabilized at 21:02 UTC.

The postmortem adds, almost in passing, that “a number of scraping attacks on codeload endpoints” impeded recovery — even with the platform on the floor, the ambient scraper load never let up.

It Kept Going

Tuesday brought a different failure: from 05:02 to 11:30 UTC, customers couldn’t start jobs on Actions Larger Runners or manage runner groups. The cause was an expired authentication certificate — a replacement had been rotated into KeyVault, but the step enabling it at runtime “had been paused to prevent recurrence of previous incidents triggered by this operation.” The safety pause outlived the certificate, a change-management irony any IT shop can recognize.

Then on Thursday, August 20, the status page opened another critical incident: Copilot Cloud Agent tasks delayed and their progress invisible for close to ten hours, root cause still pending. GitHub’s own availability reports counted eight incidents in July and six in June — this was not an isolated stumble.

The Denominator Doubled

Fedorov’s August 20 post is the more important document because it quantifies the demand side. Monthly commits grew from 1.4 billion in April to 2.9 billion in August. GitHub doesn’t say what’s driving that curve, but a doubling in four months lines up with coding agents going mainstream — every Copilot cloud task, background agent, and automated pull request lands on the same infrastructure human developers use.

The supply-side response is a sprint. GitHub has added more than 3 million CPU cores and 120 petabytes of high-speed storage, installing “as much hardware as available power allowed” in its existing data centers while accelerating the move to Azure. That migration — reported last year as a roughly 24-month effort prioritized over feature work after GitHub ran out of expansion room in Northern Virginia — now carries 58% of platform load and half of all Git operations, up from 12% in May. The immediate fixes are humbler: consistent retry limits and budgets across services, plus a review of low-priority CPU and memory alerts that let a saturating component go unnoticed.

What to Take From It

Three things travel beyond GitHub. First, identity plumbing can fail together with the platform it protects — on August 17, SAML, OIDC, and SCIM went down alongside everything else, turning a partial outage into a full stop for enterprises that gate repositories and CI/CD behind SSO. Second, audit retry behavior: a well-intentioned client loop turned one degraded endpoint into a 10x traffic amplifier, and yours can do the same to a supplier or your own middleware. Third, capacity planning at your vendors is now part of your availability posture. Something doubled GitHub’s core workload in four months; whatever your critical platforms are, someone is racing a curve like that on your behalf — and August 17 is what it looks like when the curve wins a round.

Fedorov closed his post with the right sentiment:

The candor is welcome. The commit graph suggests the race is far from over.

Sources

• GitHub Status - Incident with GitHub.com (August 17 root cause analysis)
• The GitHub Blog - The August 17 outage, and the work ahead
• GitHub Status - Incident with Actions (August 18)
• GitHub Status - Intermittent failures creating agent tasks (August 20)
• BleepingComputer - Microsoft confirms GitHub is down worldwide
• The New Stack - GitHub Will Prioritize Migrating to Azure Over Feature Development