Incident across several services
API Requests
Started:
Last official update:
Resolved:
Published incident duration: 1124 minutes.
Starting at 07:57 UTC on September 23, GitHub experienced elevated 500 and 404 responses across several application pages. This caused failures when installing GitHub Apps, creating organizations, and making some organization membership changes. Customers also experienced delayed label updates and stale search results in Projects. The infrastructure failure was isolated to our Azure Central US region. The API errors were mitigated by 10:58 UTC on September 23. Projects' processing continued to recover while an accumulated backlog was drained, and full service was restored at 04:55 UTC on September 24. The incident was caused by a failed planned maintenance operation o
Read the full official incident ↗
MiniCrafts groups the provider's published notice by affected product. Unconfirmed scope stays unconfirmed. Published incident duration is not a measurement of downtime for every user.
What to do
Commit locally and push later; Actions jobs usually re-queue after recovery.
Check current product status and troubleshoot · Usage limits and reset rules
Official update timeline
- · resolved
Starting at 07:57 UTC on September 23, GitHub experienced elevated 500 and 404 responses across several application pages. This caused failures when installing GitHub Apps, creating organizations, and making some organization membership changes. Customers also experienced delayed label updates and stale search results in Projects. The infrastructure failure was isolated to our Azure Central US region. The API errors were mitigated by 10:58 UTC on September 23. Projects' processing continued to recover while an accumulated backlog was drained, and full service was restored at 04:55 UTC on September 24. The incident was caused by a failed planned maintenance operation o
- · monitoring
The degradation has been mitigated. We are monitoring to ensure stability.
- · investigating
We are continuing to process the backlog of issue label updates for Projects. Label changes may still be delayed. All other services are operating normally.
- · investigating
We are continuing to process the backlog of issue label updates for Projects. Users may still see delays before label changes are reflected in Projects. All other services are operating normally.
- · investigating
We've deployed a change intended to accelerate processing of the backlog of issue label updates in Projects. A sizable backlog still remains and we continue working through it. All other services are operating normally. We will provide another update within the next hour.
- · investigating
Updates to issue labels may be delayed in being reflected in Projects. We are continuing to deploy a change that will accelerate processing of the backlog of label updates. All other services are available. We will provide another update within the next hour.
- · investigating
We are preparing to deploy a change that will mitigate the impact.
- · investigating
Continuing to investigate the lag that may be experienced in issue labels being accurately reflected in Projects. We are working on alternate solutions to process the backlog of label updates.