Incidents history

Apr 23, 2019

Load balancers dropping 10% of connections

Resolved - We have resolved issues with load balancers.
Apr 23, 18:04 UTC
Updated - We have identified faulty load balancers and are working on a replacement. Roughly 10% of connections were dropped without a reason.
Apr 23, 17:12 UTC
Started - We are experiencing issues with connectivity on our load balancers. Some connections are dropped. We're working on resolving the issue.
Apr 23, 16:51 UTC

Jan 27, 2019

Build servers experience intermittent connectivity issues

Resolved - We have resolved issues with the disk.
Jan 27, 15:34 UTC
Updated - We have identified faulty disks in our disk cluster. The disks are being replaced and we're waiting for the replication to finish.
Jan 27, 11:21 UTC
Started - We are experiencing connectivity issues with our build servers. We're working on resolving the issue.
Jan 27, 11:08 UTC

Aug 22, 2018

Networking connectivity outage

Resolved - The incident has been resolved. Connectivity between a couple of servers came back only after the servers were restarted. We're investigating the root cause with our hosting provider.
Aug 22, 16:40 UTC
Started - We are experiencing connectivity issues between our load balancers and application servers in US-3 region.
Aug 22, 16:02 UTC

Aug 6, 2018

Apps are not built properly on our deployment infrastructure

Resolved - Problem with deployments has been resolved
Aug 6, 15:48 UTC
Started - We have identified issues with app building on our deployment infrastructure.
Aug 6, 14:42 UTC

Jul 30, 2018

Unavailability of 30-1a MongoDB cluster

Resolved - The 30-1a cluster has been fully recovered. No data has been affected.
Jul 30, 18:24 UTC
Started - We have been notified of 30-1a MongoDB cluster unavailability. We're investigating the root cause.
Jul 30, 18:19 UTC

Jul 16, 2018

Scheduled upgrade of load balancers

Resolved - All load balancers were successfully upgraded
Jul 16, 14:36 UTC
Started - Scheduled upgrade has started. Availability of applications will not be affected.
Jul 16, 14:20 UTC