Incidents history

Mar 12, 2021

Migration of servers due to fire in data centers

Finished - Metrics infrastructure has been put back online.
Mar 12, 20:42 UTC

Mar 11, 2021

Migration of servers due to fire in data centers

In progress - FTP deployment has been fully enabled.
Mar 11, 18:48 UTC

Mar 10, 2021

Migration of servers due to fire in data centers

In progress - Deployment infrastructure is available. FTP deployment is temporarily disabled until the FTP content has been synced.
Mar 10, 9:21 UTC
In progress - All affected application servers are available again.
Mar 10, 8:37 UTC
Started - Due to a fire in data centers of our server provider we have started migration of all affected servers to a new data center.
Mar 10, 6:12 UTC

Sep 28, 2020

Scheduled maintenance of deployment infrastructure

Finished - The scheduled maintenance is finished.
Sep 28, 20:10 UTC
Started - We are going through a scheduled maintenance of our deployment infrastructure. Some deployments may be temporarily unavailable. We expect the downtime to be less than 5 minutes.
Sep 28, 18:24 UTC

Jul 23, 2020

North America apps partially unavailable

Resolved - We have fixed the issue. Apps in North America receiving traffic were affected. One of our proxy servers used for routing traffic has been misconfigured due to a bad commit from our side. We have resolved the issue and implemented checks in place to prevent such issues in the future. Roughly 30% of requests may have failed due to this issue.
Jul 23, 14:37 UTC
Started - We have identified a networking issue with one of our proxy servers. We're investigating the root cause.
Jul 23, 13:05 UTC

Mar 25, 2020

Scheduled maintenance of deployment infrastructure

Resolved - The scheduled maintenance is over.
Mar 25, 19:13 UTC
FTP deployment enabled - FTP deployment has been just enabled.
Mar 25, 17:48 UTC
Started - We have started a scheduled maintenance of our deployment infrastructure. During the maintenance git deployment will be briefly unavailable for about 5 minutes. FTP deployment is disabled until the maintenance is over. In case you need to deploy an urgent change we recommend using git deployment.
Mar 25, 16:05 UTC

Nov 9, 2019

MongoDB cluster unavailable

Resolved - Log collection has been resumed.
Nov 9, 11:42 UTC
Resolved - MongoDB cluster is in operational state.
Nov 9, 13:34 UTC
Started - We have disabled application logs to recover cluster faster.
Nov 9, 13:12 UTC
Started - We are experiencing issues with connectivity on two of our MongoDB cluster members. The cluster has been set to maintenance mode and is not available in order to preserve consistency of data.
Nov 9, 11:15 UTC

Oct 15, 2019

Log collection lags behind

Resolved - Log collection is back to normal. The issue has been resolved.
Oct 15, 17:56 UTC
Resolved - We have successfully restored operation of MongoDB cluster.
Oct 15, 13:06 UTC
Updated - We are still investigating log collection lags.
Oct 15, 13:06 UTC
Started - One of our MongoDB clusters is currently unavailable. We are investigating the issue and will keep you updated.
Oct 15, 12:27 UTC
Started - We are experiencing issues with log collection from some of our application servers. Logs may take a while until they arrive to our web dashboard.
Oct 15, 11:12 UTC

Apr 23, 2019

Load balancers dropping 10% of connections

Resolved - We have resolved issues with load balancers.
Apr 23, 18:04 UTC
Updated - We have identified faulty load balancers and are working on a replacement. Roughly 10% of connections were dropped without a reason.
Apr 23, 17:12 UTC
Started - We are experiencing issues with connectivity on our load balancers. Some connections are dropped. We're working on resolving the issue.
Apr 23, 16:51 UTC

Jan 27, 2019

Build servers experience intermittent connectivity issues

Resolved - We have resolved issues with the disk.
Jan 27, 15:34 UTC
Updated - We have identified faulty disks in our disk cluster. The disks are being replaced and we're waiting for the replication to finish.
Jan 27, 11:21 UTC
Started - We are experiencing connectivity issues with our build servers. We're working on resolving the issue.
Jan 27, 11:08 UTC

Aug 22, 2018

Networking connectivity outage

Resolved - The incident has been resolved. Connectivity between a couple of servers came back only after the servers were restarted. We're investigating the root cause with our hosting provider.
Aug 22, 16:40 UTC
Started - We are experiencing connectivity issues between our load balancers and application servers in US-3 region.
Aug 22, 16:02 UTC

Aug 6, 2018

Apps are not built properly on our deployment infrastructure

Resolved - Problem with deployments has been resolved
Aug 6, 15:48 UTC
Started - We have identified issues with app building on our deployment infrastructure.
Aug 6, 14:42 UTC

Jul 30, 2018

Unavailability of 30-1a MongoDB cluster

Resolved - The 30-1a cluster has been fully recovered. No data has been affected.
Jul 30, 18:24 UTC
Started - We have been notified of 30-1a MongoDB cluster unavailability. We're investigating the root cause.
Jul 30, 18:19 UTC

Jul 16, 2018

Scheduled upgrade of load balancers

Resolved - All load balancers were successfully upgraded
Jul 16, 14:36 UTC
Started - Scheduled upgrade has started. Availability of applications will not be affected.
Jul 16, 14:20 UTC