Incidents history

Finished - DNS is fully operational.

Started - Our DNS provider is experiencing full outage of services. We're working on migrating DNS hosting to a new provider.

Finished - Metrics infrastructure has been put back online.

In progress - FTP deployment has been fully enabled.

In progress - Deployment infrastructure is available. FTP deployment is temporarily disabled until the FTP content has been synced.

In progress - All affected application servers are available again.

Started - Due to a fire in data centers of our server provider we have started migration of all affected servers to a new data center.

Finished - The scheduled maintenance is finished.

Started - We are going through a scheduled maintenance of our deployment infrastructure. Some deployments may be temporarily unavailable. We expect the downtime to be less than 5 minutes.

Resolved - We have fixed the issue. Apps in North America receiving traffic were affected. One of our proxy servers used for routing traffic has been misconfigured due to a bad commit from our side. We have resolved the issue and implemented checks in place to prevent such issues in the future. Roughly 30% of requests may have failed due to this issue.

Started - We have identified a networking issue with one of our proxy servers. We're investigating the root cause.

Resolved - The scheduled maintenance is over.

FTP deployment enabled - FTP deployment has been just enabled.

Started - We have started a scheduled maintenance of our deployment infrastructure. During the maintenance git deployment will be briefly unavailable for about 5 minutes. FTP deployment is disabled until the maintenance is over. In case you need to deploy an urgent change we recommend using git deployment.

Resolved - Log collection has been resumed.

Resolved - MongoDB cluster is in operational state.

Started - We have disabled application logs to recover cluster faster.

Started - We are experiencing issues with connectivity on two of our MongoDB cluster members. The cluster has been set to maintenance mode and is not available in order to preserve consistency of data.

Resolved - Log collection is back to normal. The issue has been resolved.

Resolved - We have successfully restored operation of MongoDB cluster.

Updated - We are still investigating log collection lags.

Started - One of our MongoDB clusters is currently unavailable. We are investigating the issue and will keep you updated.

Started - We are experiencing issues with log collection from some of our application servers. Logs may take a while until they arrive to our web dashboard.

Resolved - We have resolved issues with load balancers.

Updated - We have identified faulty load balancers and are working on a replacement. Roughly 10% of connections were dropped without a reason.

Started - We are experiencing issues with connectivity on our load balancers. Some connections are dropped. We're working on resolving the issue.

Resolved - We have resolved issues with the disk.

Updated - We have identified faulty disks in our disk cluster. The disks are being replaced and we're waiting for the replication to finish.

Started - We are experiencing connectivity issues with our build servers. We're working on resolving the issue.

Resolved - The incident has been resolved. Connectivity between a couple of servers came back only after the servers were restarted. We're investigating the root cause with our hosting provider.

Started - We are experiencing connectivity issues between our load balancers and application servers in US-3 region.

Resolved - Problem with deployments has been resolved

Started - We have identified issues with app building on our deployment infrastructure.

Resolved - The 30-1a cluster has been fully recovered. No data has been affected.

Started - We have been notified of 30-1a MongoDB cluster unavailability. We're investigating the root cause.

Resolved - All load balancers were successfully upgraded

Started - Scheduled upgrade has started. Availability of applications will not be affected.