Update - The situation appears to be resolved. We're monitoring the situation and vetting the fixes.
Feb 23, 19:23 UTC
Update - The Compose UI is currently serving up 404s. This does not impact database deployments. All customer deployments are completely segregated from the UI and will not impact performance.
Feb 23, 18:40 UTC
Update - Functionality is restored to the UI and API. Compose is currently vetting the fix.
Feb 23, 18:16 UTC
Monitoring - Functionality appears to be restored. We're confirming status.
Feb 23, 18:03 UTC
Investigating - There is a known issue with logging in to the Compose UI at app.compose.io. This does not in any way interfere with the operation of database deployments at Compose. We're investigating the issue now.
Feb 23, 16:35 UTC
Web Application ? Under Maintenance
REST API ? Operational
Degraded Performance
Partial Outage
Major Outage
Past Incidents
Feb 22, 2017

No incidents reported.

Feb 21, 2017
Resolved - All operations have been investigated and operations are completing normally.
Feb 21, 15:40 UTC
Monitoring - The root cause has been fixed and new operations are all proceeding as normal. We are investigating a handful of jobs that got stuck in the queue and need some intervention to complete successfully. We don't anticipate further issues caused by this.
Feb 21, 14:14 UTC
Investigating - Our monitoring has detected that there are queued operations like new provisions and configuration changes to deployments that have been made over the last 6 hours that are not completing. We have diagnosed the issue and will endeavour to clear this as soon as possible.

Existing database availability should not be impacted although configuration changes to existing databases might be queued.
Feb 21, 11:25 UTC
Resolved - All capacity issues are now resolved.
Feb 21, 03:26 UTC
Identified - We are currently experience capacity issues in AWS US East 1 (Northern Virginia) region. New deployments there my fail until we can deploy more capacity. We apologise for any inconvenience this may cause. Other regions and providers should not be affected.
Feb 19, 12:56 UTC
Feb 20, 2017

No incidents reported.

Feb 18, 2017

No incidents reported.

Feb 17, 2017
Resolved - Deployments impacted by this issue have all been restored to service. If you are experiencing any abnormal behavior, please open a support ticket so that we can resolve your problem immediately.

A complete post-mortem will be sent to all impacted account owners.
Feb 17, 18:44 UTC
Update - We have resolved all alerts produced from our monitoring system so to our knowledge all affected deployments are now healthy. If you are still seeing issues please contact support@compose.io.
Feb 17, 13:15 UTC
Monitoring - At this time all impacted databases are online.

We are closely monitoring all database deployments and their underlying hosts for any signs of regression.

All accounts impacted by today's event will receive post-mortem details within the next 24-48 hours.
Feb 17, 06:07 UTC
Update - All hosts are online at this point, however some deployments are still experiencing related issues. We continue to work through individual deployment alarms.
Feb 17, 04:14 UTC
Update - At this point in time most of the affected hosts have recovered, although three remain in a degraded state:


The Compose team is inspecting individual deployments and triaging any issues discovered on the recovered hosts.
Feb 17, 01:05 UTC
Update - We are continuing to see progress, more hosts are coming online as the recovery operations complete. There still isn't an ETA, the recovery process time varies from host to host. Over 60 % of the unresponsive hosts are back up at this point in time.

Data storing members have not been affected in most cases, the issue is limited to hosts containing utility capsules: MongoDB configservers, Redis sentinels and etcd for PostgreSQL and standalone. We have seen this cause availability issues in some cases.

New deployments on AWS East are not affected.

Apologies to those impacted, we are working hard to resolve this as soon as possible.
Feb 16, 22:38 UTC
Update - Hosts are still recovering at this time. We do not have a timeline on when all hosts will be recovered, but deployments will continue to come back online as our team works diligently to restore normal operations. The hosts that are online are still under some load so while the deployment may be online, you may notice performance issues.
Feb 16, 21:46 UTC
Update - Correction: Both MongoDB and PostgreSQL deployments are affected by this issue. This is contrary to our previous update that only MongoDB was affected.
Feb 16, 20:19 UTC
Update - Hosts are still recovering. We do not have a timeline on when all hosts be recovered, but deployments should be coming online as the hosts recover.
Feb 16, 19:32 UTC
Update - Hosts that were affected by this outage have begun recovering. Deployments should begin resuming normal operations as these hosts recover.
Feb 16, 19:08 UTC
Update - The scope of the deployments affected by this issue should be limited to MongoDB deployments only.
Feb 16, 19:02 UTC
Update - Compose engineers are still in communication with AWS engineers, and working to resolve the issue.
Feb 16, 18:26 UTC
Update - Compose is working with AWS support and our full engineering team to address an issue that's causing several hosts in our fleet of AWS US East hosts to be unresponsive or with degraded performance.
Feb 16, 17:29 UTC
Update - Compose is currently experiencing an infrastructure outage within the AWS US-East-1 region. This is not related to Amazon themselves, but rather an issue within Compose's platform. This is currently priority 1 at Compose, and all hands are on deck to resolve this.

We have identified the root cause and are working with AWS to fix as soon as possible.
Feb 16, 16:34 UTC
Identified - We have identified the cause of this issue as widespread degradation of disk I/O across numerous hosts. With AWS assistance, we have been able to resolve the issue and continue to restore service to databases as quickly as possible.
Feb 16, 16:33 UTC
Investigating - We are investigating reports of database access trouble primarily in the AWS US-East region. We are working as quickly as possible to identify and resolve the problem. Further updates will be posted here as they come available.
Feb 16, 15:57 UTC
Feb 16, 2017
Resolved - This incident has been resolved.
Feb 16, 19:09 UTC
Monitoring - The majority of deployments are now restored to full health, there are a few remaining ones we are continuing to work on.
Feb 16, 15:20 UTC
Update - All capsules have been re-created on the host and are running. We are working through alerts to ensure deployments are returned to full health.
Feb 16, 14:32 UTC
Update - We identified that the instance was terminated by GCP. On reboot it came back with corrupted data drives which are taking some time to rebuild. Customer databases should still be available due to redundancy on other zones.
Feb 16, 13:22 UTC
Identified - gcp-europe-west1-data.1 rebooted outside of our control. We are working to restore this host.
Feb 16, 11:21 UTC
Resolved - Capacity constraints are now resolved.
Feb 16, 07:21 UTC
Identified - A capacity constraint has been identified for AWS US East 1 (Northern Virginia). It may not be possible to create new deployments in this location until additional capacity has been added.

Deployments may either fail to initiate provisioning and produce an error message, or if they do initiate they may fail to complete provisioning.
Feb 15, 08:03 UTC
Resolved - All remaining deployments are now healthy.
Feb 16, 06:58 UTC
Update - Most deployments are now running healthily, but there are a few remaining Postgresqls and MySQLs we are continuing to work on.
Feb 15, 23:22 UTC
Monitoring - Network stability should now be restored. We are working to resolve any affected deployments.
Feb 15, 14:57 UTC
Identified - We are aware of intermittent network connectivity issues affecting a number of sl-eu-lon-2 hosts. This is the result of Softlayer backend network connectivity degradation. Additional updates will occur as they become known.
Feb 15, 14:27 UTC
Feb 15, 2017
Resolved - All deployments should now be restored to full health. If you are still experiencing issues related to this host please contact support@compose.io.
Feb 15, 16:41 UTC
Monitoring - Host is back up and all capsules are running. We are working to resolve any remaining issues affecting deployments on that host.
Feb 15, 15:33 UTC
Identified - All MongoDB deployments that have candidate.49 as a member should not experience downtime if the application is properly using the replica set. We're working to restore the host.
Feb 15, 14:23 UTC
Feb 14, 2017
Resolved - This incident has been resolved.
Feb 14, 20:55 UTC
Monitoring - All capsules have been started. If you have issues, please contact us at support@compose.io.
Feb 14, 18:12 UTC
Identified - This host requires immediate emergency maintenance. We will be restarting this instance ASAP to prevent extended downtime. If you have questions or issues please contact support@compose.io.
Feb 14, 17:45 UTC
Feb 13, 2017

No incidents reported.

Feb 12, 2017

No incidents reported.

Feb 11, 2017

No incidents reported.

Feb 10, 2017

No incidents reported.

Feb 9, 2017
Resolved - All capsules have been started. If you have issues, please contact us at support@compose.io.
Feb 9, 21:53 UTC
Identified - Compose engineers are working to resolve a issue with this host. If you are experiencing trouble as a result of this, please open a support ticket.
Feb 9, 21:36 UTC
Resolved - This incident has been resolved.
Feb 9, 15:34 UTC
Monitoring - Softlayer has stabilized the backend network. Deployments in Softlayer London-02 should now be operating normally. Please contact support@compose.io with any questions.
Feb 9, 11:32 UTC
Update - Softlayer is continuing to experience backend network degradation which is affecting the operation of deployment with london-02. We are working to restore normal operations as soon as possible.
Feb 9, 01:09 UTC
Identified - This outage is the result of Softlayer backend network connectivity degradation. Additional updates will occur as they become known.
Feb 8, 15:51 UTC
Investigating - We are still investigating this issue. All provisioning to this region has been temporarily disabled. If you have questions, please contact us at support@compose.io.
Feb 8, 15:29 UTC