- bh007 -is operating as expected - bh009 - is operating as expected - bh004 -is operating as expected
Posted Jun 10, 2021 - 20:58 UTC
Update
- bh007 - bh007 is operating as expected
- bh009 - bh009 is operating as expected - bh004 (which is running the reverse proxy) is operating as expected - The server is currently rate limited by ZeroSSL and LetsEncrypt - This is causing SSL errors/issues to occur - The process is still on-going - About 1/4 of sites have been restored - The SSL issues should be resolved within 48 hours - No manual intervention is possible due to the scale
Posted Jun 10, 2021 - 20:56 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 10, 2021 - 20:22 UTC
Update
We are doing monitoring the situation with the reverse proxy on bh004 to ensure it's operating as expected.
Posted Jun 10, 2021 - 16:16 UTC
Update
This issue is partially resolved, we're awaiting manual intervention from the datacenter.
- bh007 - bh007 is unavailable - A hard reset will resolve the issue - There is no way for the control plane to communicate with this server, nor remote management software - bh009 - bh009 is having some monitoring issues but is operating as expected - bh004 (which is running the reverse proxy) is operating as expected - The server is currently rate limited by ZeroSSL and LetsEncrypt - This is causing SSL errors/issues to occur - The process is still on-going - About 1/4 of sites have been restored - The SSL issues should be resolved within 48 hours - No manual intervention is possible due to the scale
We will inform you on any further changes to the incident
Posted Jun 09, 2021 - 22:48 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jun 09, 2021 - 22:44 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jun 09, 2021 - 22:15 UTC
Identified
A response team has been assigned to the incident and are working on restoring full access to the services.
Posted Jun 09, 2021 - 20:54 UTC
This incident affected: Datacenters (Dallas, US/1 (BHDA1)).