Current issue is resolved. A routine online schema change table alter caused database node to stop functioning. To resolve we automatically deployed a DNS switch to healthy node and decommissioned the non functioning database node. We will immediately start working on a means to make recovery from such an error take far less time until services are fully restored. We are also finding the root cause of why the table alter failed to begin with, and will make corrections to avoid this type of table alter issue in the future.
Dec 3, 02:28 PST
Dec 3, 02:15 PST
App performance slow as servers are recovering from backlogged requests.
Dec 3, 02:07 PST
We're deploying the fix now. A table alter caused db issues. Going to be about 10 more mins to full resolution.
Dec 3, 01:58 PST
A routine table alter has impacted the app. We are aware of the issue and working on restoring full service.
Dec 3, 01:50 PST