Current issue is resolved. A routine online schema change table alter caused database node to stop functioning. To resolve we automatically deployed a DNS switch to healthy node and decommissioned the non functioning database node. We will immediately start working on a means to make recovery from such an error take far less time until services are fully restored. We are also finding the root cause of why the table alter failed to begin with, and will make corrections to avoid this type of table alter issue in the future.
Posted Dec 03, 2018 - 02:28 PST
Update
Issues resolved.
Posted Dec 03, 2018 - 02:15 PST
Update
App performance slow as servers are recovering from backlogged requests.
Posted Dec 03, 2018 - 02:07 PST
Update
We're deploying the fix now. A table alter caused db issues. Going to be about 10 more mins to full resolution.
Posted Dec 03, 2018 - 01:58 PST
Investigating
A routine table alter has impacted the app. We are aware of the issue and working on restoring full service.
Posted Dec 03, 2018 - 01:50 PST
This incident affected: Recharge API, Recharge Customer Portal, and Recharge Checkout.