Table Alter Issue
Incident Report for ReCharge
Current issue is resolved. A routine online schema change table alter caused database node to stop functioning. To resolve we automatically deployed a DNS switch to healthy node and decommissioned the non functioning database node. We will immediately start working on a means to make recovery from such an error take far less time until services are fully restored. We are also finding the root cause of why the table alter failed to begin with, and will make corrections to avoid this type of table alter issue in the future.
Posted Dec 03, 2018 - 02:28 PST
Issues resolved.
Posted Dec 03, 2018 - 02:15 PST
App performance slow as servers are recovering from backlogged requests.
Posted Dec 03, 2018 - 02:07 PST
We're deploying the fix now. A table alter caused db issues. Going to be about 10 more mins to full resolution.
Posted Dec 03, 2018 - 01:58 PST
A routine table alter has impacted the app. We are aware of the issue and working on restoring full service.
Posted Dec 03, 2018 - 01:50 PST
This incident affected: ReCharge API, ReCharge Admin panel, and ReCharge Checkout.