Table Alter Issue
Incident Report for ReCharge
Resolved
Current issue is resolved. A routine online schema change table alter caused database node to stop functioning. To resolve we automatically deployed a DNS switch to healthy node and decommissioned the non functioning database node. We will immediately start working on a means to make recovery from such an error take far less time until services are fully restored. We are also finding the root cause of why the table alter failed to begin with, and will make corrections to avoid this type of table alter issue in the future.
Posted 6 days ago. Dec 03, 2018 - 02:28 PST
Update
Issues resolved.
Posted 6 days ago. Dec 03, 2018 - 02:15 PST
Update
App performance slow as servers are recovering from backlogged requests.
Posted 6 days ago. Dec 03, 2018 - 02:07 PST
Update
We're deploying the fix now. A table alter caused db issues. Going to be about 10 more mins to full resolution.
Posted 6 days ago. Dec 03, 2018 - 01:58 PST
Investigating
A routine table alter has impacted the app. We are aware of the issue and working on restoring full service.
Posted 6 days ago. Dec 03, 2018 - 01:50 PST
This incident affected: ReCharge API, ReCharge Admin panel, and ReCharge Checkout.