Summary
Issue in one of Future Orderings payment system resulted in chainevents causing guests inability to complete payments or experience a slow frontend navigation.
Problem determined starting creating significant affect on sales at 11:45 and resolved at 13:05.
Root cause was related to one of the running instances of Future Ordering payment services failed to resolve requests, when the instance was healed a follow-up affect of suddenly creating double amount of throughput of requests created slow responses of multiple systems until they had scaled appropriately.
This also identified additional limitations on other related systems that had none functioning or none scaling functionality enabled.
Scope
All sales channels affected
Severity
Majority of guests unable to complete payment
Timeline
All timestamps and dates are CET.
- 11:30 Frontend experienced increased response time
- 11:45 Issue determined to be critical as Majority of payments where unable to be completed.
- 12:40 Issue identified from Frontend system, workaround actions started to resolve scaling issue.
- 12:45 Issue identified in Payment system, workaround implemented.
- 13:05 Frontend and Payment system determined to be fully recovered and no issues no longer present
Cause
- Multiple services failure to scale according to load or not configured to be able to scale at all.
- Payment service failing to handle requests.
Actions for resolution
- Manually scale out affected services with failing services.
- Restarting affected Payment service experiencing inability to resolve requests.
Planned improvements
- Add automatic scalability on all services and resources.
- Utilize Microsoft Rule based scalability functionality as it's determined more stable and predictable.
- Resolve cause of service inability to handle requests.
- Resolve occurrence of false positive alerts for failing payments to improve response.