Summary
Issue with one of Future Orderings STSgen2 ordering integration, causing orders via kiosk, app & online to time out and unable to complete.
The issue started at 01:00 AM and resolved at 08:06 AM.
Root cause was that the STSgen2 ordering instance was unable to start because of expired internal credentials. Issue was instigated by automatic restart functionality that allows for stability.
Scope
Kiosk, app & online.
Severity
Guest's were unable to complete orders via kiosk, app & online.
Timeline
All timestamps and dates are CET.
- 01:00 - Automatic restart was initiated, causing the issue.
- 03:10 - FO Support notified of ongoing incident.
- 03:20 - FO Support performing incident troubleshooting, trying to fix the issue by restarting instance, increasing instances & changing instances.
- 03:40 - Issue is escalated internally to developers.
- 07:20 - Developers identifies issue with STSGen2 ordering instance internal credentials.
- 08:06 - STSGen2 Ordering Instance internal credentials successfully updated.
Cause
- STSGen2 Ordering instance internal credentials had expired.
Planned improvements
- Implementing new escalation point for system that fail to start.
- Improvement of automated processes for updating internal system credentials.
- Improvement of customer & internal communication during critical incidents for out of office hours.