Summary
Issue during a STSGen2 release resulted in guests not being able to complete orders online & in the APP.
The issue started at 08:05 and was resolved at 08:28 AM.
Root cause was that the system responsible for managing resource deployment encountered an issue where it unintentionally removed certain resources and was unable to automatically restore them as expected. This was detected during release monitoring and a rollback was performed which resolved the issue.
Scope
Kiosk, app & online.
Severity
Guest's were unable to complete orders via kiosk, app & online.
Timeline
All timestamps and dates are CET.
- 07:45 - STSGen2 release started.
- 08:05 - Issue started. Developers are notified and prepares rollback.
- 08:20 - Information about ongoing outage via Statuspage.
- 08:22 - Rollback started.
- 08:24 - Services fully restored.
- 08:30 - Information about closed incident via Statuspage.
Cause
- Our system responsible for managing resource deployment encountered an issue where it unintentionally removed certain resources and was unable to automatically restore them as expected.
Planned improvements
- Improving deployment procedure, adding additional automatic review steps during deployments, preventing similar incidents.