Leadup

Describe the circumstances that led to this incident

• Reports of poor performance
• Local metrics reported an increase in CPU usage overtime
• Report from the client specifying issue specifically with bh006

Fault

Describe what failed to work as expected

The Pterodactyl/Wings@1.4 daemon was using up all the CPU resources of the system.

Restarting the service and reloading systemctl produced the same outcome.

Detection

Describe how the incident was detected

Reported by the client.

Root causes

Run a 5-whys analysis to understand the true causes of the incident

• An update was pushed out to Pterodactyl/Wings
• The update had an issue causing unordinary resource consumption
• The resource consumption issue caused other services to slow
• The Wings API failed to respond to requests as it could not keep up
• The service was flagged as degraded

‌

Mitigation and resolution

What steps did you take to resolve this incident?

• Stop wings
• Clear the container states cache
• Stop (or kill) all containers
• Delete all containers
• Remove wings from the system
• Fresh install of wings
• Enable wings on boot
• Enable docker on boot
• Restart the system
• Monitor wings until determined functioning as expected

Lessons learnt

What went well? What could have gone better? What else did you learn?

Ensure more in-depth monitoring, and that partial monitoring outages do not effect our response times.