Member-only story

Postmortem: Shell-Thon Application Outage

Michael
2 min readJun 10, 2023

Read Full Article Here for Non-Medium members

Issue Summary Duration: May 15, 2023, 8:00 AM - May 17, 2023, 12:00 PM (UTC)

Timeline
May 15, 2023, 8:00 AM: The issue was detected when multiple users reported encountering errors and the monitoring system detected a sharp increase in error rates.
Actions taken: The development team investigated the backend codebase and server logs to identify the root cause. Initial assumption: Bug in the PHP code causing a critical system failure.
Misleading investigation/debugging paths: Extensive code reviews were performed, but no specific bug was found that could explain the complete system failure.
The incident was escalated to the senior development team and the security team.
Resolution
Root cause: The investigation revealed that the server hosting the Shell-Thon Application had multiple PHP vulnerabilities that were exploited, leading to a complete system crash.
Resolution: The server’s PHP version was updated to the latest secure release, and all known vulnerabilities were patched. Additionally, a security audit was conducted to identify and fix any other potential vulnerabilities.
Corrective and Preventative Measures
Code review and testing: Implement a comprehensive code review process for the Shell-Thon Application to catch potential bugs and vulnerabilities before deployment. Enhance testing practices, including security testing, to ensure robustness.
Regular vulnerability assessments: Conduct periodic vulnerability…

--

--

Michael
Michael

Written by Michael

Software Engineer | Cyber Security Specialist

Responses (3)