Root Cause Analysis: 

The delay experienced was attributed to a significant influx of requests directed to the /ajax/ path originating from a small number of sites. Investigation revealed that this surge in traffic was due to a Google bot accessing this path, resulting in unforeseen congestion within the databases and a subsequent decrease in performance across two key databases. 

Upon further examination, it was observed that affected sites lacked specific rules within their robots.txt files, which are intended to restrict bot access to content within the /ajax/ path. 

Actions Taken: 

In response to the performance degradation, our team immediately conducted a comprehensive post-incident analysis to identify the fundamental causes and implement preventive measures to mitigate the risk of similar occurrences in the future. Subsequently, we deployed optimised firewall rules to counteract such bot-driven activity. Additionally, we ensured the necessary rules were added to the robots.txt files of affected sites, with plans to extend this measure to other relevant sites. 

Improvements Implemented: 

Updated Cloudflare Rules: Introduced a new managed captcha rule tailored to address bot activity targeting the /ajax/ path. 

Updated robots.txt files: Implemented the requisite rules within robots.txt files to reinforce access restrictions. 

Resolution: 

The underlying issue contributing to the performance degradation has been effectively remedied. Since the implementation of these measures, we have diligently monitored the platform's performance and are confident that operations are now proceeding optimally.   

Continued Commitment:  

We want to reassure you that we remain committed to delivering a seamless and reliable experience for our customers. Your trust and satisfaction are of utmost importance to us, and we will continue to prioritise the stability and performance of our platform.  

Feedback and Support:  

If you have any questions, concerns, or feedback regarding the recent downtime or our ongoing efforts to improve our services, please do not hesitate to reach out to our support team. We are here to assist you and ensure your experience with our platform exceeds your expectations.