Website and Control Panel Degraded Access and Performance
Incident Report for Neto by Maropost
Postmortem

Root Cause Analysis: The degraded performance and slowdown were attributed to unprecedented traffic volume and requests, particularly on two of our database clusters. This surge was primarily caused by a simultaneous increase in AI bot activity, leading to deadlocks within the databases, which took considerable time to resolve.

Actions Taken: Upon detection, our systems promptly alerted our teams, initiating an immediate investigation. Following a thorough post-mortem analysis, we identified the root causes and implemented preventative measures to forestall similar incidents. Substantial changes were made to our alerting mechanism, and adjustments to our rules were enacted to handle future occurrences more effectively.

Improvements Implemented:

  1. Enhanced Monitoring Mechanism: Our monitoring capabilities have been fortified to provide earlier alerts, allowing for swift intervention before any adverse impacts on our platform.
  2. Updated Rules: We have bolstered and refined our rules to better manage similar events should they arise in the future.

Resolution: The underlying issues contributing to the slowdown and downtime have been successfully rectified through database modifications. Since implementing these changes, we have diligently monitored the platform's performance, observing it to be operating within expected parameters.

Continued Commitment: We remain steadfast in our commitment to delivering a seamless and dependable experience for all our customers. Your trust and satisfaction are paramount, and we pledge to prioritize the stability and performance of our platform continuously.

Feedback and Support: Should you have any inquiries, concerns, or feedback regarding the recent disruption or our ongoing efforts to enhance our services, please don't hesitate to reach out to our dedicated support team. We are here to assist you and ensure your experience with our platform consistently exceeds your expectations.

Posted Apr 09, 2024 - 16:02 AEST

Resolved
This incident has been resolved.
Posted Mar 30, 2024 - 22:53 AEST
Monitoring
A fix has been implemented and we are monitoring the results
Posted Mar 30, 2024 - 19:50 AEST
Investigating
We are aware of degraded performance and access limitations affecting some Neto Webstore, POS and Control Panel accounts. Our development teams are actively investigating and we will provide further updates as soon as possible.
Posted Mar 30, 2024 - 11:15 AEST
This incident affected: Merchant Store Fronts, Control Panel, and Point Of Sale Registers.