On Wednesday 2024-10-09, starting from 23:17 EEST (GMT+3) until Thursday 2024-10-10 9:56, a major part of the purchase authorizations for the merchants in Finland using Nets acquiring were failing. We are deeply sorry and apologize for the disturbance.
We have now analyzed the incident, identified the root cause, implemented complete mitigation and started working on improving our systems to avoid similar problems in the future, and especially to notice them faster.
The first hint of problems rose to our attention when an automatic alarm triggered in the morning at 8:15. The alarm did not trigger earlier as during the night and early morning there weren’t enough transactions.
At 8:22 we had an internal response team discussing the incident, the actual error was discovered at 8:32 and by then, it was obvious that it was a real issue. Statuspage was updated shortly after to communicate publicly.
Due to the error being outside of our control, we created an incident with the relevant party at 8:50. At 9:02, we had an emergency meeting with all the relevant parties present. The error was quickly explained and it was discovered that a change was deployed at the same time as the errors began. A rollback procedure for the change was triggered and it finished at 9:56. At the same time, we saw that no further errors were occurring.
At 9:58, we publicly communicated via Statuspage that the incident was over and we started monitoring and verifying that no further problems appeared.
By 10:19, the incident response team had concluded their work and we resumed normal duties.
There was an error at the acquirer side with the deployment of a new version, and the error was unfortunately not caught during testing. Additionally it only affected some payment service providers and most of the integrations with the acquirer were working correctly after the deployment. A return value of a field was out of spec and our system did not tolerate the wrong value as we verify that the messaging follows the specification. Hence affected transactions were reversed automatically and purchases on the terminal side failed. Reversal was working correctly so cardholders did not experience other issues such as money being reserved from their accounts.
While during quiet hours transaction volumes are much lower, we acknowledge that it is not a sufficient reason to not discover a problem like this. We try to avoid false positives but investing effort into better alarms is possible, and we have already implemented some enhancements to the alarms in question. Specifically we monitor per acquirer traffic in more detail now and have tuned the thresholds for the lower traffic volume during night time and early morning.
We have identified further follow-up actions to make our systems even more robust in the future.
Finally, we hope you have a thriving business and a pleasant upcoming Halloween season.