On August 19th - Sept 1st, 2022, the Zingle platform experienced degraded service where customers were unable to send/receive messages. Zingle engineers were immediately alerted to the outage and began working to resolve the issues.
On August 19th, 2022, a database (DB) swap & Redis server split took place during a scheduled downtime maintenance window. After the facilitation of a DB swap and Redis split, two issues became apparent.
The two issues above caused a cascading failure where the web servers aggressively tried to grab more connections of the DB than the DB was configured for, causing sporadic but consistent data connection issues across the entire platform. As a result, message receiving and message sending were impacted as well as data importing and Inbox stability. Zings were also impacted as well from executing correctly.
After much monitoring and observations to understand where the issues were, work began to upgrade all hardware that communicates with the DB. Work also began to upgrade the DB hardware. This required an emergency maintenance window of approx. 30 minutes on Sept 1st. We also increased the RAM used per web server thread to take advantage of the available memory.
The Zingle engineering team has taken the following corrective actions to prevent this scenario from occurring again in the future:
Any change to our DB hardware that requires a physical move of a DB will now go through a more stringent process to understand the impacts that will take place. We will also write out a plan of action on how to ensure all hardware is updated correctly when we facilitate this move. Refactoring of certain code that was DB inefficient has taken place. Also, a plan to improve the performance of our DB schema has also taken place with an investigation on where we can improve our DB performance.
We sincerely apologize for this inconvenience caused during that time, as we know Zingle is mission-critical to many teams and businesses. If you have any questions or concerns, please do not hesitate to reach out to your account manager or our Support team.