Issues with inbound Conversations for some Services

Incident Report for Medallia Concierge

Postmortem

Summary

From approximately 11:30 am until around 1:00 pm EST on January 26, 2023, the Concierge App in the Main Environment (app.zingle.me) experienced issues receiving and sending messages. During this time, users were unable to access conversations or reply to inbound messages.

‌

Root Cause

A hardware table unexpectedly reached capacity, which caused connectivity issues for some services within the Seattle data center, including the Concierge App.

‌

Corrective Actions

To mitigate the issue, Medallia Engineering rebooted some devices and changed network configurations to reduce entries back to normal usage levels for the affected hardware table. To prevent this issue from occurring in the future, Medallia Engineering is implementing more robust monitoring and alerting for hardware tables. Additional network configuration changes will be applied to safeguard against similar capacity issues.

Posted May 16, 2023 - 13:25 EDT

Resolved

This incident has been resolved.

Posted Jan 27, 2023 - 10:48 EST

Update

We've identified the source and nature of the issues and have implemented the necessary measures to mitigate. We are continuing to monitor services to ensure full recovery, and will continue to provide updates. Any services still affected should reach out to Support for assistance and continued troubleshooting.

Posted Jan 26, 2023 - 13:10 EST

Monitoring

A fix has been implemented to help address the issue with missing msgs. Users should begin observing recovery within the hour.

We will continue to monitor this issue and share updates here.

Thanks for your patience.

Posted Jan 26, 2023 - 13:07 EST

Investigating

We have received reports of inbound messages and conversations not appearing in the Inbox for a small subset of Zingle services. Our team is actively investigating these reports and working internally to address any issues. We'll provide updates as they are available

Posted Jan 26, 2023 - 11:46 EST

This incident affected: Web App - [app.zingle.me] (Web Inbox, SMS Messaging).