Webhooks reliability issues

Incident Report for yasoon

Resolved

Webhook handling was degraded by some workers with incorrect initialization.

Affected dates
2023/08/23 2pm-4pm UTC
2023/08/24 8am-11am UTC

Impact:
Roughly 20% of handled webhooks have been affected by the incident. That lead to missing Todos synced or missing channel notifications in Teams.

Fix:
Initialization script has been updated and made more robust. We are monitoring the situation

Posted Aug 24, 2023 - 10:00 CEST