Users reported workflows were not running, we investigated and could not find the root cause but did solve the issue.
Incident Timeline
| Time | State |
|---|---|
| 12:00 | Workflows Invocations stop moving from new state to scheduled for unknown reasons |
| 15:43 | We receive simultaneous reports from multiple users |
| 17:28 | We restart handlers and the problem goes away. |
Resolution
We have added monitoring for this to identify
We have begun monitoring the states of workflows and will receive an alert whenever there are more than 10 unscheduled workflows, staying unscheduled for at least 30 minutes.
Hopefully this will let us respond significantly more quickly to similar issues in the future