Users reported workflows were not running, we investigated and could not find the root cause but did solve the issue.
|12:00||Workflows Invocations stop moving from
|15:43||We receive simultaneous reports from multiple users|
|17:28||We restart handlers and the problem goes away.|
We have added monitoring for this to identify
We have begun monitoring the states of workflows and will receive an alert whenever there are more than 10 unscheduled workflows, staying unscheduled for at least 30 minutes.
Hopefully this will let us respond significantly more quickly to similar issues in the future