From code inspection, it looks like the changes in dffcb944 have made the AlertingAction class effectively useless.
The class has a thread local that is set when we start sending an alert. But dffcb944 changed the alert process such that shortly after the thread local is set, the bulk of the alerting logic happens in a separate thread, without the thread local, in ClusterAlertMessaging.sendAlert.
Therefore, this thread local has no effect. If this behavior was previously useful, we need to set the thread local inside this new executor thread, perhaps? Or find some other way to prevent the executor from triggering new alerts?