In IBM Big SQL, we have included several alerts. One is an aggregate alert which reports how many of our "slave" components are running.
In the cluster (originally installed with one master and three slave components), the slave component was removed from one host, and the individual alert on that host was removed, but the aggregate alert still shows three slave components in total, and fires the alert because it can only find two of them as "OK".
Then, when we trigger an alert event (by stopping one of the remaining slave components) the alert is updated to show the new, correct total count.
Judging by all of this, it sounds like we just need to make deleting a component an "alert event" for any alerts that were associated with that component. The event will trigger an update of the aggregate alert.