There is a race condition when killing an executor and reregistering an executor happen at the same time. Here is the execution steps to reproduce it.
1. master find a worker is dead
2. master tells driver to remove executor
3. driver remove executor
4. BlockManagerMasterEndpoint remove the block manager
5. executor finds it's not reigstered via heartbeat
6. executor send reregister block manager
7. register block manager
8. executor is killed by worker
9. CoarseGrainedSchedulerBackend ignores onDisconnected as this address is not in the executor list
10. BlockManagerMasterEndpoint.blockManagerInfo contains dead block managers
As BlockManagerMasterEndpoint.blockManagerInfo contains some dead block managers, when we unpersist a RDD, remove a broadcast, or clean a shuffle block via a RPC endpoint of a dead block manager, we will get ClosedChannelException.