Details
-
Sub-task
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
Description
While debugging, I've noticed this log.
INFO 07-07 16:41:48,631 BlockManagerMaster:172 [NettyMessagingTransport:ServerWorker:pool-11:thread-005] - Block Block-edge-d47_0 location is in state AVAILABLE
This means that BlockManagerMaster is being operated by a networking thread, which I don't think is a good idea.
My suggestion is that we let the RuntimeMaster thread also operate BlockManagerMaster, such that task state transitions and block state transitions are serialized and easy to understand/debug.
If a single RuntimeMaster thread does turn out to be a bottleneck in large-scale experiments (although it's only doing cheap in-memory metadata updating operations) then we may start thinking about introducing a dedicated thread(or a thread pool) for the BlockManagerMaster.