[NEMO-145] Network threads should not touch BlockManagerMaster

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Labels: runtime
• Assignee: Unassigned
• Reporter: John Yang
• Created: 08/Jul/18 06:25
• Updated: 09/Jul/18 05:53

Description

While debugging, I've noticed this log.

INFO 07-07 16:41:48,631 BlockManagerMaster:172 [NettyMessagingTransport:ServerWorker:pool-11:thread-005] - Block Block-edge-d47_0 location is in state AVAILABLE

This means that BlockManagerMaster is being operated by a networking thread, which I don't think is a good idea.

My suggestion is that we let the RuntimeMaster thread also operate BlockManagerMaster, such that task state transitions and block state transitions are serialized and easy to understand/debug.

If a single RuntimeMaster thread does turn out to be a bottleneck in large-scale experiments (although it's only doing cheap in-memory metadata updating operations) then we may start thinking about introducing a dedicated thread(or a thread pool) for the BlockManagerMaster.