Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-144 Improve Data Plane Code
  3. NEMO-145

Network threads should not touch BlockManagerMaster

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:

      Description

      While debugging, I've noticed this log.

      INFO 07-07 16:41:48,631 BlockManagerMaster:172 [NettyMessagingTransport:ServerWorker:pool-11:thread-005] - Block Block-edge-d47_0 location is in state AVAILABLE

      This means that BlockManagerMaster is being operated by a networking thread, which I don't think is a good idea.

      My suggestion is that we let the RuntimeMaster thread also operate BlockManagerMaster, such that task state transitions and block state transitions are serialized and easy to understand/debug.

      If a single RuntimeMaster thread does turn out to be a bottleneck in large-scale experiments (although it's only doing cheap in-memory metadata updating operations) then we may start thinking about introducing a dedicated thread(or a thread pool) for the BlockManagerMaster.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              johnyangk John Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: