Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      We might need to prioritize DN RPC over client RPC so that no matter what application do to NN RPC and FSNamesystem's global lock, DN's requests will be processed timely. After a cluster is configured to have service RPC server separated from client RPC server, it is mitigated to some degree with fair FSNamesystem's global lock. Also if the NN global lock can be made more fine grained; such need becomes less important. Still, it will be good to evaluate if this is a good option.

        Activity

        Hide
        Daryn Sharp added a comment -

        With the current global fair lock, I wonder if DNs could effectively starve clients. The block updates may create a glut of write ops at the front of the call queue. With no prioritization, clients create read bubbles in between the write ops.

        I think you've touched upon the angle we are using to attack the problem. Now once I can finally get away from webhdfs hardening which derailed the efforts, we're working on fine grain FSN locking + distinct BM locking + BM has its own RPC service. This should effectively achieve prioritization of DNs with much better performance characteristics overall.

        Show
        Daryn Sharp added a comment - With the current global fair lock, I wonder if DNs could effectively starve clients. The block updates may create a glut of write ops at the front of the call queue. With no prioritization, clients create read bubbles in between the write ops. I think you've touched upon the angle we are using to attack the problem. Now once I can finally get away from webhdfs hardening which derailed the efforts, we're working on fine grain FSN locking + distinct BM locking + BM has its own RPC service. This should effectively achieve prioritization of DNs with much better performance characteristics overall.

          People

          • Assignee:
            Unassigned
            Reporter:
            Ming Ma
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development