Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4002

make ResourceTrackerService.nodeHeartbeat more concurrent


    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Hadoop Flags:


      We have multiple RPC threads to handle NodeHeartbeatRequest from NMs. By design the method ResourceTrackerService.nodeHeartbeat should be concurrent enough to scale for large clusters.
      But we have a "BIG" lock in NodesListManager.isValidNode which I think it's unnecessary.
      First, the fields "includes" and "excludes" of HostsFileReader are only updated on "refresh nodes". All RPC threads handling node heartbeats are only readers. So RWLock could be used to alow concurrent access by RPC threads.
      Second, since he fields "includes" and "excludes" of HostsFileReader are always updated by "reference assignment", which is atomic in Java, the reader side lock could just be skipped.


        1. 0001-YARN-4002.patch
          3 kB
          Rohith Sharma K S
        2. YARN-4002-lockless-read.patch
          1 kB
          Hong Zhiguo
        3. YARN-4002-rwlock.patch
          4 kB
          Hong Zhiguo
        4. YARN-4002-rwlock-v2.patch
          3 kB
          Rohith Sharma K S
        5. YARN-4002-rwlock-v2.patch
          3 kB
          Hong Zhiguo
        6. YARN-4002-rwlock-v3.patch
          5 kB
          Rohith Sharma K S
        7. YARN-4002-rwlock-v3-rebase.patch
          5 kB
          Rohith Sharma K S
        8. YARN-4002-rwlock-v4.patch
          12 kB
          Rohith Sharma K S
        9. YARN-4002-rwlock-v5.patch
          14 kB
          Rohith Sharma K S
        10. YARN-4002-rwlock-v6.patch
          14 kB
          Rohith Sharma K S
        11. YARN-4002-v0.patch
          3 kB
          Brook Zhou

          Issue Links



              • Assignee:
                zhiguohong Hong Zhiguo
                zhiguohong Hong Zhiguo
              • Votes:
                0 Vote for this issue
                18 Start watching this issue


                • Created: