Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4002

make ResourceTrackerService.nodeHeartbeat more concurrent

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We have multiple RPC threads to handle NodeHeartbeatRequest from NMs. By design the method ResourceTrackerService.nodeHeartbeat should be concurrent enough to scale for large clusters.
      But we have a "BIG" lock in NodesListManager.isValidNode which I think it's unnecessary.
      First, the fields "includes" and "excludes" of HostsFileReader are only updated on "refresh nodes". All RPC threads handling node heartbeats are only readers. So RWLock could be used to alow concurrent access by RPC threads.
      Second, since he fields "includes" and "excludes" of HostsFileReader are always updated by "reference assignment", which is atomic in Java, the reader side lock could just be skipped.

      1. 0001-YARN-4002.patch
        3 kB
        Rohith Sharma K S
      2. YARN-4002-lockless-read.patch
        1 kB
        Hong Zhiguo
      3. YARN-4002-rwlock.patch
        4 kB
        Hong Zhiguo
      4. YARN-4002-rwlock-v2.patch
        3 kB
        Rohith Sharma K S
      5. YARN-4002-rwlock-v2.patch
        3 kB
        Hong Zhiguo
      6. YARN-4002-rwlock-v3.patch
        5 kB
        Rohith Sharma K S
      7. YARN-4002-rwlock-v3-rebase.patch
        5 kB
        Rohith Sharma K S
      8. YARN-4002-rwlock-v4.patch
        12 kB
        Rohith Sharma K S
      9. YARN-4002-rwlock-v5.patch
        14 kB
        Rohith Sharma K S
      10. YARN-4002-rwlock-v6.patch
        14 kB
        Rohith Sharma K S
      11. YARN-4002-v0.patch
        3 kB
        Brook Zhou

        Issue Links

          Activity

          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks for reporting, Hong Zhiguo. +1 for this proposal, we need to fix this coarse synchronized lock.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks for reporting, Hong Zhiguo . +1 for this proposal, we need to fix this coarse synchronized lock.
          Hide
          xinxianyin Xianyin Xin added a comment -

          +1 for the proposal. Wangda Tan, I think YARN-3091 should also be carried out which is left there for a long time.

          Show
          xinxianyin Xianyin Xin added a comment - +1 for the proposal. Wangda Tan , I think YARN-3091 should also be carried out which is left there for a long time.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Xianyin Xin, YARN-3091 is mostly for grouping all RM locking issues, some of critical items are resolved, other improvements are pending since they didn't become bottleneck yet.

          Show
          leftnoteasy Wangda Tan added a comment - Xianyin Xin , YARN-3091 is mostly for grouping all RM locking issues, some of critical items are resolved, other improvements are pending since they didn't become bottleneck yet.
          Hide
          brookz Brook Zhou added a comment -

          Added a patch for this.

          Show
          brookz Brook Zhou added a comment - Added a patch for this.
          Hide
          brookz Brook Zhou added a comment -

          If this is currently not being worked on, I will assign it to me.

          Show
          brookz Brook Zhou added a comment - If this is currently not being worked on, I will assign it to me.
          Hide
          zhiguohong Hong Zhiguo added a comment -

          I'm working on it. I've proposed 2 different solutions and waiting for specific comments.

          Show
          zhiguohong Hong Zhiguo added a comment - I'm working on it. I've proposed 2 different solutions and waiting for specific comments.
          Hide
          zhiguohong Hong Zhiguo added a comment -

          2 patch for the 2 proposed solutions submitted.

          Show
          zhiguohong Hong Zhiguo added a comment - 2 patch for the 2 proposed solutions submitted.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 30s trunk passed
          +1 compile 0m 25s trunk passed with JDK v1.8.0_66
          +1 compile 0m 30s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 10s trunk passed
          +1 javadoc 0m 22s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 25s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 33s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.8.0_66
          +1 javac 0m 25s the patch passed
          +1 compile 0m 30s the patch passed with JDK v1.7.0_85
          +1 javac 0m 30s the patch passed
          +1 checkstyle 0m 13s the patch passed
          +1 mvnsite 0m 36s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 18s the patch passed
          +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 25s the patch passed with JDK v1.7.0_85
          -1 unit 58m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 60m 7s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          136m 8s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775729/YARN-4002-rwlock.patch
          JIRA Issue YARN-4002
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e0b304008b1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e84d6ca
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9857/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9857/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 30s trunk passed +1 compile 0m 25s trunk passed with JDK v1.8.0_66 +1 compile 0m 30s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 10s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 33s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_66 +1 javac 0m 25s the patch passed +1 compile 0m 30s the patch passed with JDK v1.7.0_85 +1 javac 0m 30s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 25s the patch passed with JDK v1.7.0_85 -1 unit 58m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 60m 7s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 136m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775729/YARN-4002-rwlock.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e0b304008b1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e84d6ca findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9857/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9857/console This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updated patch myself with small correction.

          Show
          rohithsharma Rohith Sharma K S added a comment - Updated patch myself with small correction.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Recently we hit this issue in 2K nodes testing. It is good to go in for branch-2.8.
          nit on the patch : need not to have read lock on method printConfiguredHosts since it is called from refreshHostsReader which is write locked.

          Show
          rohithsharma Rohith Sharma K S added a comment - Recently we hit this issue in 2K nodes testing. It is good to go in for branch-2.8. nit on the patch : need not to have read lock on method printConfiguredHosts since it is called from refreshHostsReader which is write locked.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 27s trunk passed
          +1 compile 0m 26s trunk passed with JDK v1.8.0_72
          +1 compile 0m 29s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 4s trunk passed
          +1 javadoc 0m 21s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 24s the patch passed with JDK v1.8.0_72
          +1 javac 0m 24s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.7.0_95
          +1 javac 0m 26s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 15s the patch passed
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
          -1 unit 70m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72.
          -1 unit 71m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          158m 14s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790697/0001-YARN-4002.patch
          JIRA Issue YARN-4002
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 78a7e9f839bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d251e55
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10676/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10676/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 27s trunk passed +1 compile 0m 26s trunk passed with JDK v1.8.0_72 +1 compile 0m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 4s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_72 +1 javac 0m 24s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 15s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 70m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. -1 unit 71m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 158m 14s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790697/0001-YARN-4002.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 78a7e9f839bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d251e55 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10676/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10676/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Wangda Tan would you like to have look at patch? If no comments I will go ahead with committing it.

          Show
          rohithsharma Rohith Sharma K S added a comment - Wangda Tan would you like to have look at patch? If no comments I will go ahead with committing it.
          Hide
          zhiguohong Hong Zhiguo added a comment -

          Hi, Rohith Sharma K S, thanks for the refinement.
          But why don't take the lockless version?

          Show
          zhiguohong Hong Zhiguo added a comment - Hi, Rohith Sharma K S , thanks for the refinement. But why don't take the lockless version?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          All the method in HostsFileReader are synchronized. And method isValidNode does 2 separate calls to hostsReader. There could be a scenario (if lock is not used) where after executing hostsReader.getHosts(); , hostreader can do refresh which gives updated result for hostsReader.getExcludedHosts(); but stale host details for getHosts method. Lockless read might mix up old and new values which is incorrect.

          Show
          rohithsharma Rohith Sharma K S added a comment - All the method in HostsFileReader are synchronized. And method isValidNode does 2 separate calls to hostsReader. There could be a scenario (if lock is not used) where after executing hostsReader.getHosts(); , hostreader can do refresh which gives updated result for hostsReader.getExcludedHosts(); but stale host details for getHosts method. Lockless read might mix up old and new values which is incorrect.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Rohith Sharma K S, Hong Zhiguo,
          Thanks for working on this patch, generally looks good.
          Do you think we need to acquire readlock in printConfiguredHosts and setDecomissionedNMsMetrics?

          Show
          leftnoteasy Wangda Tan added a comment - Rohith Sharma K S , Hong Zhiguo , Thanks for working on this patch, generally looks good. Do you think we need to acquire readlock in printConfiguredHosts and setDecomissionedNMsMetrics?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Thanks Wangda Tan for the looking at the patch..
          I was thought about adding these 2 places readlock, but after looking into caller of these 2 methods I felt it is not really required.

          1. Method setDecomissionedNMsMetrics is called during service init, so this will be called during service initialization.
          2. Method {{printConfiguredHosts }} is called during service init and refreshNodes.
            1. Once again, for service init, I do not think we need really acquire readlock.
            2. For refresh Node,{{printConfiguredHosts }} is with in the write lock, it is safe enough to go without read lock.

          As of now, without acquiring read lock would not cause any problem. In future, if any new method calling these methods need to think of acquiring read lock.

          Show
          rohithsharma Rohith Sharma K S added a comment - Thanks Wangda Tan for the looking at the patch.. I was thought about adding these 2 places readlock, but after looking into caller of these 2 methods I felt it is not really required. Method setDecomissionedNMsMetrics is called during service init, so this will be called during service initialization. Method {{printConfiguredHosts }} is called during service init and refreshNodes. Once again, for service init, I do not think we need really acquire readlock. For refresh Node,{{printConfiguredHosts }} is with in the write lock, it is safe enough to go without read lock. As of now, without acquiring read lock would not cause any problem. In future, if any new method calling these methods need to think of acquiring read lock.
          Hide
          zhiguohong Hong Zhiguo added a comment -

          Uploaded YARN-4002-rwlock-v2.patch for an improvement: make the read side critical section smaller.

            this.hostsReadLock.lock();
              try {
                hostsList = hostsReader.getHosts();
                excludeList = hostsReader.getExcludedHosts();
              } finally {
                this.hostsReadLock.unlock();
              }
          

          As explained by Rohith Sharma K S, this prevents mixing up old value of hostsReader.getHosts() and new value of hostsReader.getExcludedHosts(). And this is the only reason someone may prefer rwlock solution than lockless one.

          If the mixing up is not thought (for example, by meself) a problem, lockless solution is good engouth.

          Show
          zhiguohong Hong Zhiguo added a comment - Uploaded YARN-4002 -rwlock-v2.patch for an improvement: make the read side critical section smaller. this .hostsReadLock.lock(); try { hostsList = hostsReader.getHosts(); excludeList = hostsReader.getExcludedHosts(); } finally { this .hostsReadLock.unlock(); } As explained by Rohith Sharma K S , this prevents mixing up old value of hostsReader.getHosts() and new value of hostsReader.getExcludedHosts(). And this is the only reason someone may prefer rwlock solution than lockless one. If the mixing up is not thought (for example, by meself) a problem, lockless solution is good engouth.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Hi Hong Zhiguo
          In the latest patch YARN-4002-rwlock-v2.patch isValidNode return statement is out of read lock. This should be inside readlock.

          return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList
          	        .contains(ip))
          	        && !(excludeList.contains(hostName) || excludeList.contains(ip));
          

          Earlier patch 0001-YARN-4002.patch looks fine to me..

          Show
          rohithsharma Rohith Sharma K S added a comment - Hi Hong Zhiguo In the latest patch YARN-4002 -rwlock-v2.patch isValidNode return statement is out of read lock. This should be inside readlock. return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList .contains(ip)) && !(excludeList.contains(hostName) || excludeList.contains(ip)); Earlier patch 0001- YARN-4002 .patch looks fine to me..
          Hide
          zhiguohong Hong Zhiguo added a comment -

          Including return statement into readlock critical seciton doesn't make difference except longer critical section and worse performance.
          Since hostsList and excludeList of hostReader is updated by reference assignment, no race condition would exist even the lookup is not protected by readlock.

          Show
          zhiguohong Hong Zhiguo added a comment - Including return statement into readlock critical seciton doesn't make difference except longer critical section and worse performance. Since hostsList and excludeList of hostReader is updated by reference assignment, no race condition would exist even the lookup is not protected by readlock.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          That make sense to me. Thanks

          Show
          rohithsharma Rohith Sharma K S added a comment - That make sense to me. Thanks
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          It looks jenkins is triggered long back, I will kick of jenkins again to check QA report.

          Show
          rohithsharma Rohith Sharma K S added a comment - It looks jenkins is triggered long back, I will kick of jenkins again to check QA report.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Re-attaching the patch to trigger jenkins

          Show
          rohithsharma Rohith Sharma K S added a comment - Re-attaching the patch to trigger jenkins
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 46s trunk passed
          +1 compile 0m 28s trunk passed with JDK v1.8.0_74
          +1 compile 0m 29s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 34s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 9s trunk passed
          +1 javadoc 0m 21s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.8.0_74
          +1 javac 0m 26s the patch passed
          +1 compile 0m 27s the patch passed with JDK v1.7.0_95
          +1 javac 0m 27s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 17s the patch passed
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 25s the patch passed with JDK v1.7.0_95
          -1 unit 65m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74.
          -1 unit 66m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          149m 3s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestRMAdminService
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestRMAdminService
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796797/YARN-4002-rwlock-v2.patch
          JIRA Issue YARN-4002
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 031041c1b720 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / da614ca
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10954/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10954/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 46s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_74 +1 compile 0m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 9s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 30s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_74 +1 javac 0m 26s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_95 +1 javac 0m 27s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 17s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 25s the patch passed with JDK v1.7.0_95 -1 unit 65m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74. -1 unit 66m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 149m 3s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796797/YARN-4002-rwlock-v2.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 031041c1b720 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / da614ca Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10954/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10954/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Rebased the patch as per new trunk code..

          Show
          rohithsharma Rohith Sharma K S added a comment - Rebased the patch as per new trunk code..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 47s trunk passed
          +1 compile 0m 28s trunk passed with JDK v1.8.0_77
          +1 compile 0m 29s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 5s trunk passed
          +1 javadoc 0m 23s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 28s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 24s the patch passed with JDK v1.8.0_77
          +1 javac 0m 24s the patch passed
          +1 compile 0m 27s the patch passed with JDK v1.7.0_95
          +1 javac 0m 27s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 15s the patch passed
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
          -1 unit 77m 19s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77.
          -1 unit 55m 45s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          149m 57s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.webapp.TestRMWithCSRFFilter
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.webapp.TestRMWithCSRFFilter
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797464/YARN-4002-rwlock-v3.patch
          JIRA Issue YARN-4002
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 40b041cce8a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3c18a53
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10982/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10982/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 47s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_77 +1 compile 0m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 23s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 28s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 31s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_77 +1 javac 0m 24s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_95 +1 javac 0m 27s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 15s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 77m 19s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77. -1 unit 55m 45s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 149m 57s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.webapp.TestRMWithCSRFFilter   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.webapp.TestRMWithCSRFFilter   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797464/YARN-4002-rwlock-v3.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 40b041cce8a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3c18a53 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10982/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10982/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Rohith Sharma K S, is there any update for the latest patch? Cleaning up 2.8.0 tickets now.

          Thanks,

          Show
          leftnoteasy Wangda Tan added a comment - Rohith Sharma K S , is there any update for the latest patch? Cleaning up 2.8.0 tickets now. Thanks,
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Rebased v3 patch trigger jenkin.
          Wangda Tan can you look into patch please.

          Show
          rohithsharma Rohith Sharma K S added a comment - Rebased v3 patch trigger jenkin. Wangda Tan can you look into patch please.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 22s trunk passed
          +1 compile 0m 37s trunk passed with JDK v1.8.0_91
          +1 compile 0m 36s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 42s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 23s trunk passed
          +1 javadoc 0m 30s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 37s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_91
          +1 javac 0m 37s the patch passed
          +1 compile 0m 33s the patch passed with JDK v1.7.0_95
          +1 javac 0m 33s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 40s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 30s the patch passed
          +1 javadoc 0m 28s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95
          -1 unit 35m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          -1 unit 35m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          91m 22s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803569/YARN-4002-rwlock-v3-rebase.patch
          JIRA Issue YARN-4002
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 255f5ef244f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d464f4d
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11427/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11427/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 22s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_91 +1 compile 0m 36s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 42s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 23s trunk passed +1 javadoc 0m 30s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 37s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_91 +1 javac 0m 37s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_95 +1 javac 0m 33s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 40s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 30s the patch passed +1 javadoc 0m 28s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95 -1 unit 35m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 35m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 91m 22s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803569/YARN-4002-rwlock-v3-rebase.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 255f5ef244f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d464f4d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11427/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11427/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Rohith Sharma K S,

          Few suggestions:
          1) hostsWrite/ReadLock -> hostsListRead/WriteLock

          2) Return statement should included by try {...}, like:

          387	    return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList
          388	        .contains(ip))
          389	        && !(excludeList.contains(hostName) || excludeList.contains(ip));
          

          Otherwise inconsistency will happen.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Rohith Sharma K S , Few suggestions: 1) hostsWrite/ReadLock -> hostsListRead/WriteLock 2) Return statement should included by try {... }, like: 387 return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList 388 .contains(ip)) 389 && !(excludeList.contains(hostName) || excludeList.contains(ip)); Otherwise inconsistency will happen.
          Hide
          jianhe Jian He added a comment -

          Rohith Sharma K S, the patch may not work because
          hostsReader.refresh() tries to acquire the synchronized lock of HostsFileReader
          And getHosts / getExcludedHosts also tries to acquire the synchronized lock.
          So it is no difference than before.

          Show
          jianhe Jian He added a comment - Rohith Sharma K S , the patch may not work because hostsReader.refresh() tries to acquire the synchronized lock of HostsFileReader And getHosts / getExcludedHosts also tries to acquire the synchronized lock. So it is no difference than before.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          For the second suggestion, IIRC I was suggested to original patch provider and his reply was earlier-comment. His answer made sense to me. If it is not OK , I will change the patch.

          Show
          rohithsharma Rohith Sharma K S added a comment - For the second suggestion, IIRC I was suggested to original patch provider and his reply was earlier-comment . His answer made sense to me. If it is not OK , I will change the patch.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I agree this wont solve 100%, but it reduces lock time at one layer down.
          Do you mean HostsFileReader class should changed to read/write lock?

          Show
          rohithsharma Rohith Sharma K S added a comment - I agree this wont solve 100%, but it reduces lock time at one layer down. Do you mean HostsFileReader class should changed to read/write lock?
          Hide
          jianhe Jian He added a comment -

          Yeah, I think so. because otherwise node heartbeat still competes on the HostsFileReader synchronized lock.

          Show
          jianhe Jian He added a comment - Yeah, I think so. because otherwise node heartbeat still competes on the HostsFileReader synchronized lock.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updated the patch addressing comments from Wangda and Jian He.

          1. Renamed variables to hostsListRead/WriteLock
          2. Added read/write lock in HostsFileReader

          ReadLock for the methods

          1. In class HostFileReader : getHosts() and getExcludedHosts
          2. In class NodesListManager : isValidNode and isUntrackedNode

          WriteLock for the methods

          1. In class HostFileReader :
            1. refresh()
            2. refresh(InputStream inFileInputStream, InputStream exFileInputStream)
            3. setIncludesFile()
            4. setExcludesFile
            5. updateFileNames
          2. In class NodesListManager : refreshHostsReader
          Show
          rohithsharma Rohith Sharma K S added a comment - Updated the patch addressing comments from Wangda and Jian He. Renamed variables to hostsListRead/WriteLock Added read/write lock in HostsFileReader ReadLock for the methods In class HostFileReader : getHosts() and getExcludedHosts In class NodesListManager : isValidNode and isUntrackedNode WriteLock for the methods In class HostFileReader : refresh() refresh(InputStream inFileInputStream, InputStream exFileInputStream) setIncludesFile() setExcludesFile updateFileNames In class NodesListManager : refreshHostsReader
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 42s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 53s Maven dependency ordering for branch
          +1 mvninstall 6m 37s trunk passed
          +1 compile 5m 57s trunk passed with JDK v1.8.0_91
          +1 compile 6m 50s trunk passed with JDK v1.7.0_101
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 1m 36s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 2m 42s trunk passed
          +1 javadoc 1m 16s trunk passed with JDK v1.8.0_91
          +1 javadoc 1m 35s trunk passed with JDK v1.7.0_101
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 14s the patch passed
          +1 compile 5m 53s the patch passed with JDK v1.8.0_91
          +1 javac 5m 53s the patch passed
          +1 compile 6m 53s the patch passed with JDK v1.7.0_101
          +1 javac 6m 53s the patch passed
          -1 checkstyle 1m 24s root: patch generated 2 new + 16 unchanged - 2 fixed = 18 total (was 18)
          +1 mvnsite 1m 32s the patch passed
          +1 mvneclipse 0m 30s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 12s the patch passed
          +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 31s the patch passed with JDK v1.7.0_101
          +1 unit 7m 3s hadoop-common in the patch passed with JDK v1.8.0_91.
          -1 unit 29m 47s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.7.0_101.
          -1 unit 31m 13s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          142m 59s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
          JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
            hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804119/YARN-4002-rwlock-v4.patch
          JIRA Issue YARN-4002
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7f4e526ea34d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ad94411
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11476/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11476/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 42s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 6m 37s trunk passed +1 compile 5m 57s trunk passed with JDK v1.8.0_91 +1 compile 6m 50s trunk passed with JDK v1.7.0_101 +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 2m 42s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 35s trunk passed with JDK v1.7.0_101 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 5m 53s the patch passed with JDK v1.8.0_91 +1 javac 5m 53s the patch passed +1 compile 6m 53s the patch passed with JDK v1.7.0_101 +1 javac 6m 53s the patch passed -1 checkstyle 1m 24s root: patch generated 2 new + 16 unchanged - 2 fixed = 18 total (was 18) +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 12s the patch passed +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 31s the patch passed with JDK v1.7.0_101 +1 unit 7m 3s hadoop-common in the patch passed with JDK v1.8.0_91. -1 unit 29m 47s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.7.0_101. -1 unit 31m 13s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 142m 59s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804119/YARN-4002-rwlock-v4.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7f4e526ea34d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ad94411 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11476/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11476/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Rohith Sharma K S, thanks for updating the patch.
          I think we may eliminate the read/write lock in NodeListManager. we can move the NodeListManager#refreshHostsReader into the HostsFileReader itself - refreshHostsReader can take includeFiles and excludeFiles as parameters
          For the getHosts and getExcludedHosts, we can pass in two empty sets together as parameters.

          Show
          jianhe Jian He added a comment - Rohith Sharma K S , thanks for updating the patch. I think we may eliminate the read/write lock in NodeListManager. we can move the NodeListManager#refreshHostsReader into the HostsFileReader itself - refreshHostsReader can take includeFiles and excludeFiles as parameters For the getHosts and getExcludedHosts, we can pass in two empty sets together as parameters.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Update the patch with following changes
          1. Added new method refresh(String includesFile, String excludesFile) to refresh include/exclude lists
          2. Added a new method getHostDetails that return include/exclude lists
          3. Removed synchronization from NodesListManager.

          Show
          rohithsharma Rohith Sharma K S added a comment - Update the patch with following changes 1. Added new method refresh(String includesFile, String excludesFile) to refresh include/exclude lists 2. Added a new method getHostDetails that return include/exclude lists 3. Removed synchronization from NodesListManager.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 7m 34s trunk passed
          +1 compile 7m 42s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 1m 48s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 2m 39s trunk passed
          +1 javadoc 1m 37s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 8m 14s the patch passed
          +1 javac 8m 14s the patch passed
          -1 checkstyle 1m 39s root: patch generated 11 new + 26 unchanged - 2 fixed = 37 total (was 28)
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 31s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 41s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 16s the patch passed
          +1 unit 7m 38s hadoop-common in the patch passed.
          -1 unit 35m 52s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          86m 27s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            org.apache.hadoop.util.HostsFileReader.refresh(String, String) does not release lock on all exception paths At HostsFileReader.java:lock on all exception paths At HostsFileReader.java:[line 127]
          Failed junit tests hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804368/YARN-4002-rwlock-v5.patch
          JIRA Issue YARN-4002
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 17c4e2e271df 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9fe5828
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11494/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11494/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 7m 34s trunk passed +1 compile 7m 42s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 1m 48s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 2m 39s trunk passed +1 javadoc 1m 37s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 8m 14s the patch passed +1 javac 8m 14s the patch passed -1 checkstyle 1m 39s root: patch generated 11 new + 26 unchanged - 2 fixed = 37 total (was 28) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 41s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 16s the patch passed +1 unit 7m 38s hadoop-common in the patch passed. -1 unit 35m 52s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 86m 27s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   org.apache.hadoop.util.HostsFileReader.refresh(String, String) does not release lock on all exception paths At HostsFileReader.java:lock on all exception paths At HostsFileReader.java: [line 127] Failed junit tests hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804368/YARN-4002-rwlock-v5.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 17c4e2e271df 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9fe5828 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11494/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11494/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Rohith Sharma K S, could you fix the findbugs warning ?
          also, setExcludesFile, setIncludesFile, updateFileNames need not acquire the writelock or can they just be removed ?

          Show
          jianhe Jian He added a comment - Rohith Sharma K S , could you fix the findbugs warning ? also, setExcludesFile, setIncludesFile, updateFileNames need not acquire the writelock or can they just be removed ?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          also, setExcludesFile, setIncludesFile, updateFileNames need not acquire the writelock or can they just be removed ?

          I see class is annotated with limited private which are with in the hadoop project only. But still I feel since it util method in hadoop-common may be any other projects might be used. Should it still get removed?

          Show
          rohithsharma Rohith Sharma K S added a comment - also, setExcludesFile, setIncludesFile, updateFileNames need not acquire the writelock or can they just be removed ? I see class is annotated with limited private which are with in the hadoop project only. But still I feel since it util method in hadoop-common may be any other projects might be used. Should it still get removed?
          Hide
          jianhe Jian He added a comment -

          ok, let's keep it, I think the lock can be removed ?

          Show
          jianhe Jian He added a comment - ok, let's keep it, I think the lock can be removed ?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updating the patch with below changes from previous patch

          1. fixed findbugs and checkstyle warning
          2. for the setter methods, removed writelock
          Show
          rohithsharma Rohith Sharma K S added a comment - Updating the patch with below changes from previous patch fixed findbugs and checkstyle warning for the setter methods, removed writelock
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 6m 54s trunk passed
          +1 compile 8m 3s trunk passed
          +1 checkstyle 1m 35s trunk passed
          +1 mvnsite 1m 52s trunk passed
          +1 mvneclipse 0m 32s trunk passed
          +1 findbugs 2m 37s trunk passed
          +1 javadoc 1m 33s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 7m 56s the patch passed
          +1 javac 7m 56s the patch passed
          -1 checkstyle 1m 33s root: patch generated 2 new + 26 unchanged - 2 fixed = 28 total (was 28)
          +1 mvnsite 1m 42s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 49s the patch passed
          +1 javadoc 1m 28s the patch passed
          +1 unit 8m 30s hadoop-common in the patch passed.
          -1 unit 39m 57s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          91m 4s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMAdminService
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804919/YARN-4002-rwlock-v6.patch
          JIRA Issue YARN-4002
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e2aa0bfbb81c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 141873c
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11555/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11555/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 54s trunk passed +1 compile 8m 3s trunk passed +1 checkstyle 1m 35s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 32s trunk passed +1 findbugs 2m 37s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 7m 56s the patch passed +1 javac 7m 56s the patch passed -1 checkstyle 1m 33s root: patch generated 2 new + 26 unchanged - 2 fixed = 28 total (was 28) +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 49s the patch passed +1 javadoc 1m 28s the patch passed +1 unit 8m 30s hadoop-common in the patch passed. -1 unit 39m 57s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 91m 4s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804919/YARN-4002-rwlock-v6.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e2aa0bfbb81c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 141873c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11555/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11555/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Committed to trunk, branch-2, branch-2.8
          Thanks Rohith, Zhiguo !
          Thanks Wangda for reviewing the patch

          Show
          jianhe Jian He added a comment - Committed to trunk, branch-2, branch-2.8 Thanks Rohith, Zhiguo ! Thanks Wangda for reviewing the patch
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9825 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9825/)
          YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. (jianhe: rev feb90ffcca536e7deac50976b8a8774450fe089f)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9825 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9825/ ) YARN-4002 . Make ResourceTrackerService#nodeHeartbeat more concurrent. (jianhe: rev feb90ffcca536e7deac50976b8a8774450fe089f) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
          Hide
          jy03843796@gmail.com Guoyao Xu added a comment - - edited

          I think that might not be a big problem. Because hostsReader.refresh() operation does not happen during each heartbeat, and actually
          this operation is not related to isValid() operation. And the frequency of refresh() operation is much less than isValid() operation (nodes number times of call for each heartbeat interval--- default 1 second).
          Then then the synchronized lock of refresh() may not be a problem of isValid(), and it might
          not be harm to the concurrency of heartbeat call.

          Show
          jy03843796@gmail.com Guoyao Xu added a comment - - edited I think that might not be a big problem. Because hostsReader.refresh() operation does not happen during each heartbeat, and actually this operation is not related to isValid() operation. And the frequency of refresh() operation is much less than isValid() operation (nodes number times of call for each heartbeat interval--- default 1 second). Then then the synchronized lock of refresh() may not be a problem of isValid(), and it might not be harm to the concurrency of heartbeat call.
          Hide
          jlowe Jason Lowe added a comment -

          FYI we're finding this change to be quite expensive on large clusters. See HADOOP-14412.

          Show
          jlowe Jason Lowe added a comment - FYI we're finding this change to be quite expensive on large clusters. See HADOOP-14412 .

            People

            • Assignee:
              zhiguohong Hong Zhiguo
              Reporter:
              zhiguohong Hong Zhiguo
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development