Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4002

make ResourceTrackerService.nodeHeartbeat more concurrent

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We have multiple RPC threads to handle NodeHeartbeatRequest from NMs. By design the method ResourceTrackerService.nodeHeartbeat should be concurrent enough to scale for large clusters.
      But we have a "BIG" lock in NodesListManager.isValidNode which I think it's unnecessary.
      First, the fields "includes" and "excludes" of HostsFileReader are only updated on "refresh nodes". All RPC threads handling node heartbeats are only readers. So RWLock could be used to alow concurrent access by RPC threads.
      Second, since he fields "includes" and "excludes" of HostsFileReader are always updated by "reference assignment", which is atomic in Java, the reader side lock could just be skipped.

      1. 0001-YARN-4002.patch
        3 kB
        Rohith Sharma K S
      2. YARN-4002-lockless-read.patch
        1 kB
        Hong Zhiguo
      3. YARN-4002-rwlock.patch
        4 kB
        Hong Zhiguo
      4. YARN-4002-rwlock-v2.patch
        3 kB
        Rohith Sharma K S
      5. YARN-4002-rwlock-v2.patch
        3 kB
        Hong Zhiguo
      6. YARN-4002-rwlock-v3.patch
        5 kB
        Rohith Sharma K S
      7. YARN-4002-rwlock-v3-rebase.patch
        5 kB
        Rohith Sharma K S
      8. YARN-4002-rwlock-v4.patch
        12 kB
        Rohith Sharma K S
      9. YARN-4002-rwlock-v5.patch
        14 kB
        Rohith Sharma K S
      10. YARN-4002-rwlock-v6.patch
        14 kB
        Rohith Sharma K S
      11. YARN-4002-v0.patch
        3 kB
        Brook Zhou

        Activity

        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks for reporting, Hong Zhiguo. +1 for this proposal, we need to fix this coarse synchronized lock.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks for reporting, Hong Zhiguo . +1 for this proposal, we need to fix this coarse synchronized lock.
        Hide
        xinxianyin Xianyin Xin added a comment -

        +1 for the proposal. Wangda Tan, I think YARN-3091 should also be carried out which is left there for a long time.

        Show
        xinxianyin Xianyin Xin added a comment - +1 for the proposal. Wangda Tan , I think YARN-3091 should also be carried out which is left there for a long time.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Xianyin Xin, YARN-3091 is mostly for grouping all RM locking issues, some of critical items are resolved, other improvements are pending since they didn't become bottleneck yet.

        Show
        leftnoteasy Wangda Tan added a comment - Xianyin Xin , YARN-3091 is mostly for grouping all RM locking issues, some of critical items are resolved, other improvements are pending since they didn't become bottleneck yet.
        Hide
        brookz Brook Zhou added a comment -

        Added a patch for this.

        Show
        brookz Brook Zhou added a comment - Added a patch for this.
        Hide
        brookz Brook Zhou added a comment -

        If this is currently not being worked on, I will assign it to me.

        Show
        brookz Brook Zhou added a comment - If this is currently not being worked on, I will assign it to me.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        I'm working on it. I've proposed 2 different solutions and waiting for specific comments.

        Show
        zhiguohong Hong Zhiguo added a comment - I'm working on it. I've proposed 2 different solutions and waiting for specific comments.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        2 patch for the 2 proposed solutions submitted.

        Show
        zhiguohong Hong Zhiguo added a comment - 2 patch for the 2 proposed solutions submitted.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 30s trunk passed
        +1 compile 0m 25s trunk passed with JDK v1.8.0_66
        +1 compile 0m 30s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 10s trunk passed
        +1 javadoc 0m 22s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 25s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.8.0_66
        +1 javac 0m 25s the patch passed
        +1 compile 0m 30s the patch passed with JDK v1.7.0_85
        +1 javac 0m 30s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 18s the patch passed
        +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 25s the patch passed with JDK v1.7.0_85
        -1 unit 58m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 60m 7s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        136m 8s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
        JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775729/YARN-4002-rwlock.patch
        JIRA Issue YARN-4002
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e0b304008b1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e84d6ca
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9857/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9857/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 30s trunk passed +1 compile 0m 25s trunk passed with JDK v1.8.0_66 +1 compile 0m 30s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 10s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 33s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_66 +1 javac 0m 25s the patch passed +1 compile 0m 30s the patch passed with JDK v1.7.0_85 +1 javac 0m 30s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 25s the patch passed with JDK v1.7.0_85 -1 unit 58m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 60m 7s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 136m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775729/YARN-4002-rwlock.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e0b304008b1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e84d6ca findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9857/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9857/console This message was automatically generated.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Updated patch myself with small correction.

        Show
        rohithsharma Rohith Sharma K S added a comment - Updated patch myself with small correction.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Recently we hit this issue in 2K nodes testing. It is good to go in for branch-2.8.
        nit on the patch : need not to have read lock on method printConfiguredHosts since it is called from refreshHostsReader which is write locked.

        Show
        rohithsharma Rohith Sharma K S added a comment - Recently we hit this issue in 2K nodes testing. It is good to go in for branch-2.8. nit on the patch : need not to have read lock on method printConfiguredHosts since it is called from refreshHostsReader which is write locked.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 27s trunk passed
        +1 compile 0m 26s trunk passed with JDK v1.8.0_72
        +1 compile 0m 29s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 4s trunk passed
        +1 javadoc 0m 21s trunk passed with JDK v1.8.0_72
        +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 29s the patch passed
        +1 compile 0m 24s the patch passed with JDK v1.8.0_72
        +1 javac 0m 24s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.7.0_95
        +1 javac 0m 26s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 15s the patch passed
        +1 javadoc 0m 19s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
        -1 unit 70m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72.
        -1 unit 71m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        158m 14s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
        JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790697/0001-YARN-4002.patch
        JIRA Issue YARN-4002
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 78a7e9f839bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d251e55
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10676/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10676/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 27s trunk passed +1 compile 0m 26s trunk passed with JDK v1.8.0_72 +1 compile 0m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 4s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_72 +1 javac 0m 24s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 15s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 70m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. -1 unit 71m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 158m 14s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790697/0001-YARN-4002.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 78a7e9f839bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d251e55 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10676/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10676/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10676/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Wangda Tan would you like to have look at patch? If no comments I will go ahead with committing it.

        Show
        rohithsharma Rohith Sharma K S added a comment - Wangda Tan would you like to have look at patch? If no comments I will go ahead with committing it.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Hi, Rohith Sharma K S, thanks for the refinement.
        But why don't take the lockless version?

        Show
        zhiguohong Hong Zhiguo added a comment - Hi, Rohith Sharma K S , thanks for the refinement. But why don't take the lockless version?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        All the method in HostsFileReader are synchronized. And method isValidNode does 2 separate calls to hostsReader. There could be a scenario (if lock is not used) where after executing hostsReader.getHosts(); , hostreader can do refresh which gives updated result for hostsReader.getExcludedHosts(); but stale host details for getHosts method. Lockless read might mix up old and new values which is incorrect.

        Show
        rohithsharma Rohith Sharma K S added a comment - All the method in HostsFileReader are synchronized. And method isValidNode does 2 separate calls to hostsReader. There could be a scenario (if lock is not used) where after executing hostsReader.getHosts(); , hostreader can do refresh which gives updated result for hostsReader.getExcludedHosts(); but stale host details for getHosts method. Lockless read might mix up old and new values which is incorrect.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rohith Sharma K S, Hong Zhiguo,
        Thanks for working on this patch, generally looks good.
        Do you think we need to acquire readlock in printConfiguredHosts and setDecomissionedNMsMetrics?

        Show
        leftnoteasy Wangda Tan added a comment - Rohith Sharma K S , Hong Zhiguo , Thanks for working on this patch, generally looks good. Do you think we need to acquire readlock in printConfiguredHosts and setDecomissionedNMsMetrics?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Thanks Wangda Tan for the looking at the patch..
        I was thought about adding these 2 places readlock, but after looking into caller of these 2 methods I felt it is not really required.

        1. Method setDecomissionedNMsMetrics is called during service init, so this will be called during service initialization.
        2. Method {{printConfiguredHosts }} is called during service init and refreshNodes.
          1. Once again, for service init, I do not think we need really acquire readlock.
          2. For refresh Node,{{printConfiguredHosts }} is with in the write lock, it is safe enough to go without read lock.

        As of now, without acquiring read lock would not cause any problem. In future, if any new method calling these methods need to think of acquiring read lock.

        Show
        rohithsharma Rohith Sharma K S added a comment - Thanks Wangda Tan for the looking at the patch.. I was thought about adding these 2 places readlock, but after looking into caller of these 2 methods I felt it is not really required. Method setDecomissionedNMsMetrics is called during service init, so this will be called during service initialization. Method {{printConfiguredHosts }} is called during service init and refreshNodes. Once again, for service init, I do not think we need really acquire readlock. For refresh Node,{{printConfiguredHosts }} is with in the write lock, it is safe enough to go without read lock. As of now, without acquiring read lock would not cause any problem. In future, if any new method calling these methods need to think of acquiring read lock.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Uploaded YARN-4002-rwlock-v2.patch for an improvement: make the read side critical section smaller.

          this.hostsReadLock.lock();
            try {
              hostsList = hostsReader.getHosts();
              excludeList = hostsReader.getExcludedHosts();
            } finally {
              this.hostsReadLock.unlock();
            }
        

        As explained by Rohith Sharma K S, this prevents mixing up old value of hostsReader.getHosts() and new value of hostsReader.getExcludedHosts(). And this is the only reason someone may prefer rwlock solution than lockless one.

        If the mixing up is not thought (for example, by meself) a problem, lockless solution is good engouth.

        Show
        zhiguohong Hong Zhiguo added a comment - Uploaded YARN-4002 -rwlock-v2.patch for an improvement: make the read side critical section smaller. this .hostsReadLock.lock(); try { hostsList = hostsReader.getHosts(); excludeList = hostsReader.getExcludedHosts(); } finally { this .hostsReadLock.unlock(); } As explained by Rohith Sharma K S , this prevents mixing up old value of hostsReader.getHosts() and new value of hostsReader.getExcludedHosts(). And this is the only reason someone may prefer rwlock solution than lockless one. If the mixing up is not thought (for example, by meself) a problem, lockless solution is good engouth.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Hi Hong Zhiguo
        In the latest patch YARN-4002-rwlock-v2.patch isValidNode return statement is out of read lock. This should be inside readlock.

        return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList
        	        .contains(ip))
        	        && !(excludeList.contains(hostName) || excludeList.contains(ip));
        

        Earlier patch 0001-YARN-4002.patch looks fine to me..

        Show
        rohithsharma Rohith Sharma K S added a comment - Hi Hong Zhiguo In the latest patch YARN-4002 -rwlock-v2.patch isValidNode return statement is out of read lock. This should be inside readlock. return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList .contains(ip)) && !(excludeList.contains(hostName) || excludeList.contains(ip)); Earlier patch 0001- YARN-4002 .patch looks fine to me..
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Including return statement into readlock critical seciton doesn't make difference except longer critical section and worse performance.
        Since hostsList and excludeList of hostReader is updated by reference assignment, no race condition would exist even the lookup is not protected by readlock.

        Show
        zhiguohong Hong Zhiguo added a comment - Including return statement into readlock critical seciton doesn't make difference except longer critical section and worse performance. Since hostsList and excludeList of hostReader is updated by reference assignment, no race condition would exist even the lookup is not protected by readlock.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        That make sense to me. Thanks

        Show
        rohithsharma Rohith Sharma K S added a comment - That make sense to me. Thanks
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        It looks jenkins is triggered long back, I will kick of jenkins again to check QA report.

        Show
        rohithsharma Rohith Sharma K S added a comment - It looks jenkins is triggered long back, I will kick of jenkins again to check QA report.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Re-attaching the patch to trigger jenkins

        Show
        rohithsharma Rohith Sharma K S added a comment - Re-attaching the patch to trigger jenkins
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 46s trunk passed
        +1 compile 0m 28s trunk passed with JDK v1.8.0_74
        +1 compile 0m 29s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 34s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 9s trunk passed
        +1 javadoc 0m 21s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.8.0_74
        +1 javac 0m 26s the patch passed
        +1 compile 0m 27s the patch passed with JDK v1.7.0_95
        +1 javac 0m 27s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 17s the patch passed
        +1 javadoc 0m 19s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 25s the patch passed with JDK v1.7.0_95
        -1 unit 65m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74.
        -1 unit 66m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        149m 3s



        Reason Tests
        JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestRMAdminService
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
        JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestRMAdminService
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796797/YARN-4002-rwlock-v2.patch
        JIRA Issue YARN-4002
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 031041c1b720 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / da614ca
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10954/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10954/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 46s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_74 +1 compile 0m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 9s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 30s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_74 +1 javac 0m 26s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_95 +1 javac 0m 27s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 17s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 25s the patch passed with JDK v1.7.0_95 -1 unit 65m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74. -1 unit 66m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 149m 3s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796797/YARN-4002-rwlock-v2.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 031041c1b720 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / da614ca Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10954/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10954/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10954/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Rebased the patch as per new trunk code..

        Show
        rohithsharma Rohith Sharma K S added a comment - Rebased the patch as per new trunk code..
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 47s trunk passed
        +1 compile 0m 28s trunk passed with JDK v1.8.0_77
        +1 compile 0m 29s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 1m 5s trunk passed
        +1 javadoc 0m 23s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 28s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 24s the patch passed with JDK v1.8.0_77
        +1 javac 0m 24s the patch passed
        +1 compile 0m 27s the patch passed with JDK v1.7.0_95
        +1 javac 0m 27s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 34s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 15s the patch passed
        +1 javadoc 0m 19s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
        -1 unit 77m 19s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77.
        -1 unit 55m 45s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        149m 57s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.webapp.TestRMWithCSRFFilter
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
        JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
        JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.webapp.TestRMWithCSRFFilter
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797464/YARN-4002-rwlock-v3.patch
        JIRA Issue YARN-4002
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 40b041cce8a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3c18a53
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10982/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10982/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 47s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_77 +1 compile 0m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 23s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 28s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 31s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_77 +1 javac 0m 24s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_95 +1 javac 0m 27s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 15s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 77m 19s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77. -1 unit 55m 45s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 149m 57s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.webapp.TestRMWithCSRFFilter   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.webapp.TestRMWithCSRFFilter   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797464/YARN-4002-rwlock-v3.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 40b041cce8a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3c18a53 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/10982/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10982/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10982/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rohith Sharma K S, is there any update for the latest patch? Cleaning up 2.8.0 tickets now.

        Thanks,

        Show
        leftnoteasy Wangda Tan added a comment - Rohith Sharma K S , is there any update for the latest patch? Cleaning up 2.8.0 tickets now. Thanks,
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Rebased v3 patch trigger jenkin.
        Wangda Tan can you look into patch please.

        Show
        rohithsharma Rohith Sharma K S added a comment - Rebased v3 patch trigger jenkin. Wangda Tan can you look into patch please.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 22s trunk passed
        +1 compile 0m 37s trunk passed with JDK v1.8.0_91
        +1 compile 0m 36s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 42s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 1m 23s trunk passed
        +1 javadoc 0m 30s trunk passed with JDK v1.8.0_91
        +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 37s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.8.0_91
        +1 javac 0m 37s the patch passed
        +1 compile 0m 33s the patch passed with JDK v1.7.0_95
        +1 javac 0m 33s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 0m 40s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 30s the patch passed
        +1 javadoc 0m 28s the patch passed with JDK v1.8.0_91
        +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95
        -1 unit 35m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
        -1 unit 35m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        91m 22s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
        JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.TestContainerResourceUsage



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803569/YARN-4002-rwlock-v3-rebase.patch
        JIRA Issue YARN-4002
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 255f5ef244f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d464f4d
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11427/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11427/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 22s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_91 +1 compile 0m 36s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 42s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 23s trunk passed +1 javadoc 0m 30s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 37s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_91 +1 javac 0m 37s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_95 +1 javac 0m 33s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 40s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 30s the patch passed +1 javadoc 0m 28s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95 -1 unit 35m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 35m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 91m 22s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803569/YARN-4002-rwlock-v3-rebase.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 255f5ef244f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d464f4d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11427/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11427/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11427/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Rohith Sharma K S,

        Few suggestions:
        1) hostsWrite/ReadLock -> hostsListRead/WriteLock

        2) Return statement should included by try {...}, like:

        387	    return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList
        388	        .contains(ip))
        389	        && !(excludeList.contains(hostName) || excludeList.contains(ip));
        

        Otherwise inconsistency will happen.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Rohith Sharma K S , Few suggestions: 1) hostsWrite/ReadLock -> hostsListRead/WriteLock 2) Return statement should included by try {... }, like: 387 return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList 388 .contains(ip)) 389 && !(excludeList.contains(hostName) || excludeList.contains(ip)); Otherwise inconsistency will happen.
        Hide
        jianhe Jian He added a comment -

        Rohith Sharma K S, the patch may not work because
        hostsReader.refresh() tries to acquire the synchronized lock of HostsFileReader
        And getHosts / getExcludedHosts also tries to acquire the synchronized lock.
        So it is no difference than before.

        Show
        jianhe Jian He added a comment - Rohith Sharma K S , the patch may not work because hostsReader.refresh() tries to acquire the synchronized lock of HostsFileReader And getHosts / getExcludedHosts also tries to acquire the synchronized lock. So it is no difference than before.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        For the second suggestion, IIRC I was suggested to original patch provider and his reply was earlier-comment. His answer made sense to me. If it is not OK , I will change the patch.

        Show
        rohithsharma Rohith Sharma K S added a comment - For the second suggestion, IIRC I was suggested to original patch provider and his reply was earlier-comment . His answer made sense to me. If it is not OK , I will change the patch.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        I agree this wont solve 100%, but it reduces lock time at one layer down.
        Do you mean HostsFileReader class should changed to read/write lock?

        Show
        rohithsharma Rohith Sharma K S added a comment - I agree this wont solve 100%, but it reduces lock time at one layer down. Do you mean HostsFileReader class should changed to read/write lock?
        Hide
        jianhe Jian He added a comment -

        Yeah, I think so. because otherwise node heartbeat still competes on the HostsFileReader synchronized lock.

        Show
        jianhe Jian He added a comment - Yeah, I think so. because otherwise node heartbeat still competes on the HostsFileReader synchronized lock.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Updated the patch addressing comments from Wangda and Jian He.

        1. Renamed variables to hostsListRead/WriteLock
        2. Added read/write lock in HostsFileReader

        ReadLock for the methods

        1. In class HostFileReader : getHosts() and getExcludedHosts
        2. In class NodesListManager : isValidNode and isUntrackedNode

        WriteLock for the methods

        1. In class HostFileReader :
          1. refresh()
          2. refresh(InputStream inFileInputStream, InputStream exFileInputStream)
          3. setIncludesFile()
          4. setExcludesFile
          5. updateFileNames
        2. In class NodesListManager : refreshHostsReader
        Show
        rohithsharma Rohith Sharma K S added a comment - Updated the patch addressing comments from Wangda and Jian He. Renamed variables to hostsListRead/WriteLock Added read/write lock in HostsFileReader ReadLock for the methods In class HostFileReader : getHosts() and getExcludedHosts In class NodesListManager : isValidNode and isUntrackedNode WriteLock for the methods In class HostFileReader : refresh() refresh(InputStream inFileInputStream, InputStream exFileInputStream) setIncludesFile() setExcludesFile updateFileNames In class NodesListManager : refreshHostsReader
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 12m 42s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 53s Maven dependency ordering for branch
        +1 mvninstall 6m 37s trunk passed
        +1 compile 5m 57s trunk passed with JDK v1.8.0_91
        +1 compile 6m 50s trunk passed with JDK v1.7.0_101
        +1 checkstyle 1m 27s trunk passed
        +1 mvnsite 1m 36s trunk passed
        +1 mvneclipse 0m 30s trunk passed
        +1 findbugs 2m 42s trunk passed
        +1 javadoc 1m 16s trunk passed with JDK v1.8.0_91
        +1 javadoc 1m 35s trunk passed with JDK v1.7.0_101
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 14s the patch passed
        +1 compile 5m 53s the patch passed with JDK v1.8.0_91
        +1 javac 5m 53s the patch passed
        +1 compile 6m 53s the patch passed with JDK v1.7.0_101
        +1 javac 6m 53s the patch passed
        -1 checkstyle 1m 24s root: patch generated 2 new + 16 unchanged - 2 fixed = 18 total (was 18)
        +1 mvnsite 1m 32s the patch passed
        +1 mvneclipse 0m 30s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 12s the patch passed
        +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 31s the patch passed with JDK v1.7.0_101
        +1 unit 7m 3s hadoop-common in the patch passed with JDK v1.8.0_91.
        -1 unit 29m 47s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
        +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.7.0_101.
        -1 unit 31m 13s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        142m 59s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
          hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804119/YARN-4002-rwlock-v4.patch
        JIRA Issue YARN-4002
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7f4e526ea34d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ad94411
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11476/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11476/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 42s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 6m 37s trunk passed +1 compile 5m 57s trunk passed with JDK v1.8.0_91 +1 compile 6m 50s trunk passed with JDK v1.7.0_101 +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 2m 42s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 35s trunk passed with JDK v1.7.0_101 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 5m 53s the patch passed with JDK v1.8.0_91 +1 javac 5m 53s the patch passed +1 compile 6m 53s the patch passed with JDK v1.7.0_101 +1 javac 6m 53s the patch passed -1 checkstyle 1m 24s root: patch generated 2 new + 16 unchanged - 2 fixed = 18 total (was 18) +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 12s the patch passed +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 31s the patch passed with JDK v1.7.0_101 +1 unit 7m 3s hadoop-common in the patch passed with JDK v1.8.0_91. -1 unit 29m 47s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.7.0_101. -1 unit 31m 13s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 142m 59s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804119/YARN-4002-rwlock-v4.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7f4e526ea34d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ad94411 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11476/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11476/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11476/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        Rohith Sharma K S, thanks for updating the patch.
        I think we may eliminate the read/write lock in NodeListManager. we can move the NodeListManager#refreshHostsReader into the HostsFileReader itself - refreshHostsReader can take includeFiles and excludeFiles as parameters
        For the getHosts and getExcludedHosts, we can pass in two empty sets together as parameters.

        Show
        jianhe Jian He added a comment - Rohith Sharma K S , thanks for updating the patch. I think we may eliminate the read/write lock in NodeListManager. we can move the NodeListManager#refreshHostsReader into the HostsFileReader itself - refreshHostsReader can take includeFiles and excludeFiles as parameters For the getHosts and getExcludedHosts, we can pass in two empty sets together as parameters.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Update the patch with following changes
        1. Added new method refresh(String includesFile, String excludesFile) to refresh include/exclude lists
        2. Added a new method getHostDetails that return include/exclude lists
        3. Removed synchronization from NodesListManager.

        Show
        rohithsharma Rohith Sharma K S added a comment - Update the patch with following changes 1. Added new method refresh(String includesFile, String excludesFile) to refresh include/exclude lists 2. Added a new method getHostDetails that return include/exclude lists 3. Removed synchronization from NodesListManager.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 7m 34s trunk passed
        +1 compile 7m 42s trunk passed
        +1 checkstyle 1m 31s trunk passed
        +1 mvnsite 1m 48s trunk passed
        +1 mvneclipse 0m 30s trunk passed
        +1 findbugs 2m 39s trunk passed
        +1 javadoc 1m 37s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 28s the patch passed
        +1 compile 8m 14s the patch passed
        +1 javac 8m 14s the patch passed
        -1 checkstyle 1m 39s root: patch generated 11 new + 26 unchanged - 2 fixed = 37 total (was 28)
        +1 mvnsite 1m 45s the patch passed
        +1 mvneclipse 0m 31s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        -1 findbugs 1m 41s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 1m 16s the patch passed
        +1 unit 7m 38s hadoop-common in the patch passed.
        -1 unit 35m 52s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        86m 27s



        Reason Tests
        FindBugs module:hadoop-common-project/hadoop-common
          org.apache.hadoop.util.HostsFileReader.refresh(String, String) does not release lock on all exception paths At HostsFileReader.java:lock on all exception paths At HostsFileReader.java:[line 127]
        Failed junit tests hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804368/YARN-4002-rwlock-v5.patch
        JIRA Issue YARN-4002
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 17c4e2e271df 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9fe5828
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/diff-checkstyle-root.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11494/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11494/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 7m 34s trunk passed +1 compile 7m 42s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 1m 48s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 2m 39s trunk passed +1 javadoc 1m 37s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 8m 14s the patch passed +1 javac 8m 14s the patch passed -1 checkstyle 1m 39s root: patch generated 11 new + 26 unchanged - 2 fixed = 37 total (was 28) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 41s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 16s the patch passed +1 unit 7m 38s hadoop-common in the patch passed. -1 unit 35m 52s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 86m 27s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   org.apache.hadoop.util.HostsFileReader.refresh(String, String) does not release lock on all exception paths At HostsFileReader.java:lock on all exception paths At HostsFileReader.java: [line 127] Failed junit tests hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804368/YARN-4002-rwlock-v5.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 17c4e2e271df 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9fe5828 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11494/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11494/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11494/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        Rohith Sharma K S, could you fix the findbugs warning ?
        also, setExcludesFile, setIncludesFile, updateFileNames need not acquire the writelock or can they just be removed ?

        Show
        jianhe Jian He added a comment - Rohith Sharma K S , could you fix the findbugs warning ? also, setExcludesFile, setIncludesFile, updateFileNames need not acquire the writelock or can they just be removed ?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        also, setExcludesFile, setIncludesFile, updateFileNames need not acquire the writelock or can they just be removed ?

        I see class is annotated with limited private which are with in the hadoop project only. But still I feel since it util method in hadoop-common may be any other projects might be used. Should it still get removed?

        Show
        rohithsharma Rohith Sharma K S added a comment - also, setExcludesFile, setIncludesFile, updateFileNames need not acquire the writelock or can they just be removed ? I see class is annotated with limited private which are with in the hadoop project only. But still I feel since it util method in hadoop-common may be any other projects might be used. Should it still get removed?
        Hide
        jianhe Jian He added a comment -

        ok, let's keep it, I think the lock can be removed ?

        Show
        jianhe Jian He added a comment - ok, let's keep it, I think the lock can be removed ?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Updating the patch with below changes from previous patch

        1. fixed findbugs and checkstyle warning
        2. for the setter methods, removed writelock
        Show
        rohithsharma Rohith Sharma K S added a comment - Updating the patch with below changes from previous patch fixed findbugs and checkstyle warning for the setter methods, removed writelock
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 6m 54s trunk passed
        +1 compile 8m 3s trunk passed
        +1 checkstyle 1m 35s trunk passed
        +1 mvnsite 1m 52s trunk passed
        +1 mvneclipse 0m 32s trunk passed
        +1 findbugs 2m 37s trunk passed
        +1 javadoc 1m 33s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 26s the patch passed
        +1 compile 7m 56s the patch passed
        +1 javac 7m 56s the patch passed
        -1 checkstyle 1m 33s root: patch generated 2 new + 26 unchanged - 2 fixed = 28 total (was 28)
        +1 mvnsite 1m 42s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 49s the patch passed
        +1 javadoc 1m 28s the patch passed
        +1 unit 8m 30s hadoop-common in the patch passed.
        -1 unit 39m 57s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        91m 4s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMAdminService
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804919/YARN-4002-rwlock-v6.patch
        JIRA Issue YARN-4002
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e2aa0bfbb81c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 141873c
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11555/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11555/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 54s trunk passed +1 compile 8m 3s trunk passed +1 checkstyle 1m 35s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 32s trunk passed +1 findbugs 2m 37s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 7m 56s the patch passed +1 javac 7m 56s the patch passed -1 checkstyle 1m 33s root: patch generated 2 new + 26 unchanged - 2 fixed = 28 total (was 28) +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 49s the patch passed +1 javadoc 1m 28s the patch passed +1 unit 8m 30s hadoop-common in the patch passed. -1 unit 39m 57s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 91m 4s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804919/YARN-4002-rwlock-v6.patch JIRA Issue YARN-4002 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e2aa0bfbb81c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 141873c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11555/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11555/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        Committed to trunk, branch-2, branch-2.8
        Thanks Rohith, Zhiguo !
        Thanks Wangda for reviewing the patch

        Show
        jianhe Jian He added a comment - Committed to trunk, branch-2, branch-2.8 Thanks Rohith, Zhiguo ! Thanks Wangda for reviewing the patch
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9825 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9825/)
        YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. (jianhe: rev feb90ffcca536e7deac50976b8a8774450fe089f)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9825 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9825/ ) YARN-4002 . Make ResourceTrackerService#nodeHeartbeat more concurrent. (jianhe: rev feb90ffcca536e7deac50976b8a8774450fe089f) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
        Hide
        jy03843796@gmail.com Guoyao Xu added a comment - - edited

        I think that might not be a big problem. Because hostsReader.refresh() operation does not happen during each heartbeat, and actually
        this operation is not related to isValid() operation. And the frequency of refresh() operation is much less than isValid() operation (nodes number times of call for each heartbeat interval--- default 1 second).
        Then then the synchronized lock of refresh() may not be a problem of isValid(), and it might
        not be harm to the concurrency of heartbeat call.

        Show
        jy03843796@gmail.com Guoyao Xu added a comment - - edited I think that might not be a big problem. Because hostsReader.refresh() operation does not happen during each heartbeat, and actually this operation is not related to isValid() operation. And the frequency of refresh() operation is much less than isValid() operation (nodes number times of call for each heartbeat interval--- default 1 second). Then then the synchronized lock of refresh() may not be a problem of isValid(), and it might not be harm to the concurrency of heartbeat call.

          People

          • Assignee:
            zhiguohong Hong Zhiguo
            Reporter:
            zhiguohong Hong Zhiguo
          • Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development