Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6473

Job submission can take a long time during Cluster initialization

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: performance
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      During initialization in Cluster.java, the framework provider classes are loaded inside a sync block which can considerably increase job submission time when the number of submissions are high. The motive is to reduce time spent in this sync block safely to improve performance.

      synchronized (frameworkLoader) {
            for (ClientProtocolProvider provider : frameworkLoader) {
              LOG.debug("Trying ClientProtocolProvider : "
                  + provider.getClass().getName());
      
      1. MAPREDUCE-6473-v7.patch
        4 kB
        Kuhu Shukla
      2. MAPREDUCE-6473-v6.patch
        4 kB
        Kuhu Shukla
      3. MAPREDUCE-6473-v5.patch
        4 kB
        Kuhu Shukla
      4. MAPREDUCE-6473-v4.patch
        3 kB
        Kuhu Shukla
      5. MAPREDUCE-6473-v3.patch
        3 kB
        Kuhu Shukla
      6. MAPREDUCE-6473-v2.patch
        3 kB
        Kuhu Shukla
      7. MAPREDUCE-6473-v1.patch
        2 kB
        Kuhu Shukla
      8. MAPREDUCE-6473-PerfTest.txt
        4 kB
        Kuhu Shukla
      9. avgtime.png
        9 kB
        Kuhu Shukla
      10. 99%ile.png
        9 kB
        Kuhu Shukla

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9098 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9098/)
          MAPREDUCE-6473. Job submission can take a long time during Cluster (jlowe: rev f657b54281d517c69df0a23b97b47befa77d1dc2)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9098 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9098/ ) MAPREDUCE-6473 . Job submission can take a long time during Cluster (jlowe: rev f657b54281d517c69df0a23b97b47befa77d1dc2) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Kuhu for the contribution and to Eric for additional review! I committed this to trunk, branch-2, and branch-2.8.

          Show
          jlowe Jason Lowe added a comment - Thanks to Kuhu for the contribution and to Eric for additional review! I committed this to trunk, branch-2, and branch-2.8.
          Hide
          jlowe Jason Lowe added a comment -

          +1 committing this.

          Show
          jlowe Jason Lowe added a comment - +1 committing this.
          Hide
          eepayne Eric Payne added a comment -

          In the above scenario when thread 2 comes along and finds that the provider list is null , since thread 1 is already in the synchronized block, thread 2 will wait

          Kuhu Shukla, Yes, you are correct. I had missed that. Thanks for the clarification.

          +1 (non-binding)

          Show
          eepayne Eric Payne added a comment - In the above scenario when thread 2 comes along and finds that the provider list is null , since thread 1 is already in the synchronized block, thread 2 will wait Kuhu Shukla , Yes, you are correct. I had missed that. Thanks for the clarification. +1 (non-binding)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 9m 31s trunk passed
          +1 compile 0m 33s trunk passed with JDK v1.8.0_66
          +1 compile 0m 29s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 38s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          -1 findbugs 1m 28s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 31s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 31s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 27s the patch passed with JDK v1.8.0_66
          +1 javac 0m 27s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.7.0_91
          +1 javac 0m 25s the patch passed
          +1 checkstyle 0m 14s the patch passed
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 31s the patch passed
          +1 javadoc 0m 22s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 25s the patch passed with JDK v1.7.0_91
          +1 unit 2m 6s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 26s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          25m 0s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781671/MAPREDUCE-6473-v7.patch
          JIRA Issue MAPREDUCE-6473
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b28701a9c896 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9f4bf3b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6265/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6265/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6265/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 31s trunk passed +1 compile 0m 33s trunk passed with JDK v1.8.0_66 +1 compile 0m 29s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed -1 findbugs 1m 28s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 31s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 31s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 32s the patch passed +1 compile 0m 27s the patch passed with JDK v1.8.0_66 +1 javac 0m 27s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_91 +1 javac 0m 25s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 31s the patch passed +1 javadoc 0m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 25s the patch passed with JDK v1.7.0_91 +1 unit 2m 6s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 2m 26s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 25m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781671/MAPREDUCE-6473-v7.patch JIRA Issue MAPREDUCE-6473 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b28701a9c896 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f4bf3b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6265/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6265/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6265/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Thank you Jason Lowe for your comments. I corrected the checkstyle and the array copy issue.

          Show
          kshukla Kuhu Shukla added a comment - Thank you Jason Lowe for your comments. I corrected the checkstyle and the array copy issue.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch, Kuhu!

          Why are we making a copy of the localProviderList when initializing providerList instead of just doing providerList = localProviderList? I understand why we don't want to use providerList directly when initializing the array since we don't want to expose a partially initialized array to another thread, but I don't see what the copy accomplishes when the local array list is built.

          Please address the checkstyle nits about the brace placement since it deviates from the coding standard. I realize it was that way before, but since we're touching the code we might as well fix it.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch, Kuhu! Why are we making a copy of the localProviderList when initializing providerList instead of just doing providerList = localProviderList ? I understand why we don't want to use providerList directly when initializing the array since we don't want to expose a partially initialized array to another thread, but I don't see what the copy accomplishes when the local array list is built. Please address the checkstyle nits about the brace placement since it deviates from the coding standard. I realize it was that way before, but since we're touching the code we might as well fix it.
          Hide
          kshukla Kuhu Shukla added a comment -

          Thanks Eric Payne for your comments.

          In the above scenario when thread 2 comes along and finds that the provider list is null , since thread 1 is already in the synchronized block, thread 2 will wait and enter it only when thread 1 is done. Once it does get into the block, an additional check of whether the list is null or not helps thread 2 to not do the same work again. Please let me know if this helps or if I am missing anything? Thanks a lot again.

          Show
          kshukla Kuhu Shukla added a comment - Thanks Eric Payne for your comments. In the above scenario when thread 2 comes along and finds that the provider list is null , since thread 1 is already in the synchronized block, thread 2 will wait and enter it only when thread 1 is done. Once it does get into the block, an additional check of whether the list is null or not helps thread 2 to not do the same work again. Please let me know if this helps or if I am missing anything? Thanks a lot again.
          Hide
          eepayne Eric Payne added a comment -

          Kuhu Shukla, Thanks for documenting this problem and providing the patch for it.

          It looks like the following could happen in Cluster.java:

          • thread 1 enters initProviderList and begins to process the entries in frameworkLoader
          • thread 2 enters initProviderList, sees that providerList is null, and exits. Then, in initialize, no processing for providerList would happen and it would throw an IOE when clientProtocolProvider is null. Am I missing something?
          Show
          eepayne Eric Payne added a comment - Kuhu Shukla , Thanks for documenting this problem and providing the patch for it. It looks like the following could happen in Cluster.java : thread 1 enters initProviderList and begins to process the entries in frameworkLoader thread 2 enters initProviderList , sees that providerList is null, and exits. Then, in initialize , no processing for providerList would happen and it would throw an IOE when clientProtocolProvider is null. Am I missing something?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 39s trunk passed
          +1 compile 0m 21s trunk passed with JDK v1.8.0_66
          +1 compile 0m 26s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 12s trunk passed
          +1 mvnsite 0m 32s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          -1 findbugs 1m 11s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 26s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 25s the patch passed
          +1 compile 0m 19s the patch passed with JDK v1.8.0_66
          +1 javac 0m 19s the patch passed
          +1 compile 0m 21s the patch passed with JDK v1.7.0_91
          +1 javac 0m 21s the patch passed
          -1 checkstyle 0m 12s Patch generated 2 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core (total was 15, now 15).
          +1 mvnsite 0m 30s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 16s the patch passed
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91
          +1 unit 1m 54s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 12s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          20m 49s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12762269/MAPREDUCE-6473-v6.patch
          JIRA Issue MAPREDUCE-6473
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f76dafebeef3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8c1adea
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6264/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6264/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6264/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6264/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 39s trunk passed +1 compile 0m 21s trunk passed with JDK v1.8.0_66 +1 compile 0m 26s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 13s trunk passed -1 findbugs 1m 11s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 25s the patch passed +1 compile 0m 19s the patch passed with JDK v1.8.0_66 +1 javac 0m 19s the patch passed +1 compile 0m 21s the patch passed with JDK v1.7.0_91 +1 javac 0m 21s the patch passed -1 checkstyle 0m 12s Patch generated 2 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core (total was 15, now 15). +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91 +1 unit 1m 54s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 2m 12s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 20m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12762269/MAPREDUCE-6473-v6.patch JIRA Issue MAPREDUCE-6473 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f76dafebeef3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8c1adea Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6264/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6264/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6264/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6264/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Jason Lowe, Eric Payne Request for review comments. Thanks a lot.

          Show
          kshukla Kuhu Shukla added a comment - Jason Lowe , Eric Payne Request for review comments. Thanks a lot.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 8m 2s There were no new javac warning messages.
          +1 javadoc 10m 22s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 38s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 48s Tests passed in hadoop-mapreduce-client-core.
              41m 55s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762269/MAPREDUCE-6473-v6.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d1b9b85
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6017/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6017/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6017/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 2s There were no new javac warning messages. +1 javadoc 10m 22s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 38s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 48s Tests passed in hadoop-mapreduce-client-core.     41m 55s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762269/MAPREDUCE-6473-v6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d1b9b85 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6017/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6017/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6017/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 10s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 9m 58s There were no new javac warning messages.
          +1 javadoc 11m 41s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 59s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse.
          +1 findbugs 1m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 2m 0s Tests passed in hadoop-mapreduce-client-core.
              48m 20s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762269/MAPREDUCE-6473-v6.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d1b9b85
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6015/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6015/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6015/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 10s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 9m 58s There were no new javac warning messages. +1 javadoc 11m 41s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 59s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse. +1 findbugs 1m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 2m 0s Tests passed in hadoop-mapreduce-client-core.     48m 20s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762269/MAPREDUCE-6473-v6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d1b9b85 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6015/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6015/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6015/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 1s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 8m 7s There were no new javac warning messages.
          +1 javadoc 10m 11s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 47s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 50s Tests passed in hadoop-mapreduce-client-core.
              41m 55s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762245/MAPREDUCE-6473-v6.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d1b9b85
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6014/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6014/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6014/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 7s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 47s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 50s Tests passed in hadoop-mapreduce-client-core.     41m 55s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762245/MAPREDUCE-6473-v6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d1b9b85 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6014/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6014/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6014/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Got it! Thank you Jason Lowe. Updated patch.

          Show
          kshukla Kuhu Shukla added a comment - Got it! Thank you Jason Lowe . Updated patch.
          Hide
          jlowe Jason Lowe added a comment -

          The problem is this step:

          T1 hasn't finished populating the list but has initialized it

          and the fix is to use a local variable while the list is being initialized then set providerList to the completely built list when we're done.

          Show
          jlowe Jason Lowe added a comment - The problem is this step: T1 hasn't finished populating the list but has initialized it and the fix is to use a local variable while the list is being initialized then set providerList to the completely built list when we're done.
          Hide
          kshukla Kuhu Shukla added a comment -

          Thank you Jason Lowe .
          I thought of making the list volatile. Here is a scenario which I felt could be a problem.

          Thread T1 comes into initialize() , calls initProviderList(), checks that the list is null, which is true the first(few) times. It gets into the sync block, since no one is in there. T1 hasn't finished populating the list but has initialized it (so now the list == null is false) and right then another Thread T2, calls initProviderList() checks that the list is still null, gets false and returns to initialize() , now it starts iterating over the list which is still being written by T1. This case would happen right at the beginning but since I suspected it being ideal for a concurrency exception, I went with another volatile object. If we remove the first (list == null) check before the sync block and have only the one inside it, then every thread has to contend for the sync block even if it is to check whether the list is initialized or not. If the latter case is not a concern then making the list volatile works.

          Show
          kshukla Kuhu Shukla added a comment - Thank you Jason Lowe . I thought of making the list volatile. Here is a scenario which I felt could be a problem. Thread T1 comes into initialize() , calls initProviderList(), checks that the list is null, which is true the first(few) times. It gets into the sync block, since no one is in there. T1 hasn't finished populating the list but has initialized it (so now the list == null is false) and right then another Thread T2, calls initProviderList() checks that the list is still null, gets false and returns to initialize() , now it starts iterating over the list which is still being written by T1. This case would happen right at the beginning but since I suspected it being ideal for a concurrency exception, I went with another volatile object. If we remove the first (list == null) check before the sync block and have only the one inside it, then every thread has to contend for the sync block even if it is to check whether the list is initialized or not. If the latter case is not a concern then making the list volatile works.
          Hide
          jlowe Jason Lowe added a comment -

          Curious, why are we using a separate boolean rather than just using volatile List<...> providerList = null and checking if providerList is null to see if it needs to be initialized? The separate boolean seems unnecessary.

          Show
          jlowe Jason Lowe added a comment - Curious, why are we using a separate boolean rather than just using volatile List<...> providerList = null and checking if providerList is null to see if it needs to be initialized? The separate boolean seems unnecessary.
          Hide
          kshukla Kuhu Shukla added a comment -

          Added a volatile boolean to ensure the provider list is read only after it has been fully populated. Removed the list.isEmpty() check.

          Show
          kshukla Kuhu Shukla added a comment - Added a volatile boolean to ensure the provider list is read only after it has been fully populated. Removed the list.isEmpty() check.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Kuhu!

          There's a race condition in the initialization of the provider list. A thread can try to init while another thread is busy building the provider list. In that case the second thread may try to walk the ArrayList while it is being constructed since providerList.isEmpty() will be false yet the list isn't fully constructed. This could cause a client to fail to find a provider that should have been found and violates the concurrency safety of ArrayList since we're reading it while it's being written. We could use a volatile that's initialized to null when it hasn't been set. That way either the list isn't there and we'll synchronize the threads that are trying to create it within the null check block or the list will be there in its entirety and not modified beyond that.

          Nit: initProviderList is probably a better name since set implies the ability to provide the value to set it to

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Kuhu! There's a race condition in the initialization of the provider list. A thread can try to init while another thread is busy building the provider list. In that case the second thread may try to walk the ArrayList while it is being constructed since providerList.isEmpty() will be false yet the list isn't fully constructed. This could cause a client to fail to find a provider that should have been found and violates the concurrency safety of ArrayList since we're reading it while it's being written. We could use a volatile that's initialized to null when it hasn't been set. That way either the list isn't there and we'll synchronize the threads that are trying to create it within the null check block or the list will be there in its entirety and not modified beyond that. Nit: initProviderList is probably a better name since set implies the ability to provide the value to set it to
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 4s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 8m 1s There were no new javac warning messages.
          +1 javadoc 10m 17s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 46s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 48s Tests passed in hadoop-mapreduce-client-core.
              41m 54s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12757093/MAPREDUCE-6473-v4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 58d1a02
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5999/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5999/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5999/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 4s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 17s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 46s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 48s Tests passed in hadoop-mapreduce-client-core.     41m 54s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12757093/MAPREDUCE-6473-v4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 58d1a02 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5999/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5999/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5999/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 22m 25s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 9m 29s There were no new javac warning messages.
          +1 javadoc 10m 56s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 51s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 41s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 1m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 56s Tests passed in hadoop-mapreduce-client-core.
              50m 6s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12756291/MAPREDUCE-6473-v3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bf2f2b4
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5996/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5996/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5996/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5996/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 22m 25s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 9m 29s There were no new javac warning messages. +1 javadoc 10m 56s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 51s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 1m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 56s Tests passed in hadoop-mapreduce-client-core.     50m 6s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756291/MAPREDUCE-6473-v3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bf2f2b4 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5996/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5996/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5996/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5996/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12756253/MAPREDUCE-6473-v2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bf2f2b4
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5995/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756253/MAPREDUCE-6473-v2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bf2f2b4 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5995/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Corrected checkstyle issues.

          Show
          kshukla Kuhu Shukla added a comment - Corrected checkstyle issues.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 39s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 10m 0s There were no new javac warning messages.
          +1 javadoc 15m 27s There were no new javadoc warning messages.
          +1 release audit 0m 44s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 19s The applied patch generated 23 new checkstyle issues (total was 15, now 38).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 2m 16s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 2m 3s Tests passed in hadoop-mapreduce-client-core.
              54m 51s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755512/MAPREDUCE-6473-v1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6955771
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 10m 0s There were no new javac warning messages. +1 javadoc 15m 27s There were no new javadoc warning messages. +1 release audit 0m 44s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 19s The applied patch generated 23 new checkstyle issues (total was 15, now 38). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 2m 16s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 2m 3s Tests passed in hadoop-mapreduce-client-core.     54m 51s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755512/MAPREDUCE-6473-v1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6955771 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          The two plots show the time spent in millisecconds in cluster initialization for old and new implementation. There is a considerable drop after the change. The test was a simple unit test to record the the time spent doing the cluster initialization per thread. There are a total of 5000 such threads calling cluster initialization() .

          Jason Lowe Request you to please share comments. Thank you!

          Show
          kshukla Kuhu Shukla added a comment - The two plots show the time spent in millisecconds in cluster initialization for old and new implementation. There is a considerable drop after the change. The test was a simple unit test to record the the time spent doing the cluster initialization per thread. There are a total of 5000 such threads calling cluster initialization() . Jason Lowe Request you to please share comments. Thank you!

            People

            • Assignee:
              kshukla Kuhu Shukla
              Reporter:
              kshukla Kuhu Shukla
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development