Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6354

ShuffleHandler should be able to log shuffle connections

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way

      1. MAPREDUCE-6354.patch
        2 kB
        Chang Li
      2. MAPREDUCE-6354.2.patch
        2 kB
        Chang Li
      3. MAPREDUCE-6354.3.patch
        2 kB
        Chang Li
      4. MAPREDUCE-6354.4.patch
        3 kB
        Chang Li
      5. MAPREDUCE-6354.5.patch
        3 kB
        Chang Li
      6. MAPREDUCE-6354.6.patch
        3 kB
        Chang Li
      7. MAPREDUCE-6354.7.patch
        3 kB
        Chang Li
      8. MAPREDUCE-6354.8.patch
        5 kB
        Chang Li

        Activity

        Hide
        lichangleo Chang Li added a comment -

        Jason Lowe, could you please help review this patch? Thanks

        Show
        lichangleo Chang Li added a comment - Jason Lowe , could you please help review this patch? Thanks
        Hide
        aw Allen Wittenauer added a comment -

        I don't think this is a good idea. This is going to result in MASSIVE log increases with large MR jobs.

        Show
        aw Allen Wittenauer added a comment - I don't think this is a good idea. This is going to result in MASSIVE log increases with large MR jobs.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 48s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 34s There were no new javac warning messages.
        +1 javadoc 9m 44s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 21s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 0m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.
            35m 57s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730212/MAPREDUCE-6354.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 8f65c79
        hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5636/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5636/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5636/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 48s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 21s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.     35m 57s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730212/MAPREDUCE-6354.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f65c79 hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5636/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5636/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5636/console This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        I don't think this is a good idea. This is going to result in MASSIVE log increases with large MR jobs.

        It will create a lot of logging for a large job, but we're already logging a not-so-useful message per shuffle connection after MAPREDUCE-5787. And having a useful message per shuffle connection is very useful for tracking down abusive jobs whose shuffle phase causes NM file descriptors, network or disk to go haywire.

        Allen, would it mitigate your concerns if this were logged with a separately configurable logger, e.g.: ShuffleHandlerAuditLogger? That way users could configure it on when they want to audit shuffle transfers or off when they don't.

        Other comments on the patch: please don't add newlines to the output, it just makes the logs visibly long. I'd prefer a brief one-line message per connection, and IMHO it's redundant to label a job ID with "job:" since the jobID has "job_" in it already.

        Show
        jlowe Jason Lowe added a comment - I don't think this is a good idea. This is going to result in MASSIVE log increases with large MR jobs. It will create a lot of logging for a large job, but we're already logging a not-so-useful message per shuffle connection after MAPREDUCE-5787 . And having a useful message per shuffle connection is very useful for tracking down abusive jobs whose shuffle phase causes NM file descriptors, network or disk to go haywire. Allen, would it mitigate your concerns if this were logged with a separately configurable logger, e.g.: ShuffleHandlerAuditLogger? That way users could configure it on when they want to audit shuffle transfers or off when they don't. Other comments on the patch: please don't add newlines to the output, it just makes the logs visibly long. I'd prefer a brief one-line message per connection, and IMHO it's redundant to label a job ID with "job:" since the jobID has "job_" in it already.
        Hide
        lichangleo Chang Li added a comment -

        Jason Lowe Thanks for review! Have updated my patch. Have tested the logging and the connection info is logged as following

        shuffle for job_1430770193094_0002 reducer 0, maps=[attempt_1430770193094_0002_m_000000_0, attempt_1430770193094_0002_m_000001_0]
        
        Show
        lichangleo Chang Li added a comment - Jason Lowe Thanks for review! Have updated my patch. Have tested the logging and the connection info is logged as following shuffle for job_1430770193094_0002 reducer 0, maps=[attempt_1430770193094_0002_m_000000_0, attempt_1430770193094_0002_m_000001_0]
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 0s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 43s There were no new javac warning messages.
        +1 javadoc 9m 49s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 20s There were no new checkstyle issues.
        -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 0m 40s The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.
            36m 26s  



        Reason Tests
        FindBugs module:hadoop-mapreduce-client-shuffle
          Nullcheck of jobQ at line 746 of value previously dereferenced in org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext, MessageEvent) At ShuffleHandler.java:746 of value previously dereferenced in org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext, MessageEvent) At ShuffleHandler.java:[line 746]
          Nullcheck of reduceQ at line 746 of value previously dereferenced in org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext, MessageEvent) At ShuffleHandler.java:746 of value previously dereferenced in org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext, MessageEvent) At ShuffleHandler.java:[line 746]



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730274/MAPREDUCE-6354.2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / bf70c5a
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
        hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 49s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 20s There were no new checkstyle issues. -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 0m 40s The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.     36m 26s   Reason Tests FindBugs module:hadoop-mapreduce-client-shuffle   Nullcheck of jobQ at line 746 of value previously dereferenced in org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext, MessageEvent) At ShuffleHandler.java:746 of value previously dereferenced in org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext, MessageEvent) At ShuffleHandler.java: [line 746]   Nullcheck of reduceQ at line 746 of value previously dereferenced in org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext, MessageEvent) At ShuffleHandler.java:746 of value previously dereferenced in org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext, MessageEvent) At ShuffleHandler.java: [line 746] Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730274/MAPREDUCE-6354.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bf70c5a whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 35s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 28s There were no new javac warning messages.
        +1 javadoc 9m 33s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 18s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 0m 36s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.
            35m 21s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730294/MAPREDUCE-6354.3.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 71f4de2
        hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5640/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5640/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5640/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 18s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 36s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.     35m 21s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730294/MAPREDUCE-6354.3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71f4de2 hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5640/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5640/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5640/console This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        I have the same concerns as Allen. We should also improve existing logs to avoid logging for every connection.

        Allen, would it mitigate your concerns if this were logged with a separately configurable logger, e.g.: ShuffleHandlerAuditLogger? That way users could configure it on when they want to audit shuffle transfers or off when they don't.

        +1. Sounds like a good compromise. One could look at the shuffle connections if they want to and roll up as aggressively as desired.

        Show
        kasha Karthik Kambatla added a comment - I have the same concerns as Allen. We should also improve existing logs to avoid logging for every connection. Allen, would it mitigate your concerns if this were logged with a separately configurable logger, e.g.: ShuffleHandlerAuditLogger? That way users could configure it on when they want to audit shuffle transfers or off when they don't. +1. Sounds like a good compromise. One could look at the shuffle connections if they want to and roll up as aggressively as desired.
        Hide
        jlowe Jason Lowe added a comment -

        OK, let's go with the approach of adding an audit log for the ShuffleHandler. To further mitigate concerns, let's set this up as a disabled log (e.g.: only emits at the WARN level by default but users can modify log properties to enable INFO logging and redirect to another file if desired).

        Show
        jlowe Jason Lowe added a comment - OK, let's go with the approach of adding an audit log for the ShuffleHandler. To further mitigate concerns, let's set this up as a disabled log (e.g.: only emits at the WARN level by default but users can modify log properties to enable INFO logging and redirect to another file if desired).
        Hide
        lichangleo Chang Li added a comment -

        Thanks Jason Lowe for suggestions! have updated my patch, use a separate logger to log connection info in shuffle handler, and that logger is disabled by default.

        Show
        lichangleo Chang Li added a comment - Thanks Jason Lowe for suggestions! have updated my patch, use a separate logger to log connection info in shuffle handler, and that logger is disabled by default.
        Hide
        lichangleo Chang Li added a comment -

        added some more comments on .5 patch

        Show
        lichangleo Chang Li added a comment - added some more comments on .5 patch
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 29s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 26s There were no new javac warning messages.
        +1 javadoc 9m 32s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 26s The applied patch generated 3 new checkstyle issues (total was 60, now 63).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 2m 16s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 common tests 23m 16s Tests passed in hadoop-common.
        +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.
            61m 18s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730963/MAPREDUCE-6354.4.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / b725078
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 29s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 26s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 26s The applied patch generated 3 new checkstyle issues (total was 60, now 63). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 16s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 16s Tests passed in hadoop-common. +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.     61m 18s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730963/MAPREDUCE-6354.4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b725078 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/artifact/patchprocess/testrun_hadoop-common.txt hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5652/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 42s There were no new javac warning messages.
        +1 javadoc 9m 55s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 25s The applied patch generated 4 new checkstyle issues (total was 61, now 65).
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
        +1 findbugs 2m 17s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 common tests 23m 13s Tests passed in hadoop-common.
        +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.
            62m 4s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730967/MAPREDUCE-6354.5.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / b725078
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 25s The applied patch generated 4 new checkstyle issues (total was 61, now 65). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 2m 17s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 13s Tests passed in hadoop-common. +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.     62m 4s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730967/MAPREDUCE-6354.5.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b725078 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/artifact/patchprocess/testrun_hadoop-common.txt hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5653/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 34s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 28s There were no new javac warning messages.
        +1 javadoc 9m 33s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 27s There were no new checkstyle issues.
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 2m 17s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 common tests 24m 27s Tests passed in hadoop-common.
        +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.
            62m 38s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731183/MAPREDUCE-6354.6.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 8e991f4
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 27s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 17s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 24m 27s Tests passed in hadoop-common. +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.     62m 38s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731183/MAPREDUCE-6354.6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8e991f4 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/artifact/patchprocess/testrun_hadoop-common.txt hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/console This message was automatically generated.
        Hide
        lichangleo Chang Li added a comment -

        That whitespace issue is not caused by my change. Jason Lowe could you help review my patch

        Show
        lichangleo Chang Li added a comment - That whitespace issue is not caused by my change. Jason Lowe could you help review my patch
        Hide
        lichangleo Chang Li added a comment -
        Show
        lichangleo Chang Li added a comment - Karthik Kambatla , Allen Wittenauer
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for updating the patch, Chang. Sorry for taking so long to respond.

        Now that I thought about it further there's still an issue where existing users who have custom log4j.properties and do not update to setup the new ShuffleHandler.audit logger will get shuffle connections logged by default. Therefore it may make more sense to log connections at the debug or trace level rather than info, so users will have to go out of their way to enable them if they want to see them. When making this change we should still update log4j.properties in the source, but instead of setting anything in there we should just put a commented-out directive to set the ShuffleHandler.audit logger to debug/trace with a comment above saying it can be uncommented to enable logging of shuffle connections.

        Also in the interest of making the logging message more efficient, we probably only need to log the job ID and the reducer number. The main purpose of logging this information is to track which reducers from which jobs are connecting to the NM at a particular time to help narrow down jobs that are spamming NMs. The list of map IDs is probably not that useful in that context. If we really want to know that arguably we should also be logging the data size being sent and when it completes, but that can be too much logging information. Maybe we can log the job and reducer number at the debug level and the map ID and data size at the trace level? The latter should not log when the futures are setup but rather when they are executed, as we don't want to log that we are sending data to a reducer until we are actually sending it. A bit tricky to orchestrate but doable, and we could defer the detailed trace logging to another JIRA if desired.

        Show
        jlowe Jason Lowe added a comment - Thanks for updating the patch, Chang. Sorry for taking so long to respond. Now that I thought about it further there's still an issue where existing users who have custom log4j.properties and do not update to setup the new ShuffleHandler.audit logger will get shuffle connections logged by default. Therefore it may make more sense to log connections at the debug or trace level rather than info, so users will have to go out of their way to enable them if they want to see them. When making this change we should still update log4j.properties in the source, but instead of setting anything in there we should just put a commented-out directive to set the ShuffleHandler.audit logger to debug/trace with a comment above saying it can be uncommented to enable logging of shuffle connections. Also in the interest of making the logging message more efficient, we probably only need to log the job ID and the reducer number. The main purpose of logging this information is to track which reducers from which jobs are connecting to the NM at a particular time to help narrow down jobs that are spamming NMs. The list of map IDs is probably not that useful in that context. If we really want to know that arguably we should also be logging the data size being sent and when it completes, but that can be too much logging information. Maybe we can log the job and reducer number at the debug level and the map ID and data size at the trace level? The latter should not log when the futures are setup but rather when they are executed, as we don't want to log that we are sending data to a reducer until we are actually sending it. A bit tricky to orchestrate but doable, and we could defer the detailed trace logging to another JIRA if desired.
        Hide
        lichangleo Chang Li added a comment -

        Jason Lowe thanks a lot for thoughtful review and hearty discussion of how to make this logging more efficient! I have made changes of debug level logging. As for the trace level logging, I think we could wait till this get committed and file another jira to address that issue. Let me know what you think of the latest patch. Thanks!

        Show
        lichangleo Chang Li added a comment - Jason Lowe thanks a lot for thoughtful review and hearty discussion of how to make this logging more efficient! I have made changes of debug level logging. As for the trace level logging, I think we could wait till this get committed and file another jira to address that issue. Let me know what you think of the latest patch. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 0s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 32s There were no new javac warning messages.
        +1 javadoc 9m 32s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 27s There were no new checkstyle issues.
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 23m 1s Tests passed in hadoop-common.
        +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.
            63m 58s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12737602/MAPREDUCE-6354.7.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / d7e7f6a
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 27s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 1s Tests passed in hadoop-common. +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.     63m 58s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737602/MAPREDUCE-6354.7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d7e7f6a whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/testrun_hadoop-common.txt hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5772/console This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for updating the patch, Chang. The debug log should be wrapped in AUDITLOG.isDebugEnabled() so we don't perform the string processing if we aren't going to log it.

        Also, even though this wasn't technically part of this fix, please go ahead and fixup the trailing whitespace in log4j.properties that checkstyle is whining about. Otherwise it will continue to whine for any other patch that happens to touch nearby lines.

        Show
        jlowe Jason Lowe added a comment - Thanks for updating the patch, Chang. The debug log should be wrapped in AUDITLOG.isDebugEnabled() so we don't perform the string processing if we aren't going to log it. Also, even though this wasn't technically part of this fix, please go ahead and fixup the trailing whitespace in log4j.properties that checkstyle is whining about. Otherwise it will continue to whine for any other patch that happens to touch nearby lines.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 58s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 34s There were no new javac warning messages.
        +1 javadoc 9m 34s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 26s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 23m 2s Tests passed in hadoop-common.
        +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.
            64m 2s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12737665/MAPREDUCE-6354.8.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 9f6d678
        hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 58s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 26s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 2s Tests passed in hadoop-common. +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.     64m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737665/MAPREDUCE-6354.8.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9f6d678 hadoop-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/artifact/patchprocess/testrun_hadoop-common.txt hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5774/console This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        +1 latest patch lgtm. Will commit this tomorrow if there are no objections.

        Show
        jlowe Jason Lowe added a comment - +1 latest patch lgtm. Will commit this tomorrow if there are no objections.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Chang! I committed this to trunk and branch-2.

        Show
        jlowe Jason Lowe added a comment - Thanks, Chang! I committed this to trunk and branch-2.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #7981 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7981/)
        MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7981 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7981/ ) MAPREDUCE-6354 . ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53) hadoop-mapreduce-project/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/log4j.properties hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/220/)
        MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/220/ ) MAPREDUCE-6354 . ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-mapreduce-project/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #950 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/950/)
        MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #950 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/950/ ) MAPREDUCE-6354 . ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2148/)
        MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2148/ ) MAPREDUCE-6354 . ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-mapreduce-project/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/209/)
        MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/209/ ) MAPREDUCE-6354 . ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-mapreduce-project/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #218 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/218/)
        MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #218 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/218/ ) MAPREDUCE-6354 . ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53) hadoop-mapreduce-project/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/log4j.properties hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2166 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2166/)
        MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2166 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2166/ ) MAPREDUCE-6354 . ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53) hadoop-mapreduce-project/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/log4j.properties hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

          People

          • Assignee:
            lichangleo Chang Li
            Reporter:
            lichangleo Chang Li
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development