Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5932

Provide an option to use a dedicated reduce-side shuffle log

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.7.0
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      For reducers in large jobs our users cannot easily spot portions of the log associated with problems with their code. An example reducer with INFO-level logging generates ~3500 lines / ~700KiB lines per second. 95% of the log is the client-side of the shuffle org.apache.hadoop.mapreduce.task.reduce.*

      $ wc syslog 
          3642   48192  691013 syslog
      $ grep task.reduce syslog | wc 
          3424   46534  659038
      $ grep task.reduce.ShuffleScheduler syslog | wc 
          1521   17745  251458
      $ grep task.reduce.Fetcher syslog | wc 
          1045   15340  223683
      $ grep task.reduce.InMemoryMapOutput syslog | wc 
           400    4800   72060
      $ grep task.reduce.MergeManagerImpl syslog | wc 
           432    8200  106555
      

      Byte percentage breakdown:

      Shuffle total:           95%
      
      ShuffleScheduler:        36%
      Fetcher:                 32%
      InMemoryMapOutput:       10%
      MergeManagerImpl:        15%
      

      While this is information is actually often useful for devops debugging shuffle performance issues, the job users are often lost.

      We propose to have a dedicated syslog.shuffle file.

      1. MAPREDUCE-5932.v06.patch
        29 kB
        Gera Shegalov
      2. MAPREDUCE-5932.v05.patch
        29 kB
        Gera Shegalov
      3. MAPREDUCE-5932.v04.patch
        28 kB
        Gera Shegalov
      4. MAPREDUCE-5932.v03.patch
        20 kB
        Gera Shegalov
      5. MAPREDUCE-5932.v02.patch
        13 kB
        Gera Shegalov
      6. MAPREDUCE-5932.v01.patch
        12 kB
        Gera Shegalov

        Activity

        Hide
        Gera Shegalov added a comment -

        Thank you for a thorough review, Jason!

        Show
        Gera Shegalov added a comment - Thank you for a thorough review, Jason!
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #6642 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6642/)
        MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov (jlowe: rev 03ab24aa01ffea1cacf1fa9cbbf73c3f2904d981)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLogAppender.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6642 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6642/ ) MAPREDUCE-5932 . Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov (jlowe: rev 03ab24aa01ffea1cacf1fa9cbbf73c3f2904d981) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLogAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java
        Hide
        Jason Lowe added a comment -

        Thanks, Gera! I committed this to trunk and branch-2.

        Show
        Jason Lowe added a comment - Thanks, Gera! I committed this to trunk and branch-2.
        Hide
        Jason Lowe added a comment -

        +1 lgtm. Thanks for working through all the review iterations! Committing this.

        Show
        Jason Lowe added a comment - +1 lgtm. Thanks for working through all the review iterations! Committing this.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12684853/MAPREDUCE-5932.v06.patch
        against trunk revision 7caa3bc.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5058//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5058//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12684853/MAPREDUCE-5932.v06.patch against trunk revision 7caa3bc. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5058//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5058//console This message is automatically generated.
        Hide
        Gera Shegalov added a comment -

        I agree with including Merger, Jason. I did not include it initially because Merger is not specific to reducer, but it makes sense indeed to have all shuffle/merge logging in a single place.

        Show
        Gera Shegalov added a comment - I agree with including Merger, Jason. I did not include it initially because Merger is not specific to reducer, but it makes sense indeed to have all shuffle/merge logging in a single place.
        Hide
        Jason Lowe added a comment -

        Thanks for updating the patch, Gera. Looks good overall. Gave the patch a test drive, and I noticed that some of the logs that occur during shuffle are not being redirected to the shuffle log, specifically org.apache.hadoop.mapred.Merger. MergeManagerImpl is in the shuffle log but the Merger logs are not. IMHO it'd be nice if the Merger logs also appeared in the separate shuffle log since those logs occur during the shuffle phase since we merge while shuffling.

        Show
        Jason Lowe added a comment - Thanks for updating the patch, Gera. Looks good overall. Gave the patch a test drive, and I noticed that some of the logs that occur during shuffle are not being redirected to the shuffle log, specifically org.apache.hadoop.mapred.Merger. MergeManagerImpl is in the shuffle log but the Merger logs are not. IMHO it'd be nice if the Merger logs also appeared in the separate shuffle log since those logs occur during the shuffle phase since we merge while shuffling.
        Hide
        Gera Shegalov added a comment -

        I believe failures are unrelated. I reran the 2 tests on my laptop without any issues. Jason Lowe, can you take a look at this version please?

        Show
        Gera Shegalov added a comment - I believe failures are unrelated. I reran the 2 tests on my laptop without any issues. Jason Lowe , can you take a look at this version please?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12683732/MAPREDUCE-5932.v05.patch
        against trunk revision a655973.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
        org.apache.hadoop.mapred.TestJobCleanup

        The test build failed in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5051//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5051//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12683732/MAPREDUCE-5932.v05.patch against trunk revision a655973. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapred.TestJobCleanup The test build failed in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5051//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5051//console This message is automatically generated.
        Hide
        Gera Shegalov added a comment -

        There's a lot of ways to do this, including tacking on yet another boolean (which isn't great for readability given there's now one for the AppMaster), passing an enum that can differentiate AM/map/task (not sure one exists to reuse), pass the Task/TaskId object and null means AM, etc.

        Hi Jason Lowe, I opted for passing task as the easiest option.

        Show
        Gera Shegalov added a comment - There's a lot of ways to do this, including tacking on yet another boolean (which isn't great for readability given there's now one for the AppMaster), passing an enum that can differentiate AM/map/task (not sure one exists to reuse), pass the Task/TaskId object and null means AM, etc. Hi Jason Lowe , I opted for passing task as the easiest option.
        Hide
        Jason Lowe added a comment -

        Thanks for the update.

        With the addition of the AM flag to addLog4jProperties we're now inconsistent with how we convey the "task" type. AMs are passed via direct arguments while map/reduce tasks have the caller stuffing the value in the conf immediately before calling and the callee has to dig that out. It'd be simpler (and faster) to just pass the isMapTask info to addLog4jProperties directly since the caller already knows and doesn't need to dig it out of the conf. There's a lot of ways to do this, including tacking on yet another boolean (which isn't great for readability given there's now one for the AppMaster), passing an enum that can differentiate AM/map/task (not sure one exists to reuse), pass the Task/TaskId object and null means AM, etc.

        Show
        Jason Lowe added a comment - Thanks for the update. With the addition of the AM flag to addLog4jProperties we're now inconsistent with how we convey the "task" type. AMs are passed via direct arguments while map/reduce tasks have the caller stuffing the value in the conf immediately before calling and the callee has to dig that out. It'd be simpler (and faster) to just pass the isMapTask info to addLog4jProperties directly since the caller already knows and doesn't need to dig it out of the conf. There's a lot of ways to do this, including tacking on yet another boolean (which isn't great for readability given there's now one for the AppMaster), passing an enum that can differentiate AM/map/task (not sure one exists to reuse), pass the Task/TaskId object and null means AM, etc.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12682585/MAPREDUCE-5932.v04.patch
        against trunk revision a9a0cc3.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5037//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5037//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12682585/MAPREDUCE-5932.v04.patch against trunk revision a9a0cc3. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5037//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5037//console This message is automatically generated.
        Hide
        Gera Shegalov added a comment -

        Thanks for review, Jason Lowe! Test failures were result of logically incomplete refactoring that is hopefully good now in v04. Good catch regarding isMap conf key. mapred-default.xml is updated as well.

        Show
        Gera Shegalov added a comment - Thanks for review, Jason Lowe ! Test failures were result of logically incomplete refactoring that is hopefully good now in v04. Good catch regarding isMap conf key. mapred-default.xml is updated as well.
        Hide
        Jason Lowe added a comment -

        Thanks for updating the patch, Gera. Could you look into the test failures? They seem related. Other comments on the patch:

        There's already a conf to tell if the task is a map or a reduce, see MRJobConfig.TASK_ISMAP. We shouldn't need MRJobConfig.REDUCE_CONTAINER.

        The new properties should be added to mapred-default.xml for documentation purposes.

        Show
        Jason Lowe added a comment - Thanks for updating the patch, Gera. Could you look into the test failures? They seem related. Other comments on the patch: There's already a conf to tell if the task is a map or a reduce, see MRJobConfig.TASK_ISMAP. We shouldn't need MRJobConfig.REDUCE_CONTAINER. The new properties should be added to mapred-default.xml for documentation purposes.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12682342/MAPREDUCE-5932.v03.patch
        against trunk revision 79301e8.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.mapreduce.v2.TestMRJobs
        org.apache.hadoop.mapreduce.v2.TestUberAM
        org.apache.hadoop.yarn.TestContainerLogAppender

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5031//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5031//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5031//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12682342/MAPREDUCE-5932.v03.patch against trunk revision 79301e8. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.yarn.TestContainerLogAppender +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5031//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5031//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5031//console This message is automatically generated.
        Hide
        Gera Shegalov added a comment -

        Thanks for reviewing, Jason Lowe! In v03, I made the shuffle logger configurable independently from the root logger.

        Show
        Gera Shegalov added a comment - Thanks for reviewing, Jason Lowe ! In v03, I made the shuffle logger configurable independently from the root logger.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12651193/MAPREDUCE-5932.v02.patch
        against trunk revision 86ff28d.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The test build failed in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4990//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4990//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651193/MAPREDUCE-5932.v02.patch against trunk revision 86ff28d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The test build failed in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4990//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4990//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        Thanks for the patch, Gera! I think it's a useful feature. Apologies for the late review.

        One comment on the patch: it looks like the shuffle log will always use a rolling log appender, but syslog will sometimes be a standard log appender. I think the appender behavior should be consistent – if we ask for syslog to use a rolling appender then the shuffle log should roll as well, and if syslog is not rolling then the shuffle log shouldn't either.

        Show
        Jason Lowe added a comment - Thanks for the patch, Gera! I think it's a useful feature. Apologies for the late review. One comment on the patch: it looks like the shuffle log will always use a rolling log appender, but syslog will sometimes be a standard log appender. I think the appender behavior should be consistent – if we ask for syslog to use a rolling appender then the shuffle log should roll as well, and if syslog is not rolling then the shuffle log shouldn't either.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12651193/MAPREDUCE-5932.v02.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.mapred.pipes.TestPipeApplication

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4671//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4671//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651193/MAPREDUCE-5932.v02.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4671//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4671//console This message is automatically generated.
        Hide
        Gera Shegalov added a comment -

        Fixing org.apache.hadoop.mapreduce.v2.app.job.impl.TestMapReduceChildJVM

        Show
        Gera Shegalov added a comment - Fixing org.apache.hadoop.mapreduce.v2.app.job.impl.TestMapReduceChildJVM
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12651062/MAPREDUCE-5932.v01.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.mapreduce.v2.app.job.impl.TestMapReduceChildJVM

        The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.mapred.pipes.TestPipeApplication

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4670//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4670//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651062/MAPREDUCE-5932.v01.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.mapreduce.v2.app.job.impl.TestMapReduceChildJVM The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4670//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4670//console This message is automatically generated.

          People

          • Assignee:
            Gera Shegalov
            Reporter:
            Gera Shegalov
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development