Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4502

Node-level aggregation with combining the result of maps

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: applicationmaster, mrv2
    • Labels:
      None

      Description

      The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because the scope of combining is limited within only one MapTask. To solve this problem, it's a good way to aggregate the result of maps per node/rack by launch combiner.

      This JIRA is to implement the multi-level aggregation infrastructure, including combining per container(MAPREDUCE-3902 is related), coordinating containers by application master without breaking fault tolerance of jobs.

      1. speculative_draft.pdf
        142 kB
        Tsuyoshi OZAWA
      2. design_v2.pdf
        347 kB
        Tsuyoshi OZAWA
      3. MAPREDUCE-4525-pof.diff
        71 kB
        Tsuyoshi OZAWA
      4. MAPREDUCE-4502.1.patch
        113 kB
        Tsuyoshi OZAWA
      5. MAPREDUCE-4502.2.patch
        115 kB
        Tsuyoshi OZAWA
      6. MAPREDUCE-4502.3.patch
        115 kB
        Tsuyoshi OZAWA
      7. MAPREDUCE-4502.4.patch
        115 kB
        Tsuyoshi OZAWA
      8. MAPREDUCE-4502.5.patch
        115 kB
        Tsuyoshi OZAWA
      9. MAPREDUCE-4502.6.patch
        116 kB
        Tsuyoshi OZAWA
      10. design_v3.pdf
        440 kB
        Tsuyoshi OZAWA
      11. MAPREDUCE-4502.7.patch
        117 kB
        Tsuyoshi OZAWA
      12. MAPREDUCE-4502.8.patch
        117 kB
        Tsuyoshi OZAWA
      13. MAPREDUCE-4502.8.patch
        117 kB
        Tsuyoshi OZAWA
      14. MAPREDUCE-4502.9.patch
        109 kB
        Tsuyoshi OZAWA
      15. MAPREDUCE-4502.9.patch
        109 kB
        Tsuyoshi OZAWA
      16. MAPREDUCE-4502.10.patch
        109 kB
        Tsuyoshi OZAWA

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12592783/MAPREDUCE-4502.10.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4241//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12592783/MAPREDUCE-4502.10.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4241//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12592783/MAPREDUCE-4502.10.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3860//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3860//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12592783/MAPREDUCE-4502.10.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 10 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3860//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3860//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          Sorry, I attached a wrong file. Now I attach a correct one.

          Show
          Tsuyoshi OZAWA added a comment - Sorry, I attached a wrong file. Now I attach a correct one.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12592745/MAPREDUCE-4502.9.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3858//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12592745/MAPREDUCE-4502.9.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 10 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3858//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          Rebased on trunk code.

          Show
          Tsuyoshi OZAWA added a comment - Rebased on trunk code.
          Hide
          Tsuyoshi OZAWA added a comment -

          This test failure seems to be caused by MAPREDUCE-5193. I'll check it after fixing MAPREDUCE-5193.

          Show
          Tsuyoshi OZAWA added a comment - This test failure seems to be caused by MAPREDUCE-5193 . I'll check it after fixing MAPREDUCE-5193 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581124/MAPREDUCE-4502.8.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.lib.input.TestDelegatingInputFormat
          org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
          org.apache.hadoop.mapred.lib.TestDelegatingInputFormat
          org.apache.hadoop.mapred.TestFileInputFormat

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3559//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3559//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581124/MAPREDUCE-4502.8.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 10 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.lib.input.TestDelegatingInputFormat org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat org.apache.hadoop.mapred.lib.TestDelegatingInputFormat org.apache.hadoop.mapred.TestFileInputFormat +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3559//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3559//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581063/MAPREDUCE-4502.8.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.TestFileInputFormat
          org.apache.hadoop.mapreduce.lib.input.TestDelegatingInputFormat
          org.apache.hadoop.mapred.lib.TestDelegatingInputFormat
          org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3556//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3556//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581063/MAPREDUCE-4502.8.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 10 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestFileInputFormat org.apache.hadoop.mapreduce.lib.input.TestDelegatingInputFormat org.apache.hadoop.mapred.lib.TestDelegatingInputFormat org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3556//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3556//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          MAPREDUCE-4502.7.patch deletes a required line wrongly.
          Therefore, the tests failed. This patch fixes it.

          Show
          Tsuyoshi OZAWA added a comment - MAPREDUCE-4502 .7.patch deletes a required line wrongly. Therefore, the tests failed. This patch fixes it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580718/MAPREDUCE-4502.7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle
          org.apache.hadoop.mapreduce.v2.TestMRJobs
          org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
          org.apache.hadoop.mapreduce.security.TestMRCredentials
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapred.TestClusterMRNotification
          org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
          org.apache.hadoop.mapred.TestJobCounters
          org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
          org.apache.hadoop.mapreduce.TestChild
          org.apache.hadoop.mapred.TestMiniMRClientCluster
          org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
          org.apache.hadoop.mapred.TestLazyOutput
          org.apache.hadoop.mapred.TestJobCleanup
          org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
          org.apache.hadoop.mapred.TestMerge
          org.apache.hadoop.mapred.TestClusterMapReduceTestCase
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.mapreduce.v2.TestMRAppWithNodeLevelCombiner
          org.apache.hadoop.mapreduce.TestMRJobClient
          org.apache.hadoop.mapreduce.v2.TestUberAM
          org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
          org.apache.hadoop.mapred.TestJobName

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3549//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3549//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3549//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580718/MAPREDUCE-4502.7.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 10 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapreduce.security.TestMRCredentials org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestMerge org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestMRAppWithNodeLevelCombiner org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapred.TestJobName +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3549//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3549//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3549//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          Updated the patch for the current trunk.

          Show
          Tsuyoshi OZAWA added a comment - Updated the patch for the current trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579117/design_v3.pdf
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3533//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579117/design_v3.pdf against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3533//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          Updated design note including benchmark of node-level aggregation. This revision dedicates only to explain node-level aggregation design and improvement of performance.
          The rack-level aggregation still needs big change, so I'd like to discuss in the other ticket.

          Show
          Tsuyoshi OZAWA added a comment - Updated design note including benchmark of node-level aggregation. This revision dedicates only to explain node-level aggregation design and improvement of performance. The rack-level aggregation still needs big change, so I'd like to discuss in the other ticket.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12570748/MAPREDUCE-4502.6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3358//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3358//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570748/MAPREDUCE-4502.6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3358//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3358//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          Oops, I attached the wrong patch. This is correct one.

          Show
          Tsuyoshi OZAWA added a comment - Oops, I attached the wrong patch. This is correct one.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12570735/MAPREDUCE-4502.5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3357//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3357//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3357//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570735/MAPREDUCE-4502.5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3357//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3357//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3357//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          HADOOP-9112 caused regression, so I retry it based on today's trunk.

          Show
          Tsuyoshi OZAWA added a comment - HADOOP-9112 caused regression, so I retry it based on today's trunk.
          Hide
          Tsuyoshi OZAWA added a comment -

          Thanks Tsz,

          Surenkumar said that this is HADOOP-9112 related problem. I'll attach the same file and check whether it works well.

          Show
          Tsuyoshi OZAWA added a comment - Thanks Tsz, Surenkumar said that this is HADOOP-9112 related problem. I'll attach the same file and check whether it works well.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          In the console, it said "Build step 'Execute shell' marked build as failure".

          Show
          Tsz Wo Nicholas Sze added a comment - In the console , it said "Build step 'Execute shell' marked build as failure".
          Hide
          Tsuyoshi OZAWA added a comment -

          Can someone explain why my patch got "-1 overall" while this patch passed all checks?

          Show
          Tsuyoshi OZAWA added a comment - Can someone explain why my patch got "-1 overall" while this patch passed all checks?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12570292/MAPREDUCE-4502.4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3351//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3351//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570292/MAPREDUCE-4502.4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3351//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3351//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          Fixed to handle exception correctly and add timeout to TestJobConf.

          Show
          Tsuyoshi OZAWA added a comment - Fixed to handle exception correctly and add timeout to TestJobConf.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12570272/MAPREDUCE-4502.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3349//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3349//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3349//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570272/MAPREDUCE-4502.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3349//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3349//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3349//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          Fixed the overlooked findbugs warnings and Added timeout value to test.

          Show
          Tsuyoshi OZAWA added a comment - Fixed the overlooked findbugs warnings and Added timeout value to test.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12570243/MAPREDUCE-4502.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3348//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3348//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3348//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570243/MAPREDUCE-4502.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3348//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3348//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3348//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          Fixed findbugs and release audit warnings.

          Show
          Tsuyoshi OZAWA added a comment - Fixed findbugs and release audit warnings.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12570068/MAPREDUCE-4502.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 16 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570068/MAPREDUCE-4502.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 16 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3344//console This message is automatically generated.
          Hide
          Tsuyoshi OZAWA added a comment -

          Added patch including test.

          Show
          Tsuyoshi OZAWA added a comment - Added patch including test.
          Hide
          Tsuyoshi OZAWA added a comment -

          I've written updated patch sets and attached them to each subtasks.
          Now, I've started to write test case. Any comments or reviews are welcome!

          Show
          Tsuyoshi OZAWA added a comment - I've written updated patch sets and attached them to each subtasks. Now, I've started to write test case. Any comments or reviews are welcome!
          Hide
          Tsuyoshi OZAWA added a comment -

          The forked trunk is available here: https://github.com/oza/hadoop-common/tree/MR-4502

          Show
          Tsuyoshi OZAWA added a comment - The forked trunk is available here: https://github.com/oza/hadoop-common/tree/MR-4502
          Hide
          Tsuyoshi OZAWA added a comment -

          I've spitted this task into some subtasks.
          And about subtask No. 2-5, I've attached patches for each subtasks. I'm appreciated if someone review it.

          Thanks,

          • Tsuyoshi
          Show
          Tsuyoshi OZAWA added a comment - I've spitted this task into some subtasks. And about subtask No. 2-5, I've attached patches for each subtasks. I'm appreciated if someone review it. Thanks, Tsuyoshi
          Hide
          Tsuyoshi OZAWA added a comment -

          I've fixed the bug! Now I start to create patch sets.

          Show
          Tsuyoshi OZAWA added a comment - I've fixed the bug! Now I start to create patch sets.
          Hide
          Tsuyoshi OZAWA added a comment -

          I've found a bug when I benchmark with large scale data. I'll re-attach splited patches after fixing the bug.

          Show
          Tsuyoshi OZAWA added a comment - I've found a bug when I benchmark with large scale data. I'll re-attach splited patches after fixing the bug.
          Hide
          Tsuyoshi OZAWA added a comment -

          I'll create subtasks under MAPREDUCE-4525, and attach patches separately to make the review easy.

          Show
          Tsuyoshi OZAWA added a comment - I'll create subtasks under MAPREDUCE-4525 , and attach patches separately to make the review easy.
          Hide
          Tsuyoshi OZAWA added a comment -

          I've prototyped proof of concept code of node-level aggregation. I made sure that it works well, so I share it! Current situation is as follows:

          • Node level aggregation in distributed mode is supported. I made sure of the correctness of the result by using diff command.
          • Fault tolerance is partially supported. At design level, it can be supported(WIP).
          • No test codes.
          • No switch on/off feature.

          Quick benchmark is as follows:

          Default Hadoop(sec) Node level aggregation(sec)
          105.0 82.0

          Experiment Environment is as follows:

          • Input file is 1GB, generated by RandomTextWriter.
          • 8 CPU/machines.
          • 2 machines.

          I'm appreciate if you review the design and code.

          And, I have several questions to deal with:
          1. How can I write tests such a complex case? Any ideas or references?
          2. I've extended mr_protos.proto. Is it acceptable?

          My next step is creating patch sets with tests and fully fault tolerance support to merge it into trunk. I'm also planning to benchmark of node-level aggregation. I'll show you here when I do it. Note that design note is outdated, so I'm updating it now.

          If you'd like to track changes my code changes, you can see it on my github.

          Show
          Tsuyoshi OZAWA added a comment - I've prototyped proof of concept code of node-level aggregation. I made sure that it works well, so I share it! Current situation is as follows: Node level aggregation in distributed mode is supported. I made sure of the correctness of the result by using diff command. Fault tolerance is partially supported. At design level, it can be supported(WIP). No test codes. No switch on/off feature. Quick benchmark is as follows: Default Hadoop(sec) Node level aggregation(sec) 105.0 82.0 Experiment Environment is as follows: Input file is 1GB, generated by RandomTextWriter. 8 CPU/machines. 2 machines. I'm appreciate if you review the design and code. And, I have several questions to deal with: 1. How can I write tests such a complex case? Any ideas or references? 2. I've extended mr_protos.proto. Is it acceptable? My next step is creating patch sets with tests and fully fault tolerance support to merge it into trunk. I'm also planning to benchmark of node-level aggregation. I'll show you here when I do it. Note that design note is outdated, so I'm updating it now. If you'd like to track changes my code changes, you can see it on my github .
          Hide
          Tsuyoshi OZAWA added a comment -

          I uploaded the slides about this ticket at Pre Strata/Hadoop World Meetup.
          http://www.slideshare.net/ozax86/prestrata-hadoop-word-meetup

          Show
          Tsuyoshi OZAWA added a comment - I uploaded the slides about this ticket at Pre Strata/Hadoop World Meetup. http://www.slideshare.net/ozax86/prestrata-hadoop-word-meetup
          Hide
          Tsuyoshi OZAWA added a comment -

          BTW, I'm starting to implement the aggregation feature with V2 design. Please let me know if you have any opinion about the design note. Thanks.

          Show
          Tsuyoshi OZAWA added a comment - BTW, I'm starting to implement the aggregation feature with V2 design. Please let me know if you have any opinion about the design note. Thanks.
          Hide
          Tsuyoshi OZAWA added a comment -

          I've read the overview of the Sailfish paper.

          1. It makes IO faster to translate the random access of HDDs to sequential access by batching IO at dumping intermediate files with the original file format(i.e. L-File). This is effective only case that very large intermediate file is generated.
          2. Auto-tuning the number of reducers is a good feature for speedup and reducing tuning time, but users cannot control their output.

          1. is one optimization of MR jobs with very large scale intermediate data, which is larger than multi tera bytes. It needs to change HDFS to coordinate concurrent access against intermediate files. I wonder if it is commodity case and can pay the trade-offs between the implementation cost and the effect. Do you have any opinion about it?

          IMHO, 2. is another feature, so we should discuss in another JIRA.

          Show
          Tsuyoshi OZAWA added a comment - I've read the overview of the Sailfish paper. 1. It makes IO faster to translate the random access of HDDs to sequential access by batching IO at dumping intermediate files with the original file format(i.e. L-File). This is effective only case that very large intermediate file is generated. 2. Auto-tuning the number of reducers is a good feature for speedup and reducing tuning time, but users cannot control their output. 1. is one optimization of MR jobs with very large scale intermediate data, which is larger than multi tera bytes. It needs to change HDFS to coordinate concurrent access against intermediate files. I wonder if it is commodity case and can pay the trade-offs between the implementation cost and the effect. Do you have any opinion about it? IMHO, 2. is another feature, so we should discuss in another JIRA.
          Hide
          Chris Douglas added a comment -

          Thanks, Karthik; that was the tech report I meant to link to earlier (off by one error): http://research.yahoo.com/files/yl-2012-002.pdf

          Show
          Chris Douglas added a comment - Thanks, Karthik; that was the tech report I meant to link to earlier (off by one error): http://research.yahoo.com/files/yl-2012-002.pdf
          Hide
          Karthik Kambatla added a comment -

          This paper (to be presented at SoCC '12) talks about intermediate data aggregation. It might make sense to take the inputs from here into consideration as well: http://googlecode.com/files/YL-2012-002.pdf

          Show
          Karthik Kambatla added a comment - This paper (to be presented at SoCC '12) talks about intermediate data aggregation. It might make sense to take the inputs from here into consideration as well: http://googlecode.com/files/YL-2012-002.pdf
          Hide
          Tsuyoshi OZAWA added a comment -

          I redesigned its architecture with the ideas inspired by discussion among Chris and Karthik. Main diffs are as follows:

          1. All aggregations are preformed in containers(at the end of MapTasks). Because of this change, the umbilical protocol is need to change.
          3. No task types are added newly.
          4. Rack-aware aggregation is taken into account.

          Any feedback are welcome.
          Thanks,

          • Tsuyoshi
          Show
          Tsuyoshi OZAWA added a comment - I redesigned its architecture with the ideas inspired by discussion among Chris and Karthik. Main diffs are as follows: 1. All aggregations are preformed in containers(at the end of MapTasks). Because of this change, the umbilical protocol is need to change. 3. No task types are added newly. 4. Rack-aware aggregation is taken into account. Any feedback are welcome. Thanks, Tsuyoshi
          Hide
          Tsuyoshi OZAWA added a comment -

          For the benchmark, we were mostly handling cases without combiners; in our data, each combiner was too effective to benefit from an intermediate level.

          This result is also interesting. It means that the rack-level combiner is useful if its overhead is minimized. To get more detail, I'll read the tech paper.

          BTW, it takes several days to publish new design note. I'll attach it here as soon as possible.

          Show
          Tsuyoshi OZAWA added a comment - For the benchmark, we were mostly handling cases without combiners; in our data, each combiner was too effective to benefit from an intermediate level. This result is also interesting. It means that the rack-level combiner is useful if its overhead is minimized. To get more detail, I'll read the tech paper. BTW, it takes several days to publish new design note. I'll attach it here as soon as possible.
          Hide
          Chris Douglas added a comment -

          This seems to be good approach to deal with rack-level aggregation. Do you have any results about the benchmark?

          For reducing on key ranges, there's a paper in SOCC on Sailfish. I don't have a link to that paper, though there's a tech report. For the benchmark, we were mostly handling cases without combiners; in our data, each combiner was too effective to benefit from an intermediate level.

          Show
          Chris Douglas added a comment - This seems to be good approach to deal with rack-level aggregation. Do you have any results about the benchmark? For reducing on key ranges, there's a paper in SOCC on Sailfish. I don't have a link to that paper, though there's a tech report . For the benchmark, we were mostly handling cases without combiners; in our data, each combiner was too effective to benefit from an intermediate level.
          Hide
          Tsuyoshi OZAWA added a comment -

          s/Do you have some result to/Do you have any results about the benchmark?/

          Show
          Tsuyoshi OZAWA added a comment - s/Do you have some result to/Do you have any results about the benchmark?/
          Hide
          Tsuyoshi OZAWA added a comment -

          Chris and Karthik,

          Thank you for your sharing your experience and thinking. These are very useful for me.

          ShuffleHandler is an auxiliary service loaded in the NodeManager. It's shared across all containers.

          I see. I have to redesign it to run combiner in container.

          Carlo Curino and I experimented with this, but (a) saw only slight improvements in job performance and (b) the changes to the AM to accommodate a new task type were extensive.

          This is very interesting. In fact, I prototyped to run combiner at the end of MapTask as the first version. And, its performance was good. In this case, I found that it's needed to add new status to MapTask because of assuring fault tolerance. Is it acceptable for hadoop to do that?

          With logic to manage skew, we're hoping that scheduling an aggressive range can have a similar effect to combiner tasks, without introducing the new task type.

          This seems to be good approach to deal with rack-level aggregation. Do you have some result to

          1. Perform node-level aggregation (reduce) at the end of maps in co-ordination with AM.

          2. Perform rack-level aggregation at the end of node-level aggregation again in co-ordination with AM. The aggregation could be performed in parallel across the involved nodes such that each node has aggregated values of different keys.

          3. Schedule reducers taking the key-distribution into account across racks.

          Nice wrap-up

          The con will be that the shuffle won't be asynchronous to map computation, but hopefully this wouldn't offset the gains of decreased network and disk I/O.

          The balance between the gains by asynchronous processing and the one by decreasing network and disk I/O. In my previous experiment, it deeply depends on number of reducers. I think these gains are trade-off, so parameters are necessary to deal with various workloads.

          PS. http://dl.acm.org/citation.cfm?id=1901088 documents the advantages of multi-level aggregation in the context of graph algorithms modeled as iterative MR jobs.

          I'm going read it

          It's time for me to create the new revision of design note with reflecting your opinion.

          Thanks,
          – Tsuyoshi

          Show
          Tsuyoshi OZAWA added a comment - Chris and Karthik, Thank you for your sharing your experience and thinking. These are very useful for me. ShuffleHandler is an auxiliary service loaded in the NodeManager. It's shared across all containers. I see. I have to redesign it to run combiner in container. Carlo Curino and I experimented with this, but (a) saw only slight improvements in job performance and (b) the changes to the AM to accommodate a new task type were extensive. This is very interesting. In fact, I prototyped to run combiner at the end of MapTask as the first version. And, its performance was good. In this case, I found that it's needed to add new status to MapTask because of assuring fault tolerance. Is it acceptable for hadoop to do that? With logic to manage skew, we're hoping that scheduling an aggressive range can have a similar effect to combiner tasks, without introducing the new task type. This seems to be good approach to deal with rack-level aggregation. Do you have some result to 1. Perform node-level aggregation (reduce) at the end of maps in co-ordination with AM. 2. Perform rack-level aggregation at the end of node-level aggregation again in co-ordination with AM. The aggregation could be performed in parallel across the involved nodes such that each node has aggregated values of different keys. 3. Schedule reducers taking the key-distribution into account across racks. Nice wrap-up The con will be that the shuffle won't be asynchronous to map computation, but hopefully this wouldn't offset the gains of decreased network and disk I/O. The balance between the gains by asynchronous processing and the one by decreasing network and disk I/O. In my previous experiment, it deeply depends on number of reducers. I think these gains are trade-off, so parameters are necessary to deal with various workloads. PS. http://dl.acm.org/citation.cfm?id=1901088 documents the advantages of multi-level aggregation in the context of graph algorithms modeled as iterative MR jobs. I'm going read it It's time for me to create the new revision of design note with reflecting your opinion. Thanks, – Tsuyoshi
          Hide
          Karthik Kambatla added a comment -

          Hope I am not severely digressing here. Just thinking out loud. Would be great if the more experienced campaigners can weigh-in here.

          Would it make sense to do this either at the end of a map (coordinated by the AM) or, as you suggest, as a ReduceTask rather than a daemon?

          This might make even more sense taking [acmurthy]'s suggestion on MAPREDUCE-199 (data locality hints for reduce tasks) into consideration. Arun's suggestion (rephrased) was to schedule the reducers based on the distribution of keys across map-outputs.

          Putting the pieces together, it might be even more beneficial to do the following:

          1. Perform node-level aggregation (reduce) at the end of maps in co-ordination with AM.
          2. Perform rack-level aggregation at the end of node-level aggregation again in co-ordination with AM. The aggregation could be performed in parallel across the involved nodes such that each node has aggregated values of different keys.
          3. Schedule reducers taking the key-distribution into account across racks.

          The con will be that the shuffle won't be asynchronous to map computation, but hopefully this wouldn't offset the gains of decreased network and disk I/O.

          PS. http://dl.acm.org/citation.cfm?id=1901088 documents the advantages of multi-level aggregation in the context of graph algorithms modeled as iterative MR jobs.

          Show
          Karthik Kambatla added a comment - Hope I am not severely digressing here. Just thinking out loud. Would be great if the more experienced campaigners can weigh-in here. Would it make sense to do this either at the end of a map (coordinated by the AM) or, as you suggest, as a ReduceTask rather than a daemon? This might make even more sense taking [acmurthy] 's suggestion on MAPREDUCE-199 (data locality hints for reduce tasks) into consideration. Arun's suggestion (rephrased) was to schedule the reducers based on the distribution of keys across map-outputs. Putting the pieces together, it might be even more beneficial to do the following: Perform node-level aggregation (reduce) at the end of maps in co-ordination with AM. Perform rack-level aggregation at the end of node-level aggregation again in co-ordination with AM. The aggregation could be performed in parallel across the involved nodes such that each node has aggregated values of different keys. Schedule reducers taking the key-distribution into account across racks. The con will be that the shuffle won't be asynchronous to map computation, but hopefully this wouldn't offset the gains of decreased network and disk I/O. PS. http://dl.acm.org/citation.cfm?id=1901088 documents the advantages of multi-level aggregation in the context of graph algorithms modeled as iterative MR jobs.
          Hide
          Chris Douglas added a comment -

          >> Will there be one LocalAggregator and one ShuffleHandler per each
          >> reducer? Or, is it a single LocalAggregator/ShuffleHandler daemon
          >> with relevant thread(pool)s per container?
          Latter.
          It's ideal to minimize code modifications and maximize the performance. At the current MR implementation, a ShuffleHandler is launched per container. Keeping it so can save the code modification.

          ShuffleHandler is an auxiliary service loaded in the NodeManager. It's shared across all containers. Unfortunately, this makes the merge/combine difficult to implement as a daemon, because it requires loading user comparators (which shouldn't happen in the NM process).

          Would it make sense to do this either at the end of a map (coordinated by the AM) or, as you suggest, as a ReduceTask rather than a daemon?

          To make this design more generic to support rack-level aggregation, a special task like Reducer which can fetch outputs and reduce them, but write its outputs not to HDFS but to local disk is necessary. With the special task, it can be used in rack-level aggregation by extending the new APIs between mappers and reducers to launch special tasks and delegate the aggregation.

          Carlo Curino and I experimented with this, but (a) saw only slight improvements in job performance and (b) the changes to the AM to accommodate a new task type were extensive. We're currently experimenting with different heuristics for ReduceTasks to exit at the end of the shuffle stage based on skew (from sampling) or in response to resource scarcity (based on RM feedback). With logic to manage skew, we're hoping that scheduling an aggressive range can have a similar effect to combiner tasks, without introducing the new task type.

          Show
          Chris Douglas added a comment - >> Will there be one LocalAggregator and one ShuffleHandler per each >> reducer? Or, is it a single LocalAggregator/ShuffleHandler daemon >> with relevant thread(pool)s per container? Latter. It's ideal to minimize code modifications and maximize the performance. At the current MR implementation, a ShuffleHandler is launched per container. Keeping it so can save the code modification. ShuffleHandler is an auxiliary service loaded in the NodeManager. It's shared across all containers. Unfortunately, this makes the merge/combine difficult to implement as a daemon, because it requires loading user comparators (which shouldn't happen in the NM process). Would it make sense to do this either at the end of a map (coordinated by the AM) or, as you suggest, as a ReduceTask rather than a daemon? To make this design more generic to support rack-level aggregation, a special task like Reducer which can fetch outputs and reduce them, but write its outputs not to HDFS but to local disk is necessary. With the special task, it can be used in rack-level aggregation by extending the new APIs between mappers and reducers to launch special tasks and delegate the aggregation. Carlo Curino and I experimented with this, but (a) saw only slight improvements in job performance and (b) the changes to the AM to accommodate a new task type were extensive. We're currently experimenting with different heuristics for ReduceTasks to exit at the end of the shuffle stage based on skew (from sampling) or in response to resource scarcity (based on RM feedback). With logic to manage skew, we're hoping that scheduling an aggressive range can have a similar effect to combiner tasks, without introducing the new task type.
          Hide
          Tsuyoshi OZAWA added a comment -

          Ah, I'm sorry for confusing you, Karthik. This document say about not container-level aggregation but node-level aggregation. I'll fix the document. Thanks.

          Show
          Tsuyoshi OZAWA added a comment - Ah, I'm sorry for confusing you, Karthik. This document say about not container-level aggregation but node-level aggregation. I'll fix the document. Thanks.
          Hide
          Tsuyoshi OZAWA added a comment -

          Karthik, thanks for your comment.

          Is the local aggregation done asynchronously as the mappers process respective input?

          Partially, yes. After at least two map tasks finishing, local aggregation can be done.

          Will there be one LocalAggregator and one ShuffleHandler per each reducer? Or, is it a single LocalAggregator/ShuffleHandler daemon with relevant thread(pool)s per container?

          Latter.

          It's ideal to minimize code modifications and maximize the performance. At the current MR implementation, a ShuffleHandler is launched per container. Keeping it so can save the code modification.
          BTW, multi-threaded LocalAggregator is very effective for performance, however, the implementation can be more complex than single-threaded one. As a first step, it's reasonable to implement single-thread version.

          The current design doc seems to be aimed at aggregation per container. The bigger goal being aggregation and node/rack levels, does the same design extend/apply to the final goal?

          I thought node-level aggregation and container-level aggregation in MR 2.0 are exactly same.

          To make this design more generic to support rack-level aggregation, a special task like Reducer which can fetch outputs and reduce them, but write its outputs not to HDFS but to local disk is necessary. With the special task, it can be used in rack-level aggregation by extending the new APIs between mappers and reducers to launch special tasks and delegate the aggregation.

          Please ask me if you have any questions.

          • Tsuyoshi
          Show
          Tsuyoshi OZAWA added a comment - Karthik, thanks for your comment. Is the local aggregation done asynchronously as the mappers process respective input? Partially, yes. After at least two map tasks finishing, local aggregation can be done. Will there be one LocalAggregator and one ShuffleHandler per each reducer? Or, is it a single LocalAggregator/ShuffleHandler daemon with relevant thread(pool)s per container? Latter. It's ideal to minimize code modifications and maximize the performance. At the current MR implementation, a ShuffleHandler is launched per container. Keeping it so can save the code modification. BTW, multi-threaded LocalAggregator is very effective for performance, however, the implementation can be more complex than single-threaded one. As a first step, it's reasonable to implement single-thread version. The current design doc seems to be aimed at aggregation per container. The bigger goal being aggregation and node/rack levels, does the same design extend/apply to the final goal? I thought node-level aggregation and container-level aggregation in MR 2.0 are exactly same. To make this design more generic to support rack-level aggregation, a special task like Reducer which can fetch outputs and reduce them, but write its outputs not to HDFS but to local disk is necessary. With the special task, it can be used in rack-level aggregation by extending the new APIs between mappers and reducers to launch special tasks and delegate the aggregation. Please ask me if you have any questions. Tsuyoshi
          Hide
          Karthik Kambatla added a comment -

          Tsuyoshi, thanks for the design doc. The design seems reasonable for container-level aggregation.

          I might have missed something here - could you please comment on the following details for more clarity.

          1. Is the local aggregation done asynchronously as the mappers process respective input?
          2. Will there be one LocalAggregator and one ShuffleHandler per each reducer? Or, is it a single LocalAggregator/ShuffleHandler daemon with relevant thread(pool)s per container?
          3. The current design doc seems to be aimed at aggregation per container. The bigger goal being aggregation and node/rack levels, does the same design extend/apply to the final goal?
          Show
          Karthik Kambatla added a comment - Tsuyoshi, thanks for the design doc. The design seems reasonable for container-level aggregation. I might have missed something here - could you please comment on the following details for more clarity. Is the local aggregation done asynchronously as the mappers process respective input? Will there be one LocalAggregator and one ShuffleHandler per each reducer? Or, is it a single LocalAggregator/ShuffleHandler daemon with relevant thread(pool)s per container? The current design doc seems to be aimed at aggregation per container. The bigger goal being aggregation and node/rack levels, does the same design extend/apply to the final goal?
          Hide
          Tsuyoshi OZAWA added a comment -

          I attached the first speculative draft design of multi-level aggregation. My main concern is whether the changes described in this document are acceptable or not for Hadoop community. Please let me know if you have any opinion. Thanks!

          Show
          Tsuyoshi OZAWA added a comment - I attached the first speculative draft design of multi-level aggregation. My main concern is whether the changes described in this document are acceptable or not for Hadoop community. Please let me know if you have any opinion. Thanks!
          Hide
          Tsuyoshi OZAWA added a comment -

          After debating with Siddharsh, I decided to implement combining against current trunk branch.
          I'll attach the architecture overview and design note as soon as possible(maybe day after tomorrow).

          Show
          Tsuyoshi OZAWA added a comment - After debating with Siddharsh, I decided to implement combining against current trunk branch. I'll attach the architecture overview and design note as soon as possible(maybe day after tomorrow).
          Hide
          Tsuyoshi OZAWA added a comment -

          I'm working on refactoring ApplicationMaster of MapReduce to come true re-using containers(MAPREDUCE-3902) with Siddharth.

          Show
          Tsuyoshi OZAWA added a comment - I'm working on refactoring ApplicationMaster of MapReduce to come true re-using containers( MAPREDUCE-3902 ) with Siddharth.
          Hide
          Tsuyoshi OZAWA added a comment -

          MAPREDUCE-3902 is sub task of this ticket.
          I'll carry on the 2nd topic(per-container combining) of MAPREDUCE-3902.

          Show
          Tsuyoshi OZAWA added a comment - MAPREDUCE-3902 is sub task of this ticket. I'll carry on the 2nd topic(per-container combining) of MAPREDUCE-3902 .
          Hide
          Tsuyoshi OZAWA added a comment -

          We prototyped the combining the result of maps per node/rack.
          the the result of our benchmark is available at mapreduce-dev ML(http://goo.gl/wWBWQ).

          Show
          Tsuyoshi OZAWA added a comment - We prototyped the combining the result of maps per node/rack. the the result of our benchmark is available at mapreduce-dev ML( http://goo.gl/wWBWQ ).

            People

            • Assignee:
              Tsuyoshi OZAWA
              Reporter:
              Tsuyoshi OZAWA
            • Votes:
              0 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

              • Created:
                Updated:

                Development