Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6174

Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: mrv2
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams.

      In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368)

      1. MAPREDUCE-6174.002.patch
        12 kB
        Eric Payne
      2. MAPREDUCE-6174.003.patch
        14 kB
        Eric Payne
      3. MAPREDUCE-6174.004.patch
        14 kB
        Eric Payne
      4. MAPREDUCE-6174.005.patch
        16 kB
        Eric Payne
      5. MAPREDUCE-6174.006.patch
        17 kB
        Eric Payne
      6. MAPREDUCE-6174.007.patch
        17 kB
        Eric Payne
      7. MAPREDUCE-6174.v1.txt
        8 kB
        Eric Payne

        Issue Links

          Activity

          Hide
          eepayne Eric Payne added a comment -

          Gera Shegalov, I have uploaded a patch for this issue. Would you please have a look?

          Show
          eepayne Eric Payne added a comment - Gera Shegalov , I have uploaded a patch for this issue. Would you please have a look?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12697564/MAPREDUCE-6174.v1.txt
          against trunk revision af08425.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5178//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5178//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5178//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697564/MAPREDUCE-6174.v1.txt against trunk revision af08425. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 18 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5178//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5178//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5178//console This message is automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          Gera Shegalov, would you mind taking a look at this patch?
          Thanks.

          Show
          eepayne Eric Payne added a comment - Gera Shegalov , would you mind taking a look at this patch? Thanks.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Sorry for the delay Eric Payne. It's on my list. In the meantime, I noticed there are no unit tests. Can you add/modify some tests to refer to InMemoryMapOutput and OnDiskMapOutput via the base class.

          Show
          jira.shegalov Gera Shegalov added a comment - Sorry for the delay Eric Payne . It's on my list. In the meantime, I noticed there are no unit tests. Can you add/modify some tests to refer to InMemoryMapOutput and OnDiskMapOutput via the base class.
          Hide
          eepayne Eric Payne added a comment -

          Thank you, Gera Shegalov, for your help with this issue. Version 2 of the patch now contains a test that verifies that inMemoryMapOutput and OnDiskMapOutput are derived from the new class.

          Show
          eepayne Eric Payne added a comment - Thank you, Gera Shegalov , for your help with this issue. Version 2 of the patch now contains a test that verifies that inMemoryMapOutput and OnDiskMapOutput are derived from the new class.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 52s The applied patch generated 4 new checkstyle issues (total was 8, now 10).
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 14s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 1m 40s Tests passed in hadoop-mapreduce-client-core.
              38m 8s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730348/MAPREDUCE-6174.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 551615f
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 52s The applied patch generated 4 new checkstyle issues (total was 8, now 10). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 14s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 40s Tests passed in hadoop-mapreduce-client-core.     38m 8s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730348/MAPREDUCE-6174.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 551615f checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/console This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          Fix style and whitespace warnings

          Show
          eepayne Eric Payne added a comment - Fix style and whitespace warnings
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 28s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 47s The applied patch generated 2 new checkstyle issues (total was 8, now 8).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.
              37m 56s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730585/MAPREDUCE-6174.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ffce9a3
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 47s The applied patch generated 2 new checkstyle issues (total was 8, now 8). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.     37m 56s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730585/MAPREDUCE-6174.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ffce9a3 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/console This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          I don't know what caused the checkstyle warnings because https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt gets 404: "The requested resource is not available."

          Show
          eepayne Eric Payne added a comment - I don't know what caused the checkstyle warnings because https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5646/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt gets 404: "The requested resource is not available."
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Hi Eric Payne, thank you for working on the patch. We can check what checkstyle problems a patch introduced as follows.

          Run

          $ mvn clean checkstyle:checkstyle-aggregate
          

          before and after applying the patch.

          Save the resulting target/test/checkstyle-errors.xml after each run. Then we can diff both files. The diff will be concise if we get rid of line numbers.

          ]$ diff <(cat ~/checkstyle-errors-trunk.xml | sed 's/line=\"[0-9]*\"//g') <(cat ./target/test/checkstyle-errors.xml | sed 's/line=\"[0-9]*\"//g') 
          46768a46769,46772
          > <file name="/Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java">
          > <error  column="40" severity="error" message="&apos;conf&apos; hides a field." source="com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck"/>
          > <error  column="51" severity="error" message="&apos;merger&apos; hides a field." source="com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck"/>
          > </file>
          46770d46773
          < <error  severity="error" message="&apos;new&apos; have incorrect indentation level 6, expected level should be 8." source="com.puppycrawl.tools.checkstyle.checks.indentation.IndentationCheck"/>
          46911d46913
          < <error  severity="error" message="Line is longer than 80 characters (found 109)." source="com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck"/>
          
          Show
          jira.shegalov Gera Shegalov added a comment - Hi Eric Payne , thank you for working on the patch. We can check what checkstyle problems a patch introduced as follows. Run $ mvn clean checkstyle:checkstyle-aggregate before and after applying the patch. Save the resulting target/test/checkstyle-errors.xml after each run. Then we can diff both files. The diff will be concise if we get rid of line numbers. ]$ diff <(cat ~/checkstyle-errors-trunk.xml | sed 's/line=\ "[0-9]*\" //g') <(cat ./target/test/checkstyle-errors.xml | sed 's/line=\ "[0-9]*\" //g') 46768a46769,46772 > <file name= "/Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java" > > <error column= "40" severity= "error" message= "&apos;conf&apos; hides a field." source= "com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck" /> > <error column= "51" severity= "error" message= "&apos;merger&apos; hides a field." source= "com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck" /> > </file> 46770d46773 < <error severity= "error" message= "&apos; new &apos; have incorrect indentation level 6, expected level should be 8." source= "com.puppycrawl.tools.checkstyle.checks.indentation.IndentationCheck" /> 46911d46913 < <error severity= "error" message= "Line is longer than 80 characters (found 109)." source= "com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck" />
          Hide
          eepayne Eric Payne added a comment -

          Gera Shegalov, that's very cool. Thanks for the pointers.

          Attaching version 4 of the patch. This test-patch.sh passed for this version.

          Show
          eepayne Eric Payne added a comment - Gera Shegalov , that's very cool. Thanks for the pointers. Attaching version 4 of the patch. This test-patch.sh passed for this version.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks for 004, Eric Payne!
          Few suggestions:

          • We can move setting conf and merger from constructors for OnDisk and InMem to IFileWrappedMapOutput via super
          • We can remove unused parameters reduceId and mapOutputFile from the OnDiskMapOutput constructors.

          I am not sure the new test contributes a lot to code coverage. I think it's not necessary. Instead, in that same TestFetcher we can refer to special classes by the the base class. For example we can replace declarations:

          -    InMemoryMapOutput<Text, Text> immo = mock(InMemoryMapOutput.class);
          +    IFileWrappedMapOutput<Text, Text> immo = mock(InMemoryMapOutput.class);
          

          and

          -    OnDiskMapOutput<Text,Text> odmo = new OnDiskMapOutput<Text,Text>(map1ID,
          -        id, mm, 100L, job, mof, fetcher, true, fs, onDiskMapOutputPath);
          +    IFileWrappedMapOutput<Text,Text> odmo =
          +        new OnDiskMapOutput<Text,Text>(map1ID, mm, 100L, job, fetcher, true, fs,
          +            onDiskMapOutputPath);
           
          

          throughout TestFetcher.

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks for 004, Eric Payne ! Few suggestions: We can move setting conf and merger from constructors for OnDisk and InMem to IFileWrappedMapOutput via super We can remove unused parameters reduceId and mapOutputFile from the OnDiskMapOutput constructors. I am not sure the new test contributes a lot to code coverage. I think it's not necessary. Instead, in that same TestFetcher we can refer to special classes by the the base class. For example we can replace declarations: - InMemoryMapOutput<Text, Text> immo = mock(InMemoryMapOutput.class); + IFileWrappedMapOutput<Text, Text> immo = mock(InMemoryMapOutput.class); and - OnDiskMapOutput<Text,Text> odmo = new OnDiskMapOutput<Text,Text>(map1ID, - id, mm, 100L, job, mof, fetcher, true , fs, onDiskMapOutputPath); + IFileWrappedMapOutput<Text,Text> odmo = + new OnDiskMapOutput<Text,Text>(map1ID, mm, 100L, job, fetcher, true , fs, + onDiskMapOutputPath); throughout TestFetcher.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 17s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.
              38m 30s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731580/MAPREDUCE-6174.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a60f78e
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5698/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5698/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5698/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 17s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.     38m 30s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731580/MAPREDUCE-6174.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a60f78e hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5698/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5698/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5698/console This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          Thanks, Gera Shegalov, for the detailed review.

          Show
          eepayne Eric Payne added a comment - Thanks, Gera Shegalov , for the detailed review.
          Hide
          eepayne Eric Payne added a comment -

          I will work on it later today or tomorrow.

          Show
          eepayne Eric Payne added a comment - I will work on it later today or tomorrow.
          Hide
          eepayne Eric Payne added a comment -

          Thanks Gera Shegalov, for your detailed suggestions. One question:

          • We can remove unused parameters reduceId and mapOutputFile from the OnDiskMapOutput constructors.

          In the first constructor for OnDiskMapOutput, mapOutputFile#getInputFileForWrite is actually used to get the outputPath path for the second constructor.

          MergeManagerImpl appears to be the only place which calls the first constructor for OnDiskMapOutput. This call to the OnDiskMapOutputConstructor could either

          1. calculate the outputPath from within MergeManagerImpl
          2. or, since MergeManagerImpl is the only one calling this constructor and MergeManagerImpl is being modified anyway, MergeManagerImpl could use the second constructor and the first one could be eliminated.
          Show
          eepayne Eric Payne added a comment - Thanks Gera Shegalov , for your detailed suggestions. One question: We can remove unused parameters reduceId and mapOutputFile from the OnDiskMapOutput constructors. In the first constructor for OnDiskMapOutput , mapOutputFile#getInputFileForWrite is actually used to get the outputPath path for the second constructor. MergeManagerImpl appears to be the only place which calls the first constructor for OnDiskMapOutput . This call to the OnDiskMapOutputConstructor could either calculate the outputPath from within MergeManagerImpl or, since MergeManagerImpl is the only one calling this constructor and MergeManagerImpl is being modified anyway, MergeManagerImpl could use the second constructor and the first one could be eliminated.
          Hide
          eepayne Eric Payne added a comment -

          This call to the OnDiskMapOutputConstructor could either

          1. calculate the outputPath from within MergeManagerImpl
          2. or, since MergeManagerImpl is the only one calling this constructor and MergeManagerImpl is being modified anyway, MergeManagerImpl could use the second constructor and the first one could be eliminated.

          Gera Shegalov, I chose option 2 Would you mind taking a look and telling me what you thihk?
          Thanks

          Show
          eepayne Eric Payne added a comment - This call to the OnDiskMapOutputConstructor could either calculate the outputPath from within MergeManagerImpl or, since MergeManagerImpl is the only one calling this constructor and MergeManagerImpl is being modified anyway, MergeManagerImpl could use the second constructor and the first one could be eliminated. Gera Shegalov , I chose option 2 Would you mind taking a look and telling me what you thihk? Thanks
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 89, now 85).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.
              38m 58s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735042/MAPREDUCE-6174.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 446d515
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5752/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5752/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5752/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5752/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 89, now 85). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.     38m 58s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735042/MAPREDUCE-6174.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 446d515 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5752/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5752/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5752/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5752/console This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks for 005 patch, Eric Payne!

          In IFileWrappedMapOutput, we can make conf and merger final fields.

          You are right, I misspoke regarding mapOutputFile in ODMO constructors. Since the old constructors are public and package-scope, respectively, I suggest we keep both signatures unchanged for robust backwards-compatibility, and annotate them as deprecated.

          We can introduce a third package-scope constructor:

            OnDiskMapOutput(TaskAttemptID mapId,
                MergeManagerImpl<K,V> merger, long size,
                JobConf conf,
                int fetcher, boolean primaryMapOutput,
                FileSystem fs, Path outputPath) throws IOException {
              super(conf, merger, mapId, size, primaryMapOutput);
              this.fs = fs;
              this.outputPath = outputPath;
              tmpOutputPath = getTempPath(outputPath, fetcher);
              disk = CryptoUtils.wrapIfNecessary(conf, fs.create(tmpOutputPath));
            }
          

          It can be used to implement both old constructors via this and for MergeManagerImpl.

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks for 005 patch, Eric Payne ! In IFileWrappedMapOutput, we can make conf and merger final fields. You are right, I misspoke regarding mapOutputFile in ODMO constructors. Since the old constructors are public and package-scope, respectively, I suggest we keep both signatures unchanged for robust backwards-compatibility, and annotate them as deprecated. We can introduce a third package-scope constructor: OnDiskMapOutput(TaskAttemptID mapId, MergeManagerImpl<K,V> merger, long size, JobConf conf, int fetcher, boolean primaryMapOutput, FileSystem fs, Path outputPath) throws IOException { super (conf, merger, mapId, size, primaryMapOutput); this .fs = fs; this .outputPath = outputPath; tmpOutputPath = getTempPath(outputPath, fetcher); disk = CryptoUtils.wrapIfNecessary(conf, fs.create(tmpOutputPath)); } It can be used to implement both old constructors via this and for MergeManagerImpl.
          Hide
          eepayne Eric Payne added a comment -

          Thank you, Gera Shegalov, for your detailed feedback. I have made the suggested changes with the latest 006 patch.

          Show
          eepayne Eric Payne added a comment - Thank you, Gera Shegalov , for your detailed feedback. I have made the suggested changes with the latest 006 patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 15s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 9m 50s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 47s The applied patch generated 1 new checkstyle issues (total was 89, now 88).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 38s Tests passed in hadoop-mapreduce-client-core.
              40m 6s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736389/MAPREDUCE-6174.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 63e3fee
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5760/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5760/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5760/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5760/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 15s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 50s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 47s The applied patch generated 1 new checkstyle issues (total was 89, now 88). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 38s Tests passed in hadoop-mapreduce-client-core.     40m 6s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736389/MAPREDUCE-6174.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 63e3fee checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5760/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5760/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5760/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5760/console This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          The checkstyle warning is because the new OnDiskMapOutput constructor has 8 parameters instead of the checkstyle-approved 7. The deprecated constructor had 10, so my contention is that the new one is going in the right direction.

          Show
          eepayne Eric Payne added a comment - The checkstyle warning is because the new OnDiskMapOutput constructor has 8 parameters instead of the checkstyle-approved 7. The deprecated constructor had 10, so my contention is that the new one is going in the right direction.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Hi Eric Payne, thanks for 006. Agreed regarding checkstyle.

          I am sorry, I have one remaining tiny nit before committing. The second deprecated constructor should also delegate for clarity:

            @Deprecated
            @VisibleForTesting
            OnDiskMapOutput(TaskAttemptID mapId, TaskAttemptID reduceId,
                                   MergeManagerImpl<K,V> merger, long size,
                                   JobConf conf,
                                   MapOutputFile mapOutputFile,
                                   int fetcher, boolean primaryMapOutput,
                                   FileSystem fs, Path outputPath) throws IOException {
              this(mapId, merger, size, conf, fetcher, primaryMapOutput, fs, outputPath);
            }
          
          Show
          jira.shegalov Gera Shegalov added a comment - Hi Eric Payne , thanks for 006. Agreed regarding checkstyle. I am sorry, I have one remaining tiny nit before committing. The second deprecated constructor should also delegate for clarity: @Deprecated @VisibleForTesting OnDiskMapOutput(TaskAttemptID mapId, TaskAttemptID reduceId, MergeManagerImpl<K,V> merger, long size, JobConf conf, MapOutputFile mapOutputFile, int fetcher, boolean primaryMapOutput, FileSystem fs, Path outputPath) throws IOException { this (mapId, merger, size, conf, fetcher, primaryMapOutput, fs, outputPath); }
          Hide
          eepayne Eric Payne added a comment -

          Thanks, Gera Shegalov, for your review.

          The second deprecated constructor should also delegate for clarity:

          Agreed. I will make this change. Good catch.

          One additional question: Should the new constructor be @VisibleForTesting? It seems to be fine without it, but Id like to know your opinion.

          Show
          eepayne Eric Payne added a comment - Thanks, Gera Shegalov , for your review. The second deprecated constructor should also delegate for clarity: Agreed. I will make this change. Good catch. One additional question: Should the new constructor be @VisibleForTesting ? It seems to be fine without it, but Id like to know your opinion.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          I don't have a strong preference. We can argue, leave it as a reminder it was just used for tests actually, or remove it because of the new real (non)use. Leaving it up to you.

          Show
          jira.shegalov Gera Shegalov added a comment - I don't have a strong preference. We can argue, leave it as a reminder it was just used for tests actually, or remove it because of the new real (non)use. Leaving it up to you.
          Hide
          eepayne Eric Payne added a comment -

          Thanks Gera Shegalov.

          We can argue, leave it as a reminder it was just used for tests actually, or remove it because of the new real (non)use. Leaving it up to you.

          I have attache version 007.

          I took out the @VisibleForTesting from the all of the constructors, even the old one, since it is not needed.

          Show
          eepayne Eric Payne added a comment - Thanks Gera Shegalov . We can argue, leave it as a reminder it was just used for tests actually, or remove it because of the new real (non)use. Leaving it up to you. I have attache version 007. I took out the @VisibleForTesting from the all of the constructors, even the old one, since it is not needed.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 37s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 43s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 48s The applied patch generated 1 new checkstyle issues (total was 89, now 88).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 38s Tests passed in hadoop-mapreduce-client-core.
              40m 25s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12737284/MAPREDUCE-6174.007.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c59e745
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5766/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5766/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5766/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5766/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 37s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 43s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 48s The applied patch generated 1 new checkstyle issues (total was 89, now 88). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 38s Tests passed in hadoop-mapreduce-client-core.     40m 25s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737284/MAPREDUCE-6174.007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c59e745 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5766/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5766/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5766/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5766/console This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          +1, committing.

          Show
          jira.shegalov Gera Shegalov added a comment - +1, committing.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thank Eric Payne for the contribution and patience during review! Committed 007 to trunk and branch-2.

          Show
          jira.shegalov Gera Shegalov added a comment - Thank Eric Payne for the contribution and patience during review! Committed 007 to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7957 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7957/)
          MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7957 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7957/ ) MAPREDUCE-6174 . Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/)
          MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/ ) MAPREDUCE-6174 . Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/948/)
          MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/948/ ) MAPREDUCE-6174 . Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/)
          MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/ ) MAPREDUCE-6174 . Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/)
          MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/ ) MAPREDUCE-6174 . Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/)
          MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/ ) MAPREDUCE-6174 . Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/)
          MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/ ) MAPREDUCE-6174 . Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) (gera: rev d90c13e2da8867661bf19a802add70145ab9a462) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          Hide
          eepayne Eric Payne added a comment -

          Thank you, Gera Shegalov!

          Show
          eepayne Eric Payne added a comment - Thank you, Gera Shegalov !
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Eric Payne and Gera Shegalov, TestReduceFetchFromPartialMem fails after this commit. Would you check MAPREDUCE-6400?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Eric Payne and Gera Shegalov , TestReduceFetchFromPartialMem fails after this commit. Would you check MAPREDUCE-6400 ?

            People

            • Assignee:
              eepayne Eric Payne
              Reporter:
              eepayne Eric Payne
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development