Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6400

Multiple shuffle transfer fails because input is closed too early

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: task
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      TestReduceFetchFromPartialMem fails.

      Running org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.978 sec <<< FAILURE! - in org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
      testReduceFromPartialMem(org.apache.hadoop.mapred.TestReduceFetchFromPartialMem)  Time elapsed: 69.214 sec  <<< ERROR!
      java.io.IOException: Job failed!
      	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:867)
      	at org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.runJob(TestReduceFetchFromPartialMem.java:300)
      	at org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:93)
      
      
      Results :
      
      Tests in error: 
        TestReduceFetchFromPartialMem.testReduceFromPartialMem:93->runJob:300 ยป IO Job...
      
      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
      
      1. MAPREDUCE-6400.005.patch
        4 kB
        Gera Shegalov
      2. MAPREDUCE-6400.004.patch
        8 kB
        Akira Ajisaka
      3. MAPREDUCE-6400.003.patch
        8 kB
        Gera Shegalov
      4. MAPREDUCE-6400-002.patch
        7 kB
        Brahma Reddy Battula
      5. MAPREDUCE-6400.001.patch
        4 kB
        Gera Shegalov
      6. MAPREDUCE-6400.patch
        1 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          ajisakaa Akira Ajisaka added a comment -

          I used git bisect and found this issue was broken by MAPREDUCE-6174.

          Show
          ajisakaa Akira Ajisaka added a comment - I used git bisect and found this issue was broken by MAPREDUCE-6174 .
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks for pinging Akira Ajisaka. Taking a look!

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks for pinging Akira Ajisaka . Taking a look!
          Hide
          jira.shegalov Gera Shegalov added a comment -

          In MAPREDUCE-6174, Eric Payne made sure that IFileInputStream is closed after shuffle. This was not the case before, but I do think it's a good idea. But we should not close the wrapped physical stream because it shared for all outputs coming from the same node.

          Show
          jira.shegalov Gera Shegalov added a comment - In MAPREDUCE-6174 , Eric Payne made sure that IFileInputStream is closed after shuffle. This was not the case before, but I do think it's a good idea. But we should not close the wrapped physical stream because it shared for all outputs coming from the same node.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Gera Shegalov for taking this issue. The uploaded patch looks to be for HDFS-8591. Would you upload a patch for this issue?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Gera Shegalov for taking this issue. The uploaded patch looks to be for HDFS-8591 . Would you upload a patch for this issue?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Gera Shegalov do you mind,If I give patch..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Gera Shegalov do you mind,If I give patch..?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 4s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 37s There were no new javac warning messages.
          +1 javadoc 9m 46s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 46s The applied patch generated 1 new checkstyle issues (total was 0, now 1).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.
              39m 49s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740064/MAPREDUCE-6400.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ebb9a82
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5806/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5806/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5806/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5806/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 4s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 46s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 46s The applied patch generated 1 new checkstyle issues (total was 0, now 1). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.     39m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740064/MAPREDUCE-6400.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ebb9a82 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5806/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5806/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5806/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5806/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Will give patch to fix the checkstyle..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Will give patch to fix the checkstyle..
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Uploading my patch for completeness. Don't mind if we go with Brahma Reddy Battula's, either.

          Show
          jira.shegalov Gera Shegalov added a comment - Uploading my patch for completeness. Don't mind if we go with Brahma Reddy Battula 's, either.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          As a pattern, having close looks better to me, although in this case it ends doing nothing because there is no readahead and I disabled the wrapped stream close . Eric Payne, Jason Lowe, do you have a preference?

          Show
          jira.shegalov Gera Shegalov added a comment - As a pattern, having close looks better to me, although in this case it ends doing nothing because there is no readahead and I disabled the wrapped stream close . Eric Payne , Jason Lowe , do you have a preference?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Gera Shegalov and Brahma Reddy Battula for creating the patch. In Gera's patch, if inOwner is set to false, we cannot close the wrapped stream when an exception occurs. Therefore I prefer Brahma's one.
          Some comments for Brahma's patch.
          1. Would you please catch IOException in IFileWrappedMapOutput#shuffle? In the catch clause, we should close the wrapped stream and re-throw the exception.
          2. Would you remove unused imports from TestReduceFetchFromPartialMem.java and TestFetcher.java?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Gera Shegalov and Brahma Reddy Battula for creating the patch. In Gera's patch, if inOwner is set to false, we cannot close the wrapped stream when an exception occurs. Therefore I prefer Brahma's one. Some comments for Brahma's patch. 1. Would you please catch IOException in IFileWrappedMapOutput#shuffle ? In the catch clause, we should close the wrapped stream and re-throw the exception. 2. Would you remove unused imports from TestReduceFetchFromPartialMem.java and TestFetcher.java?
          Hide
          jira.shegalov Gera Shegalov added a comment -

          if inOwner is set to false, we cannot close the wrapped stream when an exception occurs.

          A good pattern is when the stream open API consumer closes the stream. In IFlileInputStream for shuffle, the stream is opened outside, and therefore should be and is closed outside.

          Show
          jira.shegalov Gera Shegalov added a comment - if inOwner is set to false, we cannot close the wrapped stream when an exception occurs. A good pattern is when the stream open API consumer closes the stream. In IFlileInputStream for shuffle, the stream is opened outside, and therefore should be and is closed outside.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks a lot Akira Ajisaka , I will address your comments and update the patch soon..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks a lot Akira Ajisaka , I will address your comments and update the patch soon..
          Hide
          ajisakaa Akira Ajisaka added a comment -

          A good pattern is when the stream open API consumer closes the stream. In IFlileInputStream for shuffle, the stream is opened outside, and therefore should be and is closed outside.

          Thanks Gera Shegalov for the comment. I agree with you that the pattern is good. However, before MAPREDUCE-6174, the stream is closed when IOException occurs. Can we create a separate jira for improving the pattern?

          Show
          ajisakaa Akira Ajisaka added a comment - A good pattern is when the stream open API consumer closes the stream. In IFlileInputStream for shuffle, the stream is opened outside, and therefore should be and is closed outside. Thanks Gera Shegalov for the comment. I agree with you that the pattern is good. However, before MAPREDUCE-6174 , the stream is closed when IOException occurs. Can we create a separate jira for improving the pattern?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 57s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 3s The applied patch generated 1 new checkstyle issues (total was 17, now 18).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 39s Tests passed in hadoop-mapreduce-client-core.
          -1 mapreduce tests 94m 56s Tests failed in hadoop-mapreduce-client-jobclient.
              137m 25s  



          Reason Tests
          Failed unit tests hadoop.mapred.TestMRTimelineEventHandling
            hadoop.ipc.TestMRCJCSocketFactory
            hadoop.mapred.TestSpecialCharactersInOutputPath
            hadoop.mapred.TestClusterMapReduceTestCase
          Timed out tests org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740125/MAPREDUCE-6400.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6e3fcff
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 3s The applied patch generated 1 new checkstyle issues (total was 17, now 18). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 39s Tests passed in hadoop-mapreduce-client-core. -1 mapreduce tests 94m 56s Tests failed in hadoop-mapreduce-client-jobclient.     137m 25s   Reason Tests Failed unit tests hadoop.mapred.TestMRTimelineEventHandling   hadoop.ipc.TestMRCJCSocketFactory   hadoop.mapred.TestSpecialCharactersInOutputPath   hadoop.mapred.TestClusterMapReduceTestCase Timed out tests org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740125/MAPREDUCE-6400.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6e3fcff checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5807/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 56s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 0s The applied patch generated 1 new checkstyle issues (total was 0, now 1).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 2m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 1m 33s Tests failed in hadoop-mapreduce-client-core.
          -1 mapreduce tests 105m 34s Tests failed in hadoop-mapreduce-client-jobclient.
              147m 53s  



          Reason Tests
          Failed unit tests hadoop.mapred.TestLineRecordReader
            hadoop.mapreduce.lib.input.TestLineRecordReader
            hadoop.mapred.TestMiniMRChildTask
            hadoop.mapred.TestMerge
            hadoop.mapred.TestReduceFetch
            hadoop.mapreduce.v2.TestUberAM
            hadoop.mapred.TestReduceFetchFromPartialMem
            hadoop.mapred.TestMRIntermediateDataEncryption



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740193/MAPREDUCE-6400-002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cc43288
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 56s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 0s The applied patch generated 1 new checkstyle issues (total was 0, now 1). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 2m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 1m 33s Tests failed in hadoop-mapreduce-client-core. -1 mapreduce tests 105m 34s Tests failed in hadoop-mapreduce-client-jobclient.     147m 53s   Reason Tests Failed unit tests hadoop.mapred.TestLineRecordReader   hadoop.mapreduce.lib.input.TestLineRecordReader   hadoop.mapred.TestMiniMRChildTask   hadoop.mapred.TestMerge   hadoop.mapred.TestReduceFetch   hadoop.mapreduce.v2.TestUberAM   hadoop.mapred.TestReduceFetchFromPartialMem   hadoop.mapred.TestMRIntermediateDataEncryption Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740193/MAPREDUCE-6400-002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cc43288 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5810/console This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Test failures don't reproduce locally.

          Thanks for the patch Brahma Reddy Battula. Thanks Akira Ajisaka for the comments.

          We don't need to do catch-rethrow. We can use abort. I also undid import reorg changes in 002. Here 003.

          Show
          jira.shegalov Gera Shegalov added a comment - Test failures don't reproduce locally. Thanks for the patch Brahma Reddy Battula . Thanks Akira Ajisaka for the comments. We don't need to do catch-rethrow. We can use abort. I also undid import reorg changes in 002. Here 003.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 42s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 21s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 16s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 41s Tests passed in hadoop-mapreduce-client-core.
          -1 mapreduce tests 64m 31s Tests failed in hadoop-mapreduce-client-jobclient.
              107m 13s  



          Reason Tests
          Failed unit tests hadoop.mapred.TestMiniMRClientCluster
          Timed out tests org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740336/MAPREDUCE-6400.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1c13519
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5814/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5814/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5814/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5814/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 21s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 16s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 41s Tests passed in hadoop-mapreduce-client-core. -1 mapreduce tests 64m 31s Tests failed in hadoop-mapreduce-client-jobclient.     107m 13s   Reason Tests Failed unit tests hadoop.mapred.TestMiniMRClientCluster Timed out tests org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740336/MAPREDUCE-6400.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1c13519 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5814/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5814/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5814/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5814/console This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Sorry for arriving late. I'm a bit confused on the latest patch. We're going to close the IFileInputStream but only when something fails? That seems odd. In addition, note that IFileInputStream.close is not trivial – it will consume the rest of the stream to verify the checksum, but we don't care since we're only trying to close when something else went wrong. At that point the checksum being valid is not important, correct? I'm thinking of a scenario where writing to the local disk fails early in a very large transfer. We don't want to spend time reading the rest of the data, rather we just want to abandon the transfer and move on.

          Show
          jlowe Jason Lowe added a comment - Sorry for arriving late. I'm a bit confused on the latest patch. We're going to close the IFileInputStream but only when something fails? That seems odd. In addition, note that IFileInputStream.close is not trivial – it will consume the rest of the stream to verify the checksum, but we don't care since we're only trying to close when something else went wrong. At that point the checksum being valid is not important, correct? I'm thinking of a scenario where writing to the local disk fails early in a very large transfer. We don't want to spend time reading the rest of the data, rather we just want to abandon the transfer and move on.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Jason Lowe, do you mean iFin in the 003 patch should be changed to the wrapped stream?

          IFileWrappedMapOutput.java
            @Override
            public void abort() {
              IOUtils.cleanup(LOG, iFin);
            }
          

          I'm +1 for doing this.

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Jason Lowe , do you mean iFin in the 003 patch should be changed to the wrapped stream? IFileWrappedMapOutput.java @Override public void abort() { IOUtils.cleanup(LOG, iFin); } I'm +1 for doing this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a sample patch to skip IFileInputStream#close.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a sample patch to skip IFileInputStream#close .
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 32s Pre-patch trunk compilation is healthy.
          +1 @author 0m 1s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 31s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 19s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core.
          +1 mapreduce tests 109m 1s Tests passed in hadoop-mapreduce-client-jobclient.
              150m 17s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740987/MAPREDUCE-6400.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 445b132
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5822/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5822/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5822/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5822/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 1s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 31s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 19s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 109m 1s Tests passed in hadoop-mapreduce-client-jobclient.     150m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740987/MAPREDUCE-6400.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 445b132 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5822/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5822/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5822/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5822/console This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          do you mean iFin in the 003 patch should be changed to the wrapped stream?

          Actually no, I'm not a big fan of that approach either. Again, it seems weird to me that we're closing the stream only sometimes, and that sometimes is only when something fails. IMHO the code that is fielding the error and calling abort should be the one closing the stream, as this is likely the one that passed the stream into the map output in the first place. As Gera mentioned above, it's simplest if the code that opens the stream closes the stream. It gets confusing when the ownership is conditional like this.

          Show
          jlowe Jason Lowe added a comment - do you mean iFin in the 003 patch should be changed to the wrapped stream? Actually no, I'm not a big fan of that approach either. Again, it seems weird to me that we're closing the stream only sometimes, and that sometimes is only when something fails. IMHO the code that is fielding the error and calling abort should be the one closing the stream, as this is likely the one that passed the stream into the map output in the first place. As Gera mentioned above, it's simplest if the code that opens the stream closes the stream. It gets confusing when the ownership is conditional like this.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          We don't want to spend time reading the rest of the data, rather we just want to abandon the transfer and move on.

          Good point that we don't care about checksum verification. Initially I was only worried about the read-ahead pool but it's not used for the shuffled data arriving via a TCP socket.

          Show
          jira.shegalov Gera Shegalov added a comment - We don't want to spend time reading the rest of the data, rather we just want to abandon the transfer and move on. Good point that we don't care about checksum verification. Initially I was only worried about the read-ahead pool but it's not used for the shuffled data arriving via a TCP socket.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          005: Minimum patch to bring close closer to open .

          Also fixing potential related file handle leak in LocalFetcher

          Show
          jira.shegalov Gera Shegalov added a comment - 005: Minimum patch to bring close closer to open . Also fixing potential related file handle leak in LocalFetcher
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 42s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 52s There were no new javac warning messages.
          +1 javadoc 9m 53s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 50s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 41s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 1m 47s Tests passed in hadoop-mapreduce-client-core.
              41m 9s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12741435/MAPREDUCE-6400.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 49dfad9
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5835/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5835/testReport/
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5835/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 9m 53s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 47s Tests passed in hadoop-mapreduce-client-core.     41m 9s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12741435/MAPREDUCE-6400.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 49dfad9 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5835/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5835/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5835/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1, the latest simple patch looks good to me. Thanks Jason Lowe for the comment and Gera Shegalov for updating the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - +1, the latest simple patch looks good to me. Thanks Jason Lowe for the comment and Gera Shegalov for updating the patch.
          Hide
          jlowe Jason Lowe added a comment -

          I suspect that connection.disconnect() is already closing the underlying input stream, but it shouldn't hurt if we close it early just in case it doesn't.

          +1 for the latest patch. Committing this.

          Show
          jlowe Jason Lowe added a comment - I suspect that connection.disconnect() is already closing the underlying input stream, but it shouldn't hurt if we close it early just in case it doesn't. +1 for the latest patch. Committing this.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks Brahma, Akira, and Gera! I committed this to trunk and branch-2.

          Show
          jlowe Jason Lowe added a comment - Thanks Brahma, Akira, and Gera! I committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8056 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8056/)
          MAPREDUCE-6400. Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8056 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8056/ ) MAPREDUCE-6400 . Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #239 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/239/)
          MAPREDUCE-6400. Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #239 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/239/ ) MAPREDUCE-6400 . Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #969 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/969/)
          MAPREDUCE-6400. Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #969 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/969/ ) MAPREDUCE-6400 . Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2167 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2167/)
          MAPREDUCE-6400. Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2167 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2167/ ) MAPREDUCE-6400 . Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/228/)
          MAPREDUCE-6400. Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/228/ ) MAPREDUCE-6400 . Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/237/)
          MAPREDUCE-6400. Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/237/ ) MAPREDUCE-6400 . Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2185 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2185/)
          MAPREDUCE-6400. Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2185 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2185/ ) MAPREDUCE-6400 . Multiple shuffle transfer fails because input is closed too early. Contributed by Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov. (jlowe: rev 72d08a0e41efda635baa985d55d67cb059a7c07c) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java hadoop-mapreduce-project/CHANGES.txt

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              ajisakaa Akira Ajisaka
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development