Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6474

ShuffleHandler can possibly exhaust nodemanager file descriptors

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The async nature of the shufflehandler can cause it to open a huge number of
      file descriptors, when it runs out it crashes.

      Scenario:
      Job with 6K reduces, slow start set to 0.95, about 40 map outputs per node.
      Let's say all 6K reduces hit a node at about same time asking for their
      outputs. Each reducer will ask for all 40 map outputs over a single socket in a
      single request (not necessarily all 40 at once, but with coalescing it is
      likely to be a large number).

      sendMapOutput() will open the file for random reading and then perform an async transfer of the particular portion of this file(). This will theoretically
      happen 6000*40=240000 times which will run the NM out of file descriptors and cause it to crash.

      The algorithm should be refactored a little to not open the fds until they're
      actually needed.

      1. YARN-2410-v1.patch
        18 kB
        Kuhu Shukla
      2. YARN-2410-v10.patch
        15 kB
        Kuhu Shukla
      3. YARN-2410-v11.patch
        16 kB
        Kuhu Shukla
      4. YARN-2410-v2.patch
        18 kB
        Kuhu Shukla
      5. YARN-2410-v3.patch
        18 kB
        Kuhu Shukla
      6. YARN-2410-v4.patch
        11 kB
        Kuhu Shukla
      7. YARN-2410-v5.patch
        16 kB
        Kuhu Shukla
      8. YARN-2410-v6.patch
        16 kB
        Kuhu Shukla
      9. YARN-2410-v7.patch
        15 kB
        Kuhu Shukla
      10. YARN-2410-v8.patch
        14 kB
        Kuhu Shukla
      11. YARN-2410-v9.patch
        15 kB
        Kuhu Shukla

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment -

          Adding this as a part of BIGTOP-1403 also. hopefully we can have some smoke tests which induce this bug.

          If you have a code snippet or patch to contribute that generates this , feel free to attach it there, we can help you to refine it.

          Show
          jayunit100 jay vyas added a comment - Adding this as a part of BIGTOP-1403 also. hopefully we can have some smoke tests which induce this bug. If you have a code snippet or patch to contribute that generates this , feel free to attach it there, we can help you to refine it.
          Hide
          airbots Chen He added a comment -

          I propose to use ChannelPipe to replace current ChannelFuture.

          Show
          airbots Chen He added a comment - I propose to use ChannelPipe to replace current ChannelFuture.
          Hide
          kshukla Kuhu Shukla added a comment -

          Chen He Are you working on this JIRA? If not, I can assign it to myself. I have a patch in progress.

          Show
          kshukla Kuhu Shukla added a comment - Chen He Are you working on this JIRA? If not, I can assign it to myself. I have a patch in progress.
          Hide
          airbots Chen He added a comment -

          Thank you for the work, Kuhu Shukla

          Show
          airbots Chen He added a comment - Thank you for the work, Kuhu Shukla
          Hide
          kshukla Kuhu Shukla added a comment -

          The ShuffleHandler messageReceived calls sendMapOutput only if the number of open files for a given reduceId is within a configurable limit value (mapreduce.shuffle.map.filecount). The count is incremented per call of sendMapOutput(). The channel is closed after this limit is reached.

          Show
          kshukla Kuhu Shukla added a comment - The ShuffleHandler messageReceived calls sendMapOutput only if the number of open files for a given reduceId is within a configurable limit value (mapreduce.shuffle.map.filecount). The count is incremented per call of sendMapOutput(). The channel is closed after this limit is reached.
          Hide
          kshukla Kuhu Shukla added a comment -

          Adding another listener which keeps count of open files per reduceId.

          Show
          kshukla Kuhu Shukla added a comment - Adding another listener which keeps count of open files per reduceId.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12745281/YARN-2410-v1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4084eaf
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8532/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745281/YARN-2410-v1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4084eaf Console output https://builds.apache.org/job/PreCommit-YARN-Build/8532/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Patch without no-prefix as git apply works without no-prefix.

          Show
          kshukla Kuhu Shukla added a comment - Patch without no-prefix as git apply works without no-prefix.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12745290/YARN-2410-v2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4084eaf
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8533/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745290/YARN-2410-v2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4084eaf Console output https://builds.apache.org/job/PreCommit-YARN-Build/8533/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 25s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 21s The applied patch generated 32 new checkstyle issues (total was 60, now 92).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 0m 48s The patch appears to introduce 4 new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 0m 26s Tests passed in hadoop-mapreduce-client-shuffle.
              36m 40s  



          Reason Tests
          FindBugs module:hadoop-mapreduce-client-shuffle



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12745525/YARN-2410-v3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3ec0a04
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8551/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8551/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
          hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/8551/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8551/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8551/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 25s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 21s The applied patch generated 32 new checkstyle issues (total was 60, now 92). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 0m 48s The patch appears to introduce 4 new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 26s Tests passed in hadoop-mapreduce-client-shuffle.     36m 40s   Reason Tests FindBugs module:hadoop-mapreduce-client-shuffle Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745525/YARN-2410-v3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3ec0a04 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8551/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8551/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/8551/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8551/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8551/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Revamped patch that uses a Map to store the number of openfiles per reduceId and passes the updated openfiles value through the channel as an attachment. The number of files that can be open per reducer is configurable.

          Show
          kshukla Kuhu Shukla added a comment - Revamped patch that uses a Map to store the number of openfiles per reduceId and passes the updated openfiles value through the channel as an attachment. The number of files that can be open per reducer is configurable.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 9m 43s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 20s The applied patch generated 19 new checkstyle issues (total was 60, now 79).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 21s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 0m 47s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 0m 21s Tests passed in hadoop-mapreduce-client-shuffle.
              36m 58s  



          Reason Tests
          FindBugs module:hadoop-mapreduce-client-shuffle



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747256/YARN-2410-v4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 156f24e
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8676/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8676/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
          hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/8676/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8676/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8676/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 9m 43s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 20s The applied patch generated 19 new checkstyle issues (total was 60, now 79). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 0m 47s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 21s Tests passed in hadoop-mapreduce-client-shuffle.     36m 58s   Reason Tests FindBugs module:hadoop-mapreduce-client-shuffle Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747256/YARN-2410-v4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 156f24e checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8676/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8676/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/8676/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8676/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8676/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Please disregard all previous patches. Uploading the latest patch shortly.

          Show
          kshukla Kuhu Shukla added a comment - Please disregard all previous patches. Uploading the latest patch shortly.
          Hide
          kshukla Kuhu Shukla added a comment -

          This is the latest revised patch. A messageReceived() call uses two counters mapsToWait and mapsToSend within the ReduceContext class for throttling the number of sendMapOutput calls. Due to asynchronous nature of Netty, these counters are atomic. A revised test case that mocks Netty operations is also included.

          Every completed IO operation by sendMapOutput will start another until the entire mapIds list for a given request is processed.

          Show
          kshukla Kuhu Shukla added a comment - This is the latest revised patch. A messageReceived() call uses two counters mapsToWait and mapsToSend within the ReduceContext class for throttling the number of sendMapOutput calls. Due to asynchronous nature of Netty, these counters are atomic. A revised test case that mocks Netty operations is also included. Every completed IO operation by sendMapOutput will start another until the entire mapIds list for a given request is processed.
          Hide
          kshukla Kuhu Shukla added a comment -

          Requesting Nathan Roberts and Jason Lowe to review the revised patch. Thanks.

          Show
          kshukla Kuhu Shukla added a comment - Requesting Nathan Roberts and Jason Lowe to review the revised patch. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 28s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 6s There were no new javac warning messages.
          +1 javadoc 10m 16s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 22s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 26s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.
              38m 46s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749100/YARN-2410-v5.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8572a5a
          hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/8782/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8782/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8782/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 6s There were no new javac warning messages. +1 javadoc 10m 16s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 22s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.     38m 46s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749100/YARN-2410-v5.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8572a5a hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/8782/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8782/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8782/console This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updating field Target version as 2.7.2. Fix version is added when the issue is committed!!

          Show
          rohithsharma Rohith Sharma K S added a comment - Updating field Target version as 2.7.2. Fix version is added when the issue is committed!!
          Hide
          kshukla Kuhu Shukla added a comment -

          Requesting Nathan Roberts , Jason Lowe , Jonathan Eagles for their comments/review. Thanks!

          Show
          kshukla Kuhu Shukla added a comment - Requesting Nathan Roberts , Jason Lowe , Jonathan Eagles for their comments/review. Thanks!
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Kuhu!

          Do we really need getShuffle/setShuffle and a new shuffle field? Much simpler to access pipelineFact.SHUFFLE directly or add a getShuffle method to HttpPipelineFactory and use that. We shouldn't be redundantly tracking shuffle in the ShuffleHandler class.

          Rather than catch and swallowing the exception from metrics.operationComplete (which shouldn't happen), can't we just let this propagate up? I would think it would eventually trigger exceptionCaught to be called which should do the right thing like any other exception on the channel.

          The ChannelHandlerContext has the corresponding channel, so do we really need SendMapOutputParams to store it separately?

          SendMapOutputParams should be treated like an immutable object. There's no need to set any of its contents after it is created, so we should just remove all the set methods. Same applies to ReduceContext. Actually I'm not really sure why SendMapOutputParams exists separate from ReduceContext. There should be a one-to-one relationship there. I think we could just promote all of the members of SendMapOutputParams into ReduceContext and cut down on some of the excess boilerplate.

          ReduceContext has an unused Configuration variable and getConf is never called.

          Seems like we could associate the ReduceContext with the ReduceMapFileCount listener object directly when we construct it. That way we don't have to fish it out of the channel attachment and the code plays nicer with other things that might want to use the channel attachment for something.

          It looks like we can now send an INTERNAL_SERVER_ERROR followed by a NOT_FOUND error since sendMap will return null when it sends an internal error. Maybe sendMap should take care of sending all the appropriate errors directly.

          Why was reduceContext added as a TestShuffleHandler instance variable? It's specific to the new test.

          The test would be a bit more readable with some factoring out of some code to utility methods, e.g.: createMockChannel method which takes care of mocking up all the stuff needed to mock a channel. Also the overridden ShuffleHandler is enough code that it should just be a separate class within the test rather than inline in the method.

          Nit: The comment for mapreduce.shuffle.max.send.map.count should mention simultaneous or concurrent otherwise it implies it will only send that many outputs total. Also it may be more clear if the property were named something like mapreduce.shuffle.max.session-open-files or something similar, although I'm not super excited about that name either.

          Nit: some (all?) of the returns in ReduceMapFileCount.operationComplete would be easier to follow using an else clause, e.g.:

                if (waitCount == 0) {
                  metrics.operationComplete(future);
                  future.getChannel().close();
                } else {
                  shuffle.sendMap(rc.getSendMapOutputParams().getCtx(),
                      rc.getSendMapOutputParams().getInfoMap());
                }
          

          Nit: whitespace between field definitions and internal class definitions and also between method definitions would help readability.

          Nit: Per the coding conventions class and instance variables should appear before constructors which in turn appear before methods

          Nit: variables should be declared when they are initialized if possible, e.g.: nextId and mapId in sendMap.

          Nit: Please organize the imports in the test, there's a mix of static and non-static imports.

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Kuhu! Do we really need getShuffle/setShuffle and a new shuffle field? Much simpler to access pipelineFact.SHUFFLE directly or add a getShuffle method to HttpPipelineFactory and use that. We shouldn't be redundantly tracking shuffle in the ShuffleHandler class. Rather than catch and swallowing the exception from metrics.operationComplete (which shouldn't happen), can't we just let this propagate up? I would think it would eventually trigger exceptionCaught to be called which should do the right thing like any other exception on the channel. The ChannelHandlerContext has the corresponding channel, so do we really need SendMapOutputParams to store it separately? SendMapOutputParams should be treated like an immutable object. There's no need to set any of its contents after it is created, so we should just remove all the set methods. Same applies to ReduceContext. Actually I'm not really sure why SendMapOutputParams exists separate from ReduceContext. There should be a one-to-one relationship there. I think we could just promote all of the members of SendMapOutputParams into ReduceContext and cut down on some of the excess boilerplate. ReduceContext has an unused Configuration variable and getConf is never called. Seems like we could associate the ReduceContext with the ReduceMapFileCount listener object directly when we construct it. That way we don't have to fish it out of the channel attachment and the code plays nicer with other things that might want to use the channel attachment for something. It looks like we can now send an INTERNAL_SERVER_ERROR followed by a NOT_FOUND error since sendMap will return null when it sends an internal error. Maybe sendMap should take care of sending all the appropriate errors directly. Why was reduceContext added as a TestShuffleHandler instance variable? It's specific to the new test. The test would be a bit more readable with some factoring out of some code to utility methods, e.g.: createMockChannel method which takes care of mocking up all the stuff needed to mock a channel. Also the overridden ShuffleHandler is enough code that it should just be a separate class within the test rather than inline in the method. Nit: The comment for mapreduce.shuffle.max.send.map.count should mention simultaneous or concurrent otherwise it implies it will only send that many outputs total. Also it may be more clear if the property were named something like mapreduce.shuffle.max.session-open-files or something similar, although I'm not super excited about that name either. Nit: some (all?) of the returns in ReduceMapFileCount.operationComplete would be easier to follow using an else clause, e.g.: if (waitCount == 0) { metrics.operationComplete( future ); future .getChannel().close(); } else { shuffle.sendMap(rc.getSendMapOutputParams().getCtx(), rc.getSendMapOutputParams().getInfoMap()); } Nit: whitespace between field definitions and internal class definitions and also between method definitions would help readability. Nit: Per the coding conventions class and instance variables should appear before constructors which in turn appear before methods Nit: variables should be declared when they are initialized if possible, e.g.: nextId and mapId in sendMap. Nit: Please organize the imports in the test, there's a mix of static and non-static imports.
          Hide
          kshukla Kuhu Shukla added a comment -

          Thank you so much Jason Lowe for the detailed feedback. I have made all but 2 changes and request your further comments on that.

          Actually I'm not really sure why SendMapOutputParams exists separate from ReduceContext. There should be a one-to-one relationship there.

          I totally agree. The only reason was findbugs which does not allow more than 7 parameters in a function call( or the constructor that would populate these values). If this is not an issue, I can move them into a single class. For now I have made SendMapOutputParams an inner class to ReduceContext.

          Why was reduceContext added as a TestShuffleHandler instance variable? It's specific to the new test.

          The reduceContext is a variable holds the value set by the setAttachment() method and is used by the getAttachment() answer. If I declare it in the test method, it needs be final which cannot be done due to it being used by the setter. I am looking for another way. Let me know what you think.

          All other items have been done.

          Show
          kshukla Kuhu Shukla added a comment - Thank you so much Jason Lowe for the detailed feedback. I have made all but 2 changes and request your further comments on that. Actually I'm not really sure why SendMapOutputParams exists separate from ReduceContext. There should be a one-to-one relationship there. I totally agree. The only reason was findbugs which does not allow more than 7 parameters in a function call( or the constructor that would populate these values). If this is not an issue, I can move them into a single class. For now I have made SendMapOutputParams an inner class to ReduceContext. Why was reduceContext added as a TestShuffleHandler instance variable? It's specific to the new test. The reduceContext is a variable holds the value set by the setAttachment() method and is used by the getAttachment() answer. If I declare it in the test method, it needs be final which cannot be done due to it being used by the setter. I am looking for another way. Let me know what you think. All other items have been done.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 44s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 44s There were no new javac warning messages.
          +1 javadoc 9m 51s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 21s The applied patch generated 14 new checkstyle issues (total was 60, now 74).
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 0m 47s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 0m 21s Tests passed in hadoop-mapreduce-client-shuffle.
              37m 15s  



          Reason Tests
          FindBugs module:hadoop-mapreduce-client-shuffle



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754414/YARN-2410-v6.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 9b68577
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9019/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9019/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9019/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
          hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9019/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9019/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9019/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 44s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 51s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 21s The applied patch generated 14 new checkstyle issues (total was 60, now 74). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 0m 47s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 21s Tests passed in hadoop-mapreduce-client-shuffle.     37m 15s   Reason Tests FindBugs module:hadoop-mapreduce-client-shuffle Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754414/YARN-2410-v6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9b68577 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9019/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9019/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9019/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9019/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9019/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9019/console This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch!

          The only reason was findbugs which does not allow more than 7 parameters in a function call

          Normally a builder pattern is used to make the code more readable in those situations. However I don't think we need more than 7. ReduceContext really only needs mapIds, reduceId, channelCtx, user, infoMap, and outputBasePathStr. The other two parameters are either known to be zero (should not be passed) and can be derived from another (size of mapIds). As such we don't need SendMapOutputParams.

          The reduceContext is a variable holds the value set by the setAttachment() method and is used by the getAttachment() answer. If I declare it in the test method, it needs be final which cannot be done due to it being used by the setter.

          createMockChannel can simply have a ReduceContext parameter, marked final, and that should solve that problem. But I thought we were getting rid of the use of channel attachments and just associating the context with the listener directly?

          Related to the last comment, we're still using channel attachments. sendMap can just take a ReduceContext parameter, and the listener can provide its context when calling it. No need for channel attachments.

          This can NPE since we're checking for null after we already use it:

          +            nextMap = sendMapOutput(
          +                reduceContext.getSendMapOutputParams().getCtx(),
          +                reduceContext.getSendMapOutputParams().getCtx().getChannel(),
          +                reduceContext.getSendMapOutputParams().getUser(), mapId,
          +                reduceContext.getSendMapOutputParams().getReduceId(), info);
          +            nextMap.addListener(new ReduceMapFileCount(reduceContext));
          +            if (null == nextMap) {
          

          maxSendMapCount should be cached during serviceInit like the other conf-derived settings so we aren't doing conf lookups on every shuffle.

          The indentation in sendMap isn't correct, as code is indented after a conditional block at the same level as the contents of the conditional block. There's other places that are over-indented.

          MockShuffleHandler only needs to override one thing, getShuffle, but the mock that method returns has to override a bunch of stuff. It makes more sense to create a separate class for the mocked Shuffle than the mocked ShuffleHandler.

          Should the mock Future stuff be part of creating the mocked channel? Can simply pass the listener list to use as an arg to the method that mocks up the channel.

          Nit: SHUFFLE_MAX_SEND_COUNT should probably be something like SHUFFLE_MAX_SESSION_OPEN_FILES to better match the property name. Similarly maxSendMapCount could have a more appropriate name.

          Nit: Format for 80 columns

          Nit: There's still instances where we have a class definition immediately after variable definitions and a lack of whitespace between classes and methods or between methods. Whitespace would help readability in those places.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch! The only reason was findbugs which does not allow more than 7 parameters in a function call Normally a builder pattern is used to make the code more readable in those situations. However I don't think we need more than 7. ReduceContext really only needs mapIds, reduceId, channelCtx, user, infoMap, and outputBasePathStr. The other two parameters are either known to be zero (should not be passed) and can be derived from another (size of mapIds). As such we don't need SendMapOutputParams. The reduceContext is a variable holds the value set by the setAttachment() method and is used by the getAttachment() answer. If I declare it in the test method, it needs be final which cannot be done due to it being used by the setter. createMockChannel can simply have a ReduceContext parameter, marked final, and that should solve that problem. But I thought we were getting rid of the use of channel attachments and just associating the context with the listener directly? Related to the last comment, we're still using channel attachments. sendMap can just take a ReduceContext parameter, and the listener can provide its context when calling it. No need for channel attachments. This can NPE since we're checking for null after we already use it: + nextMap = sendMapOutput( + reduceContext.getSendMapOutputParams().getCtx(), + reduceContext.getSendMapOutputParams().getCtx().getChannel(), + reduceContext.getSendMapOutputParams().getUser(), mapId, + reduceContext.getSendMapOutputParams().getReduceId(), info); + nextMap.addListener(new ReduceMapFileCount(reduceContext)); + if (null == nextMap) { maxSendMapCount should be cached during serviceInit like the other conf-derived settings so we aren't doing conf lookups on every shuffle. The indentation in sendMap isn't correct, as code is indented after a conditional block at the same level as the contents of the conditional block. There's other places that are over-indented. MockShuffleHandler only needs to override one thing, getShuffle, but the mock that method returns has to override a bunch of stuff. It makes more sense to create a separate class for the mocked Shuffle than the mocked ShuffleHandler. Should the mock Future stuff be part of creating the mocked channel? Can simply pass the listener list to use as an arg to the method that mocks up the channel. Nit: SHUFFLE_MAX_SEND_COUNT should probably be something like SHUFFLE_MAX_SESSION_OPEN_FILES to better match the property name. Similarly maxSendMapCount could have a more appropriate name. Nit: Format for 80 columns Nit: There's still instances where we have a class definition immediately after variable definitions and a lack of whitespace between classes and methods or between methods. Whitespace would help readability in those places.
          Hide
          kshukla Kuhu Shukla added a comment -

          Modified ShuffleHandler to not use channel attachments. Moved MockNetty code to a helper method.

          Show
          kshukla Kuhu Shukla added a comment - Modified ShuffleHandler to not use channel attachments. Moved MockNetty code to a helper method.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 59s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 51s There were no new javac warning messages.
          +1 javadoc 10m 7s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 21s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.
              37m 47s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754746/YARN-2410-v7.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d9c1fab
          hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9046/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9046/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9046/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 59s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 21s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 19s Tests passed in hadoop-mapreduce-client-shuffle.     37m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754746/YARN-2410-v7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d9c1fab hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9046/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9046/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9046/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Correcting 80 character line limit for test

          Show
          kshukla Kuhu Shukla added a comment - Correcting 80 character line limit for test
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 43s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 47s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 20s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.
              37m 12s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754910/YARN-2410-v8.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0113e45
          hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9058/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9058/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9058/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 20s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.     37m 12s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754910/YARN-2410-v8.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0113e45 hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9058/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9058/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9058/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Jason Lowe Thank you so much for the comments. I have updated the patch with all modifications but the Test Shuffle class separation one. Appreciate your feedback.

          Show
          kshukla Kuhu Shukla added a comment - Jason Lowe Thank you so much for the comments. I have updated the patch with all modifications but the Test Shuffle class separation one. Appreciate your feedback.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch. I see now that since Shuffle is not a static class it's too much trouble to try to factor out the overrides for the test.

          ReduceContext should be a private class.

          sendMap takes a reduce context, a channel context, and an info map, but the latter two are already in the reduce context. Seems like sendMap should just take a reduce context argument.

          If sendMap returns null then I don't think we want messageReceived to blindly keep calling sendMap in the loop.

          Why were the override decorators removed from verifyRequest and getMapOutputInfo? It's pretty important that those actually override a method.

          mockNetty is too monolothic like the old test and a bit unwieldy in that callers are expected to start mocking and then let mockNetty finish the job. The channel and message event mocking is just a few simple lines for each and would be fine to stay in the main test method. Utility methods like createMockChannelFuture(channel, listenerList) and createMockHttpRequest() would help keep the original test method manageable in length and factor out some of the more complicated mocking of individual objects.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch. I see now that since Shuffle is not a static class it's too much trouble to try to factor out the overrides for the test. ReduceContext should be a private class. sendMap takes a reduce context, a channel context, and an info map, but the latter two are already in the reduce context. Seems like sendMap should just take a reduce context argument. If sendMap returns null then I don't think we want messageReceived to blindly keep calling sendMap in the loop. Why were the override decorators removed from verifyRequest and getMapOutputInfo? It's pretty important that those actually override a method. mockNetty is too monolothic like the old test and a bit unwieldy in that callers are expected to start mocking and then let mockNetty finish the job. The channel and message event mocking is just a few simple lines for each and would be fine to stay in the main test method. Utility methods like createMockChannelFuture(channel, listenerList) and createMockHttpRequest() would help keep the original test method manageable in length and factor out some of the more complicated mocking of individual objects.
          Hide
          kshukla Kuhu Shukla added a comment -

          sendMap should have only reduceContext as an argument. Test refactored to have helper methods.

          Show
          kshukla Kuhu Shukla added a comment - sendMap should have only reduceContext as an argument. Test refactored to have helper methods.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 49s There were no new javac warning messages.
          +1 javadoc 10m 9s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 20s The applied patch generated 1 new checkstyle issues (total was 61, now 61).
          -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 0m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 0m 21s Tests passed in hadoop-mapreduce-client-shuffle.
              37m 53s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754953/YARN-2410-v9.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0f0e897
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9063/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9063/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9063/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9063/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9063/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 49s There were no new javac warning messages. +1 javadoc 10m 9s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 20s The applied patch generated 1 new checkstyle issues (total was 61, now 61). -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 21s Tests passed in hadoop-mapreduce-client-shuffle.     37m 53s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754953/YARN-2410-v9.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0f0e897 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9063/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-shuffle.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9063/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9063/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9063/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9063/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Fixing whitespace and checkstyle issues.

          Show
          kshukla Kuhu Shukla added a comment - Fixing whitespace and checkstyle issues.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 12s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 57s There were no new javac warning messages.
          +1 javadoc 10m 2s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 20s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.
              38m 5s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754970/YARN-2410-v10.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4014ce5
          hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9066/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9066/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9066/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 12s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 57s There were no new javac warning messages. +1 javadoc 10m 2s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 20s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.     38m 5s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754970/YARN-2410-v10.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4014ce5 hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9066/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9066/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9066/console This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Jason Lowe, thank you for your comments. I regret the oversight. Modified as suggested. Please let me know if any other changes are required.

          Show
          kshukla Kuhu Shukla added a comment - Jason Lowe , thank you for your comments. I regret the oversight. Modified as suggested. Please let me know if any other changes are required.
          Hide
          jlowe Jason Lowe added a comment -

          +1 latest patch looks good to me. Will commit tomorrow if there are no further comments.

          Show
          jlowe Jason Lowe added a comment - +1 latest patch looks good to me. Will commit tomorrow if there are no further comments.
          Hide
          nroberts Nathan Roberts added a comment -

          One minor comment. If it's not too much trouble, could you add some comments about why we need to limit the number of FDs, how the code is accomplishing it, and why atomic operations need to be used in send_map(). It will help future maintainers of the code understand some of the rationale behind the complexities in this area.

          Show
          nroberts Nathan Roberts added a comment - One minor comment. If it's not too much trouble, could you add some comments about why we need to limit the number of FDs, how the code is accomplishing it, and why atomic operations need to be used in send_map(). It will help future maintainers of the code understand some of the rationale behind the complexities in this area.
          Hide
          kshukla Kuhu Shukla added a comment -

          Adding documentation comments.

          Show
          kshukla Kuhu Shukla added a comment - Adding documentation comments.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 11s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 10m 5s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 20s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 0m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.
              38m 1s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755025/YARN-2410-v11.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4014ce5
          hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9072/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9072/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9072/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 10m 5s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 20s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 20s Tests passed in hadoop-mapreduce-client-shuffle.     38m 1s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755025/YARN-2410-v11.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4014ce5 hadoop-mapreduce-client-shuffle test log https://builds.apache.org/job/PreCommit-YARN-Build/9072/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9072/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9072/console This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Thanks for the additional code comments. +1

          Show
          nroberts Nathan Roberts added a comment - Thanks for the additional code comments. +1
          Hide
          jlowe Jason Lowe added a comment -

          +1 for the latest patch. Committing this.

          Show
          jlowe Jason Lowe added a comment - +1 for the latest patch. Committing this.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Kuhu for the contribution and to Nathan for additional review! I committed this to trunk, branch-2, and branch-2.7.

          Show
          jlowe Jason Lowe added a comment - Thanks to Kuhu for the contribution and to Nathan for additional review! I committed this to trunk, branch-2, and branch-2.7.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8429 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8429/)
          MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8429 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8429/ ) MAPREDUCE-6474 . ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #368 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/368/)
          MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #368 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/368/ ) MAPREDUCE-6474 . ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #374 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/374/)
          MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #374 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/374/ ) MAPREDUCE-6474 . ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1106 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1106/)
          MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1106 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1106/ ) MAPREDUCE-6474 . ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2316 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2316/)
          MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2316 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2316/ ) MAPREDUCE-6474 . ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2293 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2293/)
          MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2293 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2293/ ) MAPREDUCE-6474 . ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #354 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/354/)
          MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #354 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/354/ ) MAPREDUCE-6474 . ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla (jlowe: rev 8e615588d5216394d0251a9c97bd706537856c6d) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java hadoop-mapreduce-project/CHANGES.txt

            People

            • Assignee:
              kshukla Kuhu Shukla
              Reporter:
              nroberts Nathan Roberts
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development