Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5251

Reducer should not implicate map attempt if it has insufficient space to fetch map output

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.7, 2.0.4-alpha
    • Fix Version/s: 0.23.10, 2.1.1-beta
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail.

      In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output.

      1. MAPREDUCE-5251-2.txt
        7 kB
        Ashwin Shankar
      2. MAPREDUCE-5251-3.txt
        5 kB
        Ashwin Shankar
      3. MAPREDUCE-5251-4.txt
        5 kB
        Ashwin Shankar
      4. MAPREDUCE-5251-5.txt
        5 kB
        Ashwin Shankar
      5. MAPREDUCE-5251-6.txt
        5 kB
        Ashwin Shankar
      6. MAPREDUCE-5251-7.txt
        5 kB
        Ashwin Shankar
      7. MAPREDUCE-5251-7-b23.txt
        6 kB
        Ashwin Shankar

        Issue Links

          Activity

          Hide
          jlowe Jason Lowe added a comment -

          Sample exception from a reducer when this occurs:

          2013-05-14 01:23:14,382 INFO [fetcher#4]
          org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#4 failed to read map headerattempt_1368206650998_32139_m_000258_0 decomp: 3083285359, 324928373
          org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1368206650998_32139_r_001028_0/map_258.out
                  at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
                  at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
                  at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
                  at org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213)
                  at org.apache.hadoop.mapreduce.task.reduce.MapOutput.<init>(MapOutput.java:80)
                  at org.apache.hadoop.mapreduce.task.reduce.MergeManager.reserve(MergeManager.java:258)
                  at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:341)
                  at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:276)
                  at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:156)
          2013-05-14 01:23:14,382 WARN [fetcher#4] org.apache.hadoop.mapreduce.task.reduce.Fetcher: copyMapOutput failed for tasks [attempt_1368206650998_32139_m_000258_0]
          
          Show
          jlowe Jason Lowe added a comment - Sample exception from a reducer when this occurs: 2013-05-14 01:23:14,382 INFO [fetcher#4] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#4 failed to read map headerattempt_1368206650998_32139_m_000258_0 decomp: 3083285359, 324928373 org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1368206650998_32139_r_001028_0/map_258.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213) at org.apache.hadoop.mapreduce.task.reduce.MapOutput.<init>(MapOutput.java:80) at org.apache.hadoop.mapreduce.task.reduce.MergeManager.reserve(MergeManager.java:258) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:341) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:276) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:156) 2013-05-14 01:23:14,382 WARN [fetcher#4] org.apache.hadoop.mapreduce.task.reduce.Fetcher: copyMapOutput failed for tasks [attempt_1368206650998_32139_m_000258_0]
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          The issue here is was that any IOException that arises from Fetcher.copyMapOutput is always blamed on map causing the map to be re-run. But we can get an exception like DiskErrorException which is due to lack of disk space on the reduce node and has nothing to do with map. The map task should not be re-executed in this case.
          To solve this,I've added code to catch this DiskErrorException,blame the reduce and report this to the shuffle scheduler ,which in turn kills this reduce task and would hopefully re-run it on another node which has disk space.
          I manually tested this by changing the source and stubbing it to reserve 1 Peta-byte during the shuffle phase. This throws the DiskErrorException as expected,blames the reduce and most importantly doesn't run the map task again.
          I've also added a unit test to simulate this scenario.

          Show
          ashwinshankar77 Ashwin Shankar added a comment - The issue here is was that any IOException that arises from Fetcher.copyMapOutput is always blamed on map causing the map to be re-run. But we can get an exception like DiskErrorException which is due to lack of disk space on the reduce node and has nothing to do with map. The map task should not be re-executed in this case. To solve this,I've added code to catch this DiskErrorException,blame the reduce and report this to the shuffle scheduler ,which in turn kills this reduce task and would hopefully re-run it on another node which has disk space. I manually tested this by changing the source and stubbing it to reserve 1 Peta-byte during the shuffle phase. This throws the DiskErrorException as expected,blames the reduce and most importantly doesn't run the map task again. I've also added a unit test to simulate this scenario.
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          Also please note that this patch doesn't directly apply to branch-23. I'll upload a separate patch once all the comments are resolved for this current patch.

          Show
          ashwinshankar77 Ashwin Shankar added a comment - Also please note that this patch doesn't directly apply to branch-23. I'll upload a separate patch once all the comments are resolved for this current patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12587690/MAPREDUCE-5251-2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3767//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3767//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587690/MAPREDUCE-5251-2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3767//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3767//console This message is automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Ashwin. Unfortunately I didn't get to it in time, and it's gone stale. Could you please refresh it? Couple of comments on the existing patch:

          • reportLocalDiskError shouldn't assume that the disk error is due to lack of space. If DiskErrorException is ever thrown for other reasons or future code calls reportLocalDiskError for other kinds of errors, the log message could be very misleading to a user. Probably best to simply report a disk error let the exception message/traceback do most of the talking about specifics.
          • Do we want to only catch DiskErrorException when trying to reserve space for a map output? Other IOExceptions will also cause us to blame the map when the map is unlikely to be the problem. Seems like we want to report a local error for any IOException and retry (i.e.: kill) the reducer in those cases. In that sense maybe reportLocalDiskError should just be reportLocalError.
          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Ashwin. Unfortunately I didn't get to it in time, and it's gone stale. Could you please refresh it? Couple of comments on the existing patch: reportLocalDiskError shouldn't assume that the disk error is due to lack of space. If DiskErrorException is ever thrown for other reasons or future code calls reportLocalDiskError for other kinds of errors, the log message could be very misleading to a user. Probably best to simply report a disk error let the exception message/traceback do most of the talking about specifics. Do we want to only catch DiskErrorException when trying to reserve space for a map output? Other IOExceptions will also cause us to blame the map when the map is unlikely to be the problem. Seems like we want to report a local error for any IOException and retry (i.e.: kill) the reducer in those cases. In that sense maybe reportLocalDiskError should just be reportLocalError.
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          Makes sense,thanks for your comments.
          I've made the suggested changes.

          Show
          ashwinshankar77 Ashwin Shankar added a comment - Makes sense,thanks for your comments. I've made the suggested changes.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12589962/MAPREDUCE-5251-3.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3804//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3804//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12589962/MAPREDUCE-5251-3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3804//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3804//console This message is automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          +1, lgtm. One minor nit: the body of the new try block isn't indented which I'll fix on checkin.

          Show
          jlowe Jason Lowe added a comment - +1, lgtm. One minor nit: the body of the new try block isn't indented which I'll fix on checkin.
          Hide
          jlowe Jason Lowe added a comment -

          Actually on second thought, we really need a way to report the actual error being encountered. Otherwise an error will occur and the logs won't have any traceback or message indicating what the nature of the error is.

          reportLocalError should probably take a Throwable argument so we can preserve the error information when we report it. reportLocalError can then just pass that exception on to the exception reporter.

          Show
          jlowe Jason Lowe added a comment - Actually on second thought, we really need a way to report the actual error being encountered. Otherwise an error will occur and the logs won't have any traceback or message indicating what the nature of the error is. reportLocalError should probably take a Throwable argument so we can preserve the error information when we report it. reportLocalError can then just pass that exception on to the exception reporter.
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          Thanks,patch updated !

          Show
          ashwinshankar77 Ashwin Shankar added a comment - Thanks,patch updated !
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12593044/MAPREDUCE-5251-4.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3881//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3881//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12593044/MAPREDUCE-5251-4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3881//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3881//console This message is automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the update, Aswhin. Couple of minor things:

          • reportLocalError probably should just compute the hostname itself rather than requiring callers to do so
          • there is whitespace missing between arguments added in the latest patch (which will be fixed if we remove the reduceHost arg to reportLocalError)
          Show
          jlowe Jason Lowe added a comment - Thanks for the update, Aswhin. Couple of minor things: reportLocalError probably should just compute the hostname itself rather than requiring callers to do so there is whitespace missing between arguments added in the latest patch (which will be fixed if we remove the reduceHost arg to reportLocalError)
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          Thanks,patch updated.

          Show
          ashwinshankar77 Ashwin Shankar added a comment - Thanks,patch updated.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12593765/MAPREDUCE-5251-5.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3883//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3883//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12593765/MAPREDUCE-5251-5.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3883//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3883//console This message is automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          I see reportLocalError is now throwing UnknownHostException. Unfortunately since that is an IOException, if it ever does do that it will end up catching that in the outer try-catch block in copyMapOutput and a map attempt wil be blamed for it.

          Also now that I think of it, we arguably should be incrementing the ioErrs counter before calling reportLocalError since this is an I/O error during the shuffle that prevented a successful map output transfer.

          Show
          jlowe Jason Lowe added a comment - I see reportLocalError is now throwing UnknownHostException. Unfortunately since that is an IOException, if it ever does do that it will end up catching that in the outer try-catch block in copyMapOutput and a map attempt wil be blamed for it. Also now that I think of it, we arguably should be incrementing the ioErrs counter before calling reportLocalError since this is an I/O error during the shuffle that prevented a successful map output transfer.
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          Makes sense,both the comments addressed in latest patch.

          Show
          ashwinshankar77 Ashwin Shankar added a comment - Makes sense,both the comments addressed in latest patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12593790/MAPREDUCE-5251-6.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3886//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3886//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12593790/MAPREDUCE-5251-6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3886//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3886//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12593790/MAPREDUCE-5251-6.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3891//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3891//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12593790/MAPREDUCE-5251-6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3891//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3891//console This message is automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          THanks Ashwin. I think the patch is almost there, but I noticed that when we log an error we don't say anything at all about the error itself – we probably should at least log the .getMessage() of the error if we're going to bother logging there was an error. Also the handling of the unknown host error text is somewhat misleading – one could interpret the "unknown" referring to the local error that occurred rather than the fact it couldn't lookup the node name.

          Show
          jlowe Jason Lowe added a comment - THanks Ashwin. I think the patch is almost there, but I noticed that when we log an error we don't say anything at all about the error itself – we probably should at least log the .getMessage() of the error if we're going to bother logging there was an error. Also the handling of the unknown host error text is somewhat misleading – one could interpret the "unknown" referring to the local error that occurred rather than the fact it couldn't lookup the node name.
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          Thanks,patch refreshed.

          Show
          ashwinshankar77 Ashwin Shankar added a comment - Thanks,patch refreshed.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594228/MAPREDUCE-5251-7.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3900//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3900//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594228/MAPREDUCE-5251-7.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3900//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3900//console This message is automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          +1

          Show
          jlowe Jason Lowe added a comment - +1
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Ashwin! I committed to trunk and branch-2. Could you provide a patch for branch-0.23?

          Show
          jlowe Jason Lowe added a comment - Thanks, Ashwin! I committed to trunk and branch-2. Could you provide a patch for branch-0.23?
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          Thanks a lot Jason. I've attached the patch for 23.

          Show
          ashwinshankar77 Ashwin Shankar added a comment - Thanks a lot Jason. I've attached the patch for 23.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594411/MAPREDUCE-5251-7-b23.txt
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3907//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594411/MAPREDUCE-5251-7-b23.txt against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3907//console This message is automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          +1 to the branch-0.23 patch and committed to branch-0.23.

          Show
          jlowe Jason Lowe added a comment - +1 to the branch-0.23 patch and committed to branch-0.23.
          Hide
          jlowe Jason Lowe added a comment -

          I pulled this into branch-2.1-beta as well.

          Show
          jlowe Jason Lowe added a comment - I pulled this into branch-2.1-beta as well.

            People

            • Assignee:
              ashwinshankar77 Ashwin Shankar
              Reporter:
              jlowe Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development