|
Attaching the working patch, while I continue to test.
fairly tested patch. few additional things:
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389656/3829_v3.patch  against trunk revision 692996. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3205/testReport/ This message is automatically generated. Fixed an issue in ReduceTask#SkippingReduceValuesIterator.
Added more documentation to SkipBadRecords. Made writing of skip records optional. Incorporated Devaraj's offline comment to remove the SkipBadRecords.ENABLED flag as it is now redundant after the addition of MAPPER_MAX_SKIP_RECORDS/REDUCER_MAX_SKIP_GROUPS. test-patch passed on my machine.
[exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 11 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. ant test passed on my machine.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390255/3829_v4.patch against trunk revision 696525. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3293/testReport/ This message is automatically generated. TestFileAppend2.testComplexAppend has failed on Hudson which is unrelated to this patch.
I just committed this. Thanks, Sharad!
Integrated in Hadoop-trunk #611 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/
) |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
New Feature
Closed
Major
Narrown down skipped records based on user acceptable value
HADOOP-153. Please apply 153_7.patch before applying this one.The approach has been discussed earlier in
HADOOP-153as well. Here is the brief:Defines user configurable MAPPER_MAX_SKIP_RECORDS/MAPPER_REDUCE_SKIP_RECORDS -> acceptable skipped records in the neighborhood of a bad record.
If skipped range is greater than this threshold, the task will try to narrow down the skipped range using a binary search kind of algorithm during task re-executions till this threshold is met or all task attempts are exhausted. The skipped range is divided into two halves and only one half get executed. Based on the subsequent failure, it figures out which half contains the bad record.