Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-946

Fix regression in LineRecordReader to comply with line length parameters

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MAPREDUCE-773 accidentally changed code introduced in HADOOP-3144 controlling max line lengths. The behavior should be restored.

      1. M946-1.patch
        6 kB
        Chris Douglas
      2. M946-0.patch
        5 kB
        Chris Douglas

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/85/)
        . Fix a regression in LineRecordReader where the
        maxBytesToConsume parameter is not set correctly.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/85/ ) . Fix a regression in LineRecordReader where the maxBytesToConsume parameter is not set correctly.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #43 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/43/)
        . Fix a regression in LineRecordReader where the
        maxBytesToConsume parameter is not set correctly.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #43 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/43/ ) . Fix a regression in LineRecordReader where the maxBytesToConsume parameter is not set correctly.
        Hide
        Chris Douglas added a comment -

        I committed this.

        Show
        Chris Douglas added a comment - I committed this.
        Hide
        Hong Tang added a comment -

        patch looks good. +1.

        Show
        Hong Tang added a comment - patch looks good. +1.
        Hide
        Chris Douglas added a comment -

        The test failure, TestQueueManager.testUserDisabledACLForJobKill, is not related

        Show
        Chris Douglas added a comment - The test failure, TestQueueManager.testUserDisabledACLForJobKill, is not related
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12419496/M946-1.patch
        against trunk revision 814467.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/73/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/73/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419496/M946-1.patch against trunk revision 814467. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/73/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/73/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        Restored the previous behavior for uncompressed data and updated to use maxint for bytesconsumed from compressed streams. One could update LineReader to pass a Seekable to track bytes consumed from the underlying stream, which would fit the contract for HADOOP-3144 fairly well. However, this would only be effective when one read compressed, corrupt text data using a splittable codec. This should be sufficient.

        Show
        Chris Douglas added a comment - Restored the previous behavior for uncompressed data and updated to use maxint for bytesconsumed from compressed streams. One could update LineReader to pass a Seekable to track bytes consumed from the underlying stream, which would fit the contract for HADOOP-3144 fairly well. However, this would only be effective when one read compressed, corrupt text data using a splittable codec. This should be sufficient.
        Hide
        Chris Douglas added a comment -

        Hong points out that this patch offers the max bytes consumed in compressed bytes to the LineReader, which tracks uncompressed bytes. If this is to work for splittable codecs per MAPREDUCE-830 and maintain progress as in MAPREDUCE-773, we'll need to come up with a better solution.

        Show
        Chris Douglas added a comment - Hong points out that this patch offers the max bytes consumed in compressed bytes to the LineReader, which tracks uncompressed bytes. If this is to work for splittable codecs per MAPREDUCE-830 and maintain progress as in MAPREDUCE-773 , we'll need to come up with a better solution.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12419279/M946-0.patch
        against trunk revision 813660.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/63/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/63/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/63/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/63/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419279/M946-0.patch against trunk revision 813660. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/63/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/63/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/63/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/63/console This message is automatically generated.

          People

          • Assignee:
            Chris Douglas
            Reporter:
            Chris Douglas
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development