Hadoop Common
  1. Hadoop Common
  2. HADOOP-9665

BlockDecompressorStream#decompress will throw EOFException instead of return -1 when EOF

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.2, 2.1.0-beta, 2.3.0
    • Fix Version/s: 1-win, 2.1.0-beta, 1.2.1
    • Component/s: None
    • Labels:
      None

      Description

      BlockDecompressorStream#decompress ultimately calls rawReadInt, which will throw EOFException instead of return -1 when encountering end of a stream. Then, decompress will be called by read. However, InputStream#read is supposed to return -1 instead of throwing EOFException to indicate the end of a stream. This explains why in LineReader,

            if (bufferPosn >= bufferLength) {
              startPosn = bufferPosn = 0;
              if (prevCharCR)
                ++bytesConsumed; //account for CR from previous read
              bufferLength = in.read(buffer);
              if (bufferLength <= 0)
                break; // EOF
            }
      

      -1 is checked instead of catching EOFException.

      Now the problem will occur with SnappyCodec. If an input file is compressed with SnappyCodec, it needs to be decompressed through BlockDecompressorStream when it is read. Then, if it empty, EOFException will been thrown from rawReadInt and break LineReader.

      1. HADOOP-9665.1.patch
        0.9 kB
        Zhijie Shen
      2. HADOOP-9665.2.patch
        3 kB
        Zhijie Shen
      3. HADOOP-9665-branch-1.1.patch
        7 kB
        Zhijie Shen

        Issue Links

          Activity

          Hide
          Zhijie Shen added a comment -

          The patch catches EOFException in decompress and return -1 instead, preventing EOFException from being further propagated.

          Did local test by applying this patch on 1.1.2. Reading empty file compressed with SnappyCodec will no longer fail with EOFException.

          Show
          Zhijie Shen added a comment - The patch catches EOFException in decompress and return -1 instead, preventing EOFException from being further propagated. Did local test by applying this patch on 1.1.2. Reading empty file compressed with SnappyCodec will no longer fail with EOFException.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12589156/HADOOP-9665.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2695//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2695//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12589156/HADOOP-9665.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2695//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2695//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Zhijie Shen +1 the fix looks good. Can you please add a test case? Tx.

          Show
          Arun C Murthy added a comment - Zhijie Shen +1 the fix looks good. Can you please add a test case? Tx.
          Hide
          Zhijie Shen added a comment -

          Test updated.

          Show
          Zhijie Shen added a comment - Test updated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12589361/HADOOP-9665.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2699//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2699//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12589361/HADOOP-9665.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2699//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2699//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          I just committed this to hadoop-2, thanks Zhijie Shen!

          Mind providing a patch for branch-1 too?

          Show
          Arun C Murthy added a comment - I just committed this to hadoop-2, thanks Zhijie Shen ! Mind providing a patch for branch-1 too?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #4019 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4019/)
          HADOOP-9665. Fixed BlockDecompressorStream#decompress to return -1 rather than throw EOF at end of file. Contributed by Zhijie Shen. (Revision 1497922)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1497922
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BlockDecompressorStream.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #4019 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4019/ ) HADOOP-9665 . Fixed BlockDecompressorStream#decompress to return -1 rather than throw EOF at end of file. Contributed by Zhijie Shen. (Revision 1497922) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1497922 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BlockDecompressorStream.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java
          Hide
          Zhijie Shen added a comment -

          Backporting to branch-1

          Show
          Zhijie Shen added a comment - Backporting to branch-1
          Hide
          Arun C Murthy added a comment -

          I committed this to hadoop-1.2.1 too. Thanks Zhijie Shen!

          Show
          Arun C Murthy added a comment - I committed this to hadoop-1.2.1 too. Thanks Zhijie Shen !
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12590094/HADOOP-9665-branch-1.1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2710//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12590094/HADOOP-9665-branch-1.1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2710//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #255 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/255/)
          HADOOP-9665. Fixed BlockDecompressorStream#decompress to return -1 rather than throw EOF at end of file. Contributed by Zhijie Shen. (Revision 1497922)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1497922
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BlockDecompressorStream.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #255 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/255/ ) HADOOP-9665 . Fixed BlockDecompressorStream#decompress to return -1 rather than throw EOF at end of file. Contributed by Zhijie Shen. (Revision 1497922) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1497922 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BlockDecompressorStream.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1445 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1445/)
          HADOOP-9665. Fixed BlockDecompressorStream#decompress to return -1 rather than throw EOF at end of file. Contributed by Zhijie Shen. (Revision 1497922)

          Result = FAILURE
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1497922
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BlockDecompressorStream.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1445 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1445/ ) HADOOP-9665 . Fixed BlockDecompressorStream#decompress to return -1 rather than throw EOF at end of file. Contributed by Zhijie Shen. (Revision 1497922) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1497922 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BlockDecompressorStream.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1472 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1472/)
          HADOOP-9665. Fixed BlockDecompressorStream#decompress to return -1 rather than throw EOF at end of file. Contributed by Zhijie Shen. (Revision 1497922)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1497922
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BlockDecompressorStream.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1472 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1472/ ) HADOOP-9665 . Fixed BlockDecompressorStream#decompress to return -1 rather than throw EOF at end of file. Contributed by Zhijie Shen. (Revision 1497922) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1497922 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BlockDecompressorStream.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestBlockDecompressorStream.java
          Hide
          Chris Nauroth added a comment -

          I intend to merge this patch to branch-1-win later today. The branch-1 patch applies cleanly to branch-1-win.

          Show
          Chris Nauroth added a comment - I intend to merge this patch to branch-1-win later today. The branch-1 patch applies cleanly to branch-1-win.
          Hide
          Chris Nauroth added a comment -

          I've committed this to branch-1-win. Thanks again for this patch, Zhijie!

          Show
          Chris Nauroth added a comment - I've committed this to branch-1-win. Thanks again for this patch, Zhijie!

            People

            • Assignee:
              Zhijie Shen
              Reporter:
              Zhijie Shen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development