Hadoop Common
  1. Hadoop Common
  2. HADOOP-9665

BlockDecompressorStream#decompress will throw EOFException instead of return -1 when EOF

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.2, 2.1.0-beta, 2.3.0
    • Fix Version/s: 1-win, 2.1.0-beta, 1.2.1
    • Component/s: None
    • Labels:
      None

      Description

      BlockDecompressorStream#decompress ultimately calls rawReadInt, which will throw EOFException instead of return -1 when encountering end of a stream. Then, decompress will be called by read. However, InputStream#read is supposed to return -1 instead of throwing EOFException to indicate the end of a stream. This explains why in LineReader,

            if (bufferPosn >= bufferLength) {
              startPosn = bufferPosn = 0;
              if (prevCharCR)
                ++bytesConsumed; //account for CR from previous read
              bufferLength = in.read(buffer);
              if (bufferLength <= 0)
                break; // EOF
            }
      

      -1 is checked instead of catching EOFException.

      Now the problem will occur with SnappyCodec. If an input file is compressed with SnappyCodec, it needs to be decompressed through BlockDecompressorStream when it is read. Then, if it empty, EOFException will been thrown from rawReadInt and break LineReader.

      1. HADOOP-9665-branch-1.1.patch
        7 kB
        Zhijie Shen
      2. HADOOP-9665.2.patch
        3 kB
        Zhijie Shen
      3. HADOOP-9665.1.patch
        0.9 kB
        Zhijie Shen

        Issue Links

          Activity

          Zhijie Shen created issue -
          Zhijie Shen made changes -
          Field Original Value New Value
          Project Hadoop YARN [ 12313722 ] Hadoop Common [ 12310240 ]
          Key YARN-872 HADOOP-9665
          Zhijie Shen made changes -
          Link This issue is related to HADOOP-9658 [ HADOOP-9658 ]
          Zhijie Shen made changes -
          Affects Version/s 1.1.2 [ 12323596 ]
          Affects Version/s 2.1.0-beta [ 12324030 ]
          Affects Version/s 2.2.0 [ 12324637 ]
          Zhijie Shen made changes -
          Attachment HADOOP-9665.1.patch [ 12589156 ]
          Zhijie Shen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Zhijie Shen made changes -
          Attachment HADOOP-9665.2.patch [ 12589361 ]
          Zhijie Shen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Zhijie Shen made changes -
          Attachment HADOOP-9665-branch-1.1.patch [ 12590094 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.1.0-beta [ 12324030 ]
          Fix Version/s 1.2.1 [ 12324147 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Affects Version/s 2.3.0 [ 12324587 ]
          Affects Version/s 2.2.0 [ 12324637 ]
          Chris Nauroth made changes -
          Fix Version/s 1-win [ 12320361 ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Affects Version/s 2.3.0 [ 12325254 ]
          Affects Version/s 2.4.0 [ 12324587 ]

            People

            • Assignee:
              Zhijie Shen
              Reporter:
              Zhijie Shen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development