Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9505

Specifying checksum type to NULL can cause write failures with AIOBE

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None

      Description

      I have created a file with checksum disable option and I am seeing ArrayIndexOutOfBoundsException.

      out = fs.create(fileName, FsPermission.getDefault(), flags, fs.getConf()
      	  .getInt("io.file.buffer.size", 4096), replFactor, fs
      	  .getDefaultBlockSize(fileName), null, ChecksumOpt.createDisabled());
      

      See the trace here:

      java.lang.ArrayIndexOutOfBoundsException: 0
      	at org.apache.hadoop.fs.FSOutputSummer.int2byte(FSOutputSummer.java:178)
      	at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:162)
      	at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:106)
      	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:92)
      	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:54)
      	at java.io.DataOutputStream.write(DataOutputStream.java:90)
      	at org.apache.hadoop.hdfs.DFSTestUtil.createFile(DFSTestUtil.java:261)
      	at org.apache.hadoop.hdfs.TestReplication.testBadBlockReportOnTransfer(TestReplication.java:174)
      

      In FSOutputSummer#int2byte will not check any bytes length, so, do you think we have to to check the length then only we call this in CRC NULL case, as there will not be any checksum bytes?

      static byte[] int2byte(int integer, byte[] bytes) {
          bytes[0] = (byte)((integer >>> 24) & 0xFF);
          bytes[1] = (byte)((integer >>> 16) & 0xFF);
          bytes[2] = (byte)((integer >>>  8) & 0xFF);
          bytes[3] = (byte)((integer >>>  0) & 0xFF);
          return bytes;
        }
      

        Attachments

        1. HADOOP-9505.patch
          0.7 kB
          Vinayakumar B

          Issue Links

            Activity

              People

              • Assignee:
                vinayakumarb Vinayakumar B
                Reporter:
                umamaheswararao Uma Maheswara Rao G
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: