Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5843

DFSClient.getFileChecksum() throws IOException if checksum is disabled

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If a file is created with checksum disabled (using ChecksumOpt.disabled() for example), calling FileSystem.getFileChecksum() throws the following IOException:

      java.io.IOException: Fail to get block MD5 for BP-341493254-192.168.1.10-1390888724459:blk_1073741825_1001
      	at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:1965)
      	at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:1771)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1186)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1194)
      [...]
      

      From the logs, the datanode is doing some wrong arithmetics because of the crcPerBlock:

      2014-01-27 21:58:46,329 ERROR datanode.DataNode (DataXceiver.java:run(225)) - 127.0.0.1:52398:DataXceiver error processing BLOCK_CHECKSUM operation  src: /127.0.0.1:52407 dest: /127.0.0.1:52398
      java.lang.ArithmeticException: / by zero
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.blockChecksum(DataXceiver.java:658)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opBlockChecksum(Receiver.java:169)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:77)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
      	at java.lang.Thread.run(Thread.java:695)
      

        Attachments

        1. hdfs-5843.patch
          3 kB
          Laurent Goujon

          Activity

            People

            • Assignee:
              laurent Laurent Goujon
              Reporter:
              laurent Laurent Goujon
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: