Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5761

DataNode fails to validate integrity for checksum type NULL when DataNode recovers

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:

      Description

      When DataNode is down during writing blocks, the blocks are not filinalized and the next time DataNode recovers, integrity validation will run.
      But if we use NULL for checksum algorithm (we can set NULL to dfs.checksum.type), DataNode will fail to validate integrity and cannot be up.

      The cause is in BlockPoolSlice#validateIntegrity.
      In the method, there is following code.

      long numChunks = Math.min(
                (blockFileLen + bytesPerChecksum - 1)/bytesPerChecksum, 
                (metaFileLen - crcHeaderLen)/checksumSize);
      

      When we choose NULL checksum, checksumSize is 0 so ArithmeticException will be thrown and DataNode cannot be up.

      1. HDFS-5761.patch
        1 kB
        Kousuke Saruta

        Activity

        Allen Wittenauer made changes -
        Labels BB2015-05-TBR
        Kousuke Saruta made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Kousuke Saruta made changes -
        Attachment HDFS-5761.patch [ 12622648 ]
        Kousuke Saruta made changes -
        Field Original Value New Value
        Summary DataNode fail to validate integrity for checksum type NULL when DataNode recovers DataNode fails to validate integrity for checksum type NULL when DataNode recovers
        Kousuke Saruta created issue -

          People

          • Assignee:
            Kousuke Saruta
            Reporter:
            Kousuke Saruta
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development