Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5761

DataNode fails to validate integrity for checksum type NULL when DataNode recovers

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:

      Description

      When DataNode is down during writing blocks, the blocks are not filinalized and the next time DataNode recovers, integrity validation will run.
      But if we use NULL for checksum algorithm (we can set NULL to dfs.checksum.type), DataNode will fail to validate integrity and cannot be up.

      The cause is in BlockPoolSlice#validateIntegrity.
      In the method, there is following code.

      long numChunks = Math.min(
                (blockFileLen + bytesPerChecksum - 1)/bytesPerChecksum, 
                (metaFileLen - crcHeaderLen)/checksumSize);
      

      When we choose NULL checksum, checksumSize is 0 so ArithmeticException will be thrown and DataNode cannot be up.

        Attachments

        1. HDFS-5761.patch
          1 kB
          Kousuke Saruta

          Activity

            People

            • Assignee:
              sarutak Kousuke Saruta
              Reporter:
              sarutak Kousuke Saruta
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: