Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13758

DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
    • namenode
    • None

    Description

      In Hadoop 3, HDFS-8909 added an assertion assumption that if a BlockRecoveryCommand exists for a block, the block is under construction.

       

      DatanodeManager#getBlockRecoveryCommand()
      
        BlockRecoveryCommand brCommand = new BlockRecoveryCommand(blocks.length);
        for (BlockInfo b : blocks) {
          BlockUnderConstructionFeature uc = b.getUnderConstructionFeature();
          assert uc != null;
      ...
      

      This assertion accidentally fixed one of the possible scenario of HDFS-10240 data corruption, if a recoverLease() is made immediately followed by a close(), before DataNodes have the chance to heartbeat.

      In a unit test you'll get:

      2018-07-19 09:43:41,331 [IPC Server handler 9 on 57890] WARN  ipc.Server (Server.java:logException(2724)) - IPC Server handler 9 on 57890, call Call#41 Retry#0 org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.sendHeartbeat from 127.0.0.1:57903
      java.lang.AssertionError
      	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getBlockRecoveryCommand(DatanodeManager.java:1551)
      	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleHeartbeat(DatanodeManager.java:1661)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:3865)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendHeartbeat(NameNodeRpcServer.java:1504)
      	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.sendHeartbeat(DatanodeProtocolServerSideTranslatorPB.java:119)
      	at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:31660)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1689)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
      

      I propose to change this assertion even though it address the data corruption, because:

      1. We should throw an more meaningful exception than an NPE
      2. on a production cluster, the assert is ignored, and you'll get a more noticeable NPE. Future HDFS developers might fix this NPE, causing regression. An NPE is typically not captured and handled, so there's a chance to result in internal state inconsistency.
      3. It doesn't address all possible scenarios of HDFS-10240. A proper fix should reject close() if the block is being recovered.

      Attachments

        1. HDFS-10240 scenarios.jpg
          48 kB
          Wei-Chiu Chuang
        2. HDFS-13758.001.patch
          1 kB
          chencan
        3. HDFS-13758.branch-2.patch
          1 kB
          chencan

        Activity

          People

            candychencan chencan
            weichiu Wei-Chiu Chuang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: