Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11526

Fix confusing block recovery message

    XMLWordPrintableJSON

    Details

      Description

      The following error message is wrong.

      BlockRecoveryWorker#recover
              } catch (IOException e) {
                ++errorCount;
                InterDatanodeProtocol.LOG.warn(
                    "Failed to obtain replica info for block (=" + block
                        + ") from datanode (=" + id + ")", e);
              }
      

      The operation performed in the try block is an attempt to recover the block, not obtain replica info from the datanode.

      This is the error message printed by the above code:

      2017-03-01 16:15:35,884 WARN org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed to obtain replica info for block (=BP-1949147302-10.0.0.140-1423905184563:blk_1074852850_1112215) from datanode (=DatanodeInfoWithStorage[10.0.0.53:50010,null,null])
      java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: replica.getGenerationStamp() >= recoveryId = 1112223, block=blk_1074852850_1112215, replica=FinalizedReplica, blk_1074852850_1112231, FINALIZED
        getNumBytes()     = 12823160
        getBytesOnDisk()  = 12823160
        getVisibleLength()= 12823160
        getVolume()       = /dfs/dn/current
        getBlockFile()    = /dfs/dn/current/BP-1949147305-10.0.0.140-1423905184563/current/finalized/subdir16/subdir243/blk_1074852850
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2318)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2277)
              at org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:2548)
              at org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolServerSideTranslatorPB.initReplicaRecovery(InterDatanodeProtocolServerSideTranslatorPB.java:55)
              at org.apache.hadoop.hdfs.protocol.proto.InterDatanodeProtocolProtos$InterDatanodeProtocolService$2.callBlockingMethod(InterDatanodeProtocolProtos.java:2983)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
      
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
              at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
              at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
              at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker.callInitReplicaRecovery(BlockRecoveryWorker.java:339)
              at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker.access$300(BlockRecoveryWorker.java:46)
              at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover(BlockRecoveryWorker.java:118)
              at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1.run(BlockRecoveryWorker.java:374)
              at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        1. HDFS-11526-branch-2.patch
          0.9 kB
          Yiqun Lin
        2. HDFS-11526.001.patch
          1 kB
          Yiqun Lin

          Activity

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              weichiu Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: