Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8208

Namenode reports stale block locations during decommissioning

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.5.0
    • None
    • namenode
    • None

    Description

      While decommissioning some datanodes, we observed OP_READ_BLOCK failures from DFSClient instances. Tracing back the history in the logs reveals that the DFSClient instance was started after the block in question had been migrated off the DataNode that was being accessed at the time of the OP_READ_BLOCK failure. It appears that the NameNode, despite having orchestrated the migration of the data, continued to provide a DataNode as a valid source of a block, despite it no longer being present.

      Timeline:

      The DN deleted the block at 09:43:20.
      The DFSClient instance was started at 09:46:33.

      DFS Client Error was:

      15/03/25 09:46:35 WARN hdfs.DFSClient: Failed to connect to /IP.PARTLY.MASKED.192:50010 for block, add to deadNodes and continue. java.io.IOException: Got error for OP_READ_BLOCK, self=/IP.PARTLY.MASKED.24:37956, remote=/IP.PARTLY.MASKED.192:50010, for file /bundles/bundle-15edf17968d34583a7d24b30fb7e73c6, for pool BP-2091375182-IP.PARTLY.MASKED.219-1423771577555 block 1078309397_4569046
      java.io.IOException: Got error for OP_READ_BLOCK, self=/IP.PARTLY.MASKED.24:37956, remote=/IP.PARTLY.MASKED.192:50010, for file /bundles/bundle-15edf17968d34583a7d24b30fb7e73c6, for pool BP-2091375182-17.220.247.219-1423771577555 block 1078309397_4569046
      at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:445)
      at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:410)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:787)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:666)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:326)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:570)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:793)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840)
      at java.io.DataInputStream.read(DataInputStream.java:100)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:84)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
      at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:395)
      at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:327)
      at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:303)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:243)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:228)
      at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:306)
      at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:223)
      at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
      at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:200)
      at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
      at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

      Here's the log from the DataNode, for that block:

      grep 1078309397_4569046 hdfs_datanode.out
      block =BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, newGs=0, bytesRcvd=[0, 0]
      15/03/24 12:54:12 INFO datanode.DataNode: Receiving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 src: /17.219.209.92:49184 dest: /IP.PARTLY.MASKED4.192:50010
      15/03/24 12:54:12 DEBUG datanode.DataNode: BlockReceiver: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/24 12:54:12 DEBUG datanode.DataNode: writeTo metafile is /Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/rbw/blk_1078309397_4569046.meta of size 0
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=516 header data: offsetInBlock: 0
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=107, lastPacketInBlock=false, offsetInBlock=512, ackEnqueueNanoTime=1427226852662586000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 107 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 473000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 107 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 932000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 512
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=108, lastPacketInBlock=false, offsetInBlock=65536, ackEnqueueNanoTime=1427226852670673000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 65536
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=109, lastPacketInBlock=false, offsetInBlock=130560, ackEnqueueNanoTime=1427226852672795000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 108 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1820000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 108 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 3791000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 130560
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=110, lastPacketInBlock=false, offsetInBlock=195584, ackEnqueueNanoTime=1427226852674869000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 109 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1105000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 109 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2186000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 195584
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=111, lastPacketInBlock=false, offsetInBlock=260608, ackEnqueueNanoTime=1427226852676604000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 110 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1361000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 110 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2484000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 260608
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=112, lastPacketInBlock=false, offsetInBlock=325632, ackEnqueueNanoTime=1427226852678501000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 111 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1136000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 111 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2120000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 325632
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=113, lastPacketInBlock=false, offsetInBlock=390656, ackEnqueueNanoTime=1427226852679889000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 112 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1157000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 112 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2118000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 390656
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=114, lastPacketInBlock=false, offsetInBlock=455680, ackEnqueueNanoTime=1427226852681499000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 113 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1146000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 113 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2219000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 455680
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=115, lastPacketInBlock=false, offsetInBlock=520704, ackEnqueueNanoTime=1427226852683270000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 114 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1139000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 114 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2084000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 520704
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=116, lastPacketInBlock=false, offsetInBlock=585728, ackEnqueueNanoTime=1427226852684984000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 115 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1248000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 115 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2329000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 585728
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=117, lastPacketInBlock=false, offsetInBlock=650752, ackEnqueueNanoTime=1427226852686598000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 116 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1192000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 116 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2234000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 650752
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=118, lastPacketInBlock=false, offsetInBlock=715776, ackEnqueueNanoTime=1427226852687973000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 117 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1168000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 117 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2150000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 715776
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=119, lastPacketInBlock=false, offsetInBlock=780800, ackEnqueueNanoTime=1427226852689858000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 118 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1149000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 118 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2223000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 780800
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=120, lastPacketInBlock=false, offsetInBlock=845824, ackEnqueueNanoTime=1427226852691199000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 119 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1197000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 119 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2194000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 845824
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=121, lastPacketInBlock=false, offsetInBlock=910848, ackEnqueueNanoTime=1427226852693079000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 120 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1171000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 120 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2215000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 910848
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=122, lastPacketInBlock=false, offsetInBlock=975872, ackEnqueueNanoTime=1427226852694657000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 121 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1190000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 121 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2189000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 975872
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=123, lastPacketInBlock=false, offsetInBlock=1040896, ackEnqueueNanoTime=1427226852696164000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 122 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1223000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 122 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2318000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1040896
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=124, lastPacketInBlock=false, offsetInBlock=1105920, ackEnqueueNanoTime=1427226852697883000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 123 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1041000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 123 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2090000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1105920
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=125, lastPacketInBlock=false, offsetInBlock=1170944, ackEnqueueNanoTime=1427226852699500000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 124 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1341000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 124 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2489000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1170944
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=126, lastPacketInBlock=false, offsetInBlock=1235968, ackEnqueueNanoTime=1427226852701121000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 125 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1240000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 125 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2355000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1235968
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=127, lastPacketInBlock=false, offsetInBlock=1300992, ackEnqueueNanoTime=1427226852702358000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 126 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1171000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 126 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2209000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1300992
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=128, lastPacketInBlock=false, offsetInBlock=1366016, ackEnqueueNanoTime=1427226852704072000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 127 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1328000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 127 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2333000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1366016
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=129, lastPacketInBlock=false, offsetInBlock=1431040, ackEnqueueNanoTime=1427226852705571000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 128 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1217000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 128 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2230000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1431040
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=130, lastPacketInBlock=false, offsetInBlock=1496064, ackEnqueueNanoTime=1427226852707314000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 129 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1305000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 129 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2349000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1496064
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=131, lastPacketInBlock=false, offsetInBlock=1561088, ackEnqueueNanoTime=1427226852708544000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 130 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1276000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 130 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2294000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1561088
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=132, lastPacketInBlock=false, offsetInBlock=1626112, ackEnqueueNanoTime=1427226852710484000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 131 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1251000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 131 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2323000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1626112
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=133, lastPacketInBlock=false, offsetInBlock=1691136, ackEnqueueNanoTime=1427226852711771000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 132 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1223000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 132 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2278000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1691136
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=134, lastPacketInBlock=false, offsetInBlock=1756160, ackEnqueueNanoTime=1427226852713759000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 133 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1131000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 133 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2237000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1756160
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=135, lastPacketInBlock=false, offsetInBlock=1821184, ackEnqueueNanoTime=1427226852715109000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 134 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1170000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 134 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2187000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1821184
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=136, lastPacketInBlock=false, offsetInBlock=1886208, ackEnqueueNanoTime=1427226852716703000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 135 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1227000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 135 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2311000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1886208
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=137, lastPacketInBlock=false, offsetInBlock=1951232, ackEnqueueNanoTime=1427226852718074000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 136 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1153000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 136 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2204000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 1951232
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=138, lastPacketInBlock=false, offsetInBlock=2016256, ackEnqueueNanoTime=1427226852719449000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 137 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1223000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 137 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2195000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=65532 header data: offsetInBlock: 2016256
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=139, lastPacketInBlock=false, offsetInBlock=2081280, ackEnqueueNanoTime=1427226852721161000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=15480 header data: offsetInBlock: 2081280
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=140, lastPacketInBlock=false, offsetInBlock=2096640, ackEnqueueNanoTime=1427226852721337000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=516 header data: offsetInBlock: 2096640
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=141, lastPacketInBlock=false, offsetInBlock=2097152, ackEnqueueNanoTime=1427226852721426000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 138 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1202000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 138 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2266000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 139 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1152000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 139 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2239000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 140 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1108000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 140 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2178000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 141 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1149000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 141 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 2319000
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: PacketHeader with packetLen=0 header data: offsetInBlock: 2097152
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=142, lastPacketInBlock=true, offsetInBlock=2097152, ackEnqueueNanoTime=1427226852725564000, ackStatus=SUCCESS)
      15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving an empty packet or the end of the block BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 142 status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 894000
      15/03/24 12:54:12 DEBUG impl.FsDatasetImpl: addBlock: Moved /Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/rbw/blk_1078309397_4569046.meta to /Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/finalized/subdir28/subdir45/blk_1078309397_4569046.meta and /Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/rbw/blk_1078309397 to /Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/finalized/subdir28/subdir45/blk_1078309397
      15/03/24 12:54:12 INFO DataNode.clienttrace: src: /17.219.209.92:49184, dest: /IP.PARTLY.MASKED4.192:50010, bytes: 2097152, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1880564922_1, offset: 0, srvID: ece6163a-705c-4373-ad4f-72e8ee993e99, blockid: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, duration: 67061000
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 142 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1903000
      15/03/24 12:54:12 INFO datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
      15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, type=HAS_DOWNSTREAM_IN_PIPELINE: closing
      15/03/24 12:54:17 DEBUG datanode.DataNode: block=BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, replica=FinalizedReplica, blk_1078309397_4569046, FINALIZED
      15/03/24 12:54:17 DEBUG datanode.DataNode: replica=FinalizedReplica, blk_1078309397_4569046, FINALIZED
      15/03/24 12:54:19 INFO datanode.BlockPoolSliceScanner: Verification succeeded for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/24 17:01:25 INFO datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0) Starting thread to transfer BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to IP.PARTLY.MASKED5.251:50010 IP.PARTLY.MASKED6.216:50010 IP.PARTLY.MASKED6.215:50010
      15/03/24 17:01:25 DEBUG datanode.DataNode: block=BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, replica=FinalizedReplica, blk_1078309397_4569046, FINALIZED
      15/03/24 17:01:25 DEBUG datanode.DataNode: replica=FinalizedReplica, blk_1078309397_4569046, FINALIZED
      15/03/24 17:01:25 INFO datanode.DataNode: DataTransfer: Transmitted BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 (numBytes=2097152) to /IP.PARTLY.MASKED5.251:50010
      15/03/25 09:43:20 INFO impl.FsDatasetAsyncDiskService: Scheduling blk_1078309397_4569046 file /Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/finalized/subdir28/subdir45/blk_1078309397 for deletion
      15/03/25 09:43:20 INFO impl.FsDatasetAsyncDiskService: Deleted BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555 blk_1078309397_4569046 file /Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/finalized/subdir28/subdir45/blk_1078309397
      15/03/25 09:43:57 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:43:57 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.227:57872
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:44:36 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:44:36 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.227:57905
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:15 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:15 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.227:57924
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:54 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:54 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.227:57966
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:57 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:57 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.40:58165
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:58 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:58 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.87:59400
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:58 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:58 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.62:42339
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:58 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:58 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.100:54148
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:59 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:45:59 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.104:42736
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:00 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:00 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.51:41096
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:01 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:01 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.247:36436
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:01 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:01 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.54:56894
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:02 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:02 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.161:60278
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:31 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:31 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.254:50599
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:31 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:31 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.94:35042
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:31 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:31 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.135:65058
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:32 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:32 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.148:40964
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:33 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:33 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.55:33231
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:33 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:33 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.227:58010
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:33 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:33 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.197:46999
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:34 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:34 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.164:61581
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:34 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:34 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.93:63000
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:34 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:34 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.37:55288
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.229:50147
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.24:37956
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.141:57225
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.87:50226
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.65:64743
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.234:51010
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:35 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.100:38300
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:36 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:36 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.60:36632
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:36 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:36 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.224:52901
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:36 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:36 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.40:58191
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:37 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:37 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.62:42366
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:37 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:37 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.87:59427
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:37 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:37 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.100:54175
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:38 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:38 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.104:42766
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:39 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:39 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.51:41153
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:40 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:40 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.54:56921
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:41 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:41 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED6.161:60304
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:41 INFO datanode.DataNode: opReadBlock BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      15/03/25 09:46:41 WARN datanode.DataNode: DatanodeRegistration(IP.PARTLY.MASKED4.192, datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got exception while serving BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to /IP.PARTLY.MASKED5.247:36503
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046

      Attachments

        Activity

          People

            Unassigned Unassigned
            jamesmulcahy James Mulcahy
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated: