[HDFS-8586] Dead Datanode is allocated for write when client is from deadnode

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Brahma Reddy Battula
• Reporter: Brahma Reddy Battula
• Created: 12/Jun/15 08:26
• Updated: 30/Aug/16 01:32
• Resolved: 29/Jun/15 09:57

Description

DataNode marked as Dead
2015-06-11 19:39:00,862 | INFO | org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@28ec166e | BLOCK* removeDeadDatanode: lost heartbeat from XX.XX.39.33:25009 | org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:584)
2015-06-11 19:39:00,863 | INFO | org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@28ec166e | Removing a node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:488)

Deadnode got Allocated
2015-06-11 19:39:45,148 | WARN | IPC Server handler 26 on 25000 | The cluster does not contain node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
2015-06-11 19:39:45,149 | WARN | IPC Server handler 26 on 25000 | The cluster does not contain node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
2015-06-11 19:39:45,149 | WARN | IPC Server handler 26 on 25000 | The cluster does not contain node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
2015-06-11 19:39:45,149 | WARN | IPC Server handler 26 on 25000 | The cluster does not contain node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
2015-06-11 19:39:45,149 | INFO | IPC Server handler 26 on 25000 | BLOCK* allocate blk_1073754030_13252 {UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e8d29773-dfc2-4224-b1d6-9b0588bca55e:NORMAL:XX.XX.39.33:25009|RBW], ReplicaUC[[DISK]DS-f7d2ab3c-88f7-470c-9097-84387c0bec83:NORMAL:XX.XX.38.32:25009|RBW], ReplicaUC[[DISK]DS-8c2a464a-ac81-4651-890a-dbfd07ddd95f:NORMAL: XX.XX.38.33:25009|RBW]] } for /t1.COPYING | org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3657)
2015-06-11 19:39:45,191 | INFO | IPC Server handler 35 on 25000 | BLOCK* allocate blk_1073754031_13253{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ed8ad579-50c0-4e3e-8780-9776531763b6:NORMAL:XX.XX.39.31:25009|RBW], ReplicaUC[[DISK]DS-19ddd6da-4a3e-481a-8445-dde5c90aaff3:NORMAL:XX.XX.37.32:25009|RBW], ReplicaUC[[DISK]DS-4ce4ce39-4973-42ce-8c7d-cb41f899db85: NORMAL:XX.XX.37.33:25009 |RBW]]} for /t1.COPYING | org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3657)