"Browse the filesystem" link pointing to a dead data-node

• Key: HADOOP-1224
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Unassigned
• Reporter: Konstantin Shvachko
• Affects Version/s: 0.12.3
• Fix Version/s: 0.13.0

Description

On the NameNode status web page "Browse the filesystem" link can point to a dead data-node.
The reason for that is that FSNamesystem.randomDataNode() selects a random node from the
list of all nodes rather then selecting among alive nodes only.

Comments

[Enis Soztutar]

This patch
1.changes randomDataNode() so that it will skip deadnodes and decommissioned nodes. Starts with a random data node and checks the data nodes sequentially until a live node is found.
2.changes the return type of the getDatanodeByIndex() from DatanodeInfo to DatanodeDescriptor.

[Hadoop QA]

+1

http://issues.apache.org/jira/secure/attachment/12355503/DFSBrowsingDeadNode_v1.0.patch applied and successfully tested against trunk revision r528230.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/43/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/43/console

[dhruba borthakur]

+1. Code looks good.

[Tom White]

I've just committed this. Thanks Enis!

[Hadoop QA]

Integrated in Hadoop-Nightly #60 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/60/)

[Konstantin Shvachko]

Now its even worse, we select ONLY! dead nodes.
if (d != null && !d.isDecommissioned() && isDatanodeDead(d) &&
!d.isDecommissionInProgress()) {
return d.getHost() + ":" + d.getInfoPort();
Did anybody ever actually tried to click the link?

[Enis Soztutar]

This patch applies to current trunk(534354). Fixes the bug in [forgotten!] negation in check in isDataNodeDead(), introduces a function isDataNodeLive()

[Hadoop QA]

+1

http://issues.apache.org/jira/secure/attachment/12356611/DFSBrowsingDeadNode_v1.1.patch applied and successfully tested against trunk revision r534234.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/100/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/100/console

[Tom White]

Thanks Enis. Have you manually tested this latest patch as Konstantin suggests?

[Enis Soztutar]

Finally, i was able to test the patch.
Manually i have set up a cluster with 2 DN and one NN.
After intentionally killing one DN, or decommisioning one DN, browsing worked as expected. Sorry for the previously untested version

[Konstantin Shvachko]

It is confusing if a data-node can be neither dead nor alive. In your patch
isDatanodeDead() =/= ! isDatanodeLive()
The patch should merely add "!" imo.

[Tom White]

> The patch should merely add "!" imo.

+1

[Enis Soztutar]

The confusing thing here, IMO, is that the admin status of the datanode, can be either NORMAL, DECOMMISIONED or DECOMMISSION_IN_PROGRESS, and if the admin state is normal, it can be either dead or "not dead". So a data node, from the perspective of the end user, can be in one of the four states : live, dead, decommissioned or decommission_in_progress. Thus isDatanodeDead() =/= ! isDatanodeLive().

[Enis Soztutar]

one line patch that adds "!" .

[Konstantin Shvachko]

+1

[Doug Cutting]

I just committed this. Thanks, Enis!

[Hadoop QA]

Integrated in Hadoop-Nightly #82 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/82/)