[HDFS-5946] Webhdfs DN choosing code is flawed

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.4.0, 3.0.0-alpha1
• Component/s: namenode, webhdfs
• Assignee: Unassigned
• Reporter: Daryn Sharp
• Created: 13/Feb/14 15:43
• Updated: 05/Jun/16 05:59

Description

HDFS-5891 improved the performance of redirecting webhdfs clients to a DN. Instead of attempting a connection with a 1-minute timeout, the NN skips decommissioned nodes.

The logic appears flawed. It finds the index of the first decommissioned node, if any, then:

• Throws an exception if index = 0, even if other nodes later in the list are not decommissioned.
• Else picks a random node prior to the index. Let's say there are 10 replicas, 2nd location is decommissioned. All clients will be redirected to the first location even though there are 8 other valid locations.

Issue Links

• is related to: HDFS-5891 webhdfs should not try connecting the DN during redirection