[HDFS-17181] WebHDFS not considering whether a DN is good when called from outside the cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.6
• Fix Version/s: 3.5.0
• Component/s: namenode, webhdfs
• Labels: pull-request-available
• Assignee: Lars Francke
• Reporter: Lars Francke
• Created: 07/Sep/23 07:46
• Updated: 20/Feb/24 19:48
• Resolved: 20/Feb/24 18:59

Description

When calling WebHDFS to create a file (I'm sure the same problem occurs for other actions e.g. OPEN but I haven't checked all of them yet) it will happily redirect to nodes that are in maintenance.

The reason is in the chooseDatanode method in NamenodeWebHdfsMethods where it will only call the BlockPlacementPolicy (which considers all these edge cases) in case the remoteAddr (i.e. the address making the request to WebHDFS) is also running a DataNode.
 
In all other cases it just refers to NetworkTopology#chooseRandom which does not consider any of these circumstances (e.g. load, maintenance).

I don't understand the reason to not just always refer to the placement policy and we're currently testing a patch to do just that.

I have attached a draft patch for now.

 

Issue Links

• links to: GitHub Pull Request #6108