[HDFS-13098] RBF: Datanodes interacting with Routers

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Íñigo Goiri
• Created: 31/Jan/18 23:23
• Updated: 15/May/18 00:22

Description

Datanodes talk to particular Namenodes. We could use the Router infrastructure for the Datanodes to register/heartbeating into them and the Routers would forward this to particular Namenodes. This would make the assignment of Datanodes to subclusters potentially more dynamic.

The implementation would potentially include:

• Router to implement part of DatanodeProtocol
• Forwarding DN messages into Routers
• Policies to assign datanodes to subclusters
• Datanodes to make blockpool configuration dynamic

Issue Links

• depends upon: HDFS-13312 NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
• is related to: HDFS-12615 Router-based HDFS federation phase 2