[HDFS-16540] Data locality is lost when DataNode pod restarts in kubernetes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 3.4.0
• Component/s: namenode
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Huaxiang Sun
• Reporter: Huaxiang Sun
• Created: 13/Apr/22 18:08
• Updated: 29/Sep/23 17:48
• Resolved: 16/May/22 04:33

Description

We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod restarts, we found that data locality is lost after we do a major compaction of hbase regions. After some debugging, we found that upon pod restarts, its ip changes. In DatanodeManager, maps like networktopology are updated with the new info. host2DatanodeMap is not updated accordingly. When hdfs client with the new ip tries to find a local DataNode, it fails. 

 

Issue Links

• is related to: HDFS-17188 Data loss in our production clusters due to missing HDFS-16540
• links to: GitHub Pull Request #4170
• links to: GitHub Pull Request #4246
• links to: GitHub Pull Request #4326