[HDFS-15900] RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1, 3.4.0, 3.2.3
• Component/s: rbf
• Labels: pull-request-available
• Assignee: Harunobu Daikoku
• Reporter: Harunobu Daikoku
• Created: 17/Mar/21 07:13
• Updated: 10/Jun/21 07:43
• Resolved: 29/Mar/21 02:58

Description

We observed that when a NameNode becomes UNAVAILABLE, the corresponding blockpool id in MembershipStoreImpl#activeNamespaces on dfsrouter unintentionally sets to empty, its initial value.

As a result of this, concat operations through dfsrouter fail with the following error as it cannot resolve the block id in the recognized active namespaces.

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RemoteException): Cannot locate a nameservice for block pool BP-...

A possible fix is to ignore UNAVAILABLE NameNode registrations, and set proper namespace information obtained from available NameNode registrations when constructing the cache of active namespaces.
https://github.com/apache/hadoop/blob/rel/release-3.3.0/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/impl/MembershipStoreImpl.java#L207-L221

Issue Links

• causes: HDFS-15952 TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky
• links to: GitHub Pull Request #2787
• links to: GitHub Pull Request #2866