[HDFS-15555] RBF: Refresh cacheNS when SocketException occurs

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.4.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: rbf
• Labels: pull-request-available

Environment

HDFS 3.3.0, Java 11

• Target Version/s: 3.3.1, 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Akira Ajisaka
• Reporter: Akira Ajisaka
• Created: 02/Sep/20 02:39
• Updated: 24/Apr/24 07:39
• Resolved: 14/Sep/20 02:37

Description

Problem:
When active NameNode is restarted and loading fsimage, DFSRouters significantly slow down.

Investigation:
When active NameNode is restarted and loading fsimage, RouterRpcClient receives SocketException. Since RouterRpcClient#isUnavailableException(IOException) returns false when the argument is SocketException, the MembershipNameNodeResolver#cacheNS is not refreshed. That's why the order of the NameNodes returned by MemberShipNameNodeResolver#getNamenodesForNameserviceId(String) is unchanged and the active NameNode is still returned first. Therefore RouterRpcClient still tries to connect to the NameNode that is loading fsimage.

After loading the fsimage, the NameNode throws StandbyException. The exception is one of the 'Unavailable Exception' and the cacheNS is refreshed.

Workaround:
Stop NameNode and wait 1 minute before starting NameNode instead of restarting.

Issue Links

• is related to: HDFS-15575 RBF: Create test cases that simulate general exceptions on NameNodes
• links to: GitHub Pull Request #2267