[HDFS-9659] EditLogTailerThread to Active Namenode RPC should timeout - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 3.0.0-alpha1
Fix Version/s: 3.0.0-alpha1
Component/s: ha, namenode
Labels:
None

Hadoop Flags:

Reviewed

Description

EditLogTailerThread to Active Namenode RPC doesn't have timeout and it’s removed in ~~HDFS-6440~~.

When inject the disk slow and consume system IO to the active name node, the nameservice can't switch and this is because SNN not able to stop EditLogTailerThread.

Thread dump from SNN

"IPC Server handler 33 on 25000" #118 daemon prio=5 os_prio=0 tid=0x00007f2384409800 nid=0x26c89 in Object.wait() [0x00007f2376ac7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1245)
	- locked <0x00000006d517f538> (a org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread)
	at java.lang.Thread.join(Thread.java:1319)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.stop(EditLogTailer.java:183)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopStandbyServices(FSNamesystem.java:1284)
	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopStandbyServices(NameNode.java:1852)
	at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.exitState(StandbyState.java:72)
	at org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:62)
	at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1684)

Thread dump for EditLogTailerThread, it is stuck in NamenodeProtocolTranslatorPB.rollEditLog() rpc call.

"Edit log tailer" #150 prio=5 os_prio=0 tid=0x00007f2395569800 nid=0x26cac in Object.wait() [0x00007f2374aa7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.hadoop.ipc.Client.call(Client.java:1503)
	- locked <0x00000006d581bb90> (a org.apache.hadoop.ipc.Client$Call)
	at org.apache.hadoop.ipc.Client.call(Client.java:1448)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy16.rollEditLog(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:148)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:301)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:298)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:420)