[HDFS-5368] Namenode deadlock during safemode extention

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 2.2.0, 3.0.0-alpha1
• Assignee: Vinayakumar B
• Reporter: Vinayakumar B
• Created: 16/Oct/13 09:37
• Updated: 12/May/16 18:16
• Resolved: 26/Oct/13 01:26

Description

Namenode entered to safemode during restart

1. After restart NN entered to safemode extention.
2. During this time deadlock happened between datanode heartbeat and SafemodeMonitor() thread.

Found one Java-level deadlock:
=============================
"org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeMonitor@9fe953":
waiting to lock monitor 0x18c3b42c (object 0x0439c6f8, a java.util.TreeMap),
which is held by "IPC Server handler 2 on 62212"
"IPC Server handler 2 on 62212":
waiting to lock monitor 0x18c3987c (object 0x043849a0, a org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo),
which is held by "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeMonitor@9fe953"

Check attached jstack for complete stack

Issue Links

• is duplicated by: HDFS-5132 Deadlock in NameNode between SafeModeMonitor#run and DatanodeManager#handleHeartbeat
• is related to: HDFS-3846 Namenode deadlock in branch-1