[HDFS-1766] Datanode is marked dead, but datanode process is alive and verifying blocks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.23.0
• Component/s: datanode
• Assignee: Hairong Kuang
• Reporter: Hairong Kuang
• Created: 17/Mar/11 21:09
• Updated: 03/Apr/14 02:29
• Resolved: 03/Apr/14 02:29

Description

We have a datanode marked dead in the namenode, and it is not taking any traffic. But it is verifying blocks continuously, so the DataNode process is definitely not dead. Jstack shows that the main thread and the offerService thread are gone but the JVM stuck at waiting for other threads to die. It seems to me that the offerService thread has died abnormally, for example, by a runtime exception and it did not shut down other threads before exiting.