[HDFS-9445] Datanode may deadlock while handling a bad volume

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1
• Target Version/s: 2.7.2, 2.6.4
• Hadoop Flags: Reviewed
• Assignee: Walter Su
• Reporter: Kihwal Lee
• Created: 20/Nov/15 16:40
• Updated: 05/Jan/17 22:52
• Resolved: 04/Jan/16 14:24

Description

Found one Java-level deadlock:
=============================
"DataXceiver for client DFSClient_attempt_xxx at /1.2.3.4:100 [Sending block BP-xxxxx:blk_123_456]":
  waiting to lock monitor 0x00007f77d0731768 (object 0x00000000d60d9930, a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
  which is held by "Thread-565"
"Thread-565":
  waiting for ownable synchronizer 0x00000000d55613c8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "DataNode: heartbeating to my-nn:8020"
"DataNode: heartbeating to my-nn:8020":
  waiting to lock monitor 0x00007f77d0731768 (object 0x00000000d60d9930, a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
  which is held by "Thread-565"