[HDFS-7212] FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time

• Type: Bug
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0
• Component/s: datanode

Environment

PROD

• Assignee: Unassigned
• Reporter: Istvan Szukacs
• Created: 08/Oct/14 17:12
• Updated: 09/May/16 16:58

Description

There are 3000 - 8000 threads in each datanode JVM, blocking the entire VM and rendering the service unusable, missing heartbeats and stopping data access. The threads look like this:

3415 (state = BLOCKED)

- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=834 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=867 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1197 (Interpreted frame)
- java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=214 (Compiled frame)
- java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290 (Compiled frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher.add(org.apache.hadoop.net.unix.DomainSocket, org.apache.hadoop.net.unix.DomainSocketWatcher$Handler) @bci=4, line=286 (Interpreted frame)
- org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(java.lang.String, org.apache.hadoop.net.unix.DomainSocket) @bci=169, line=283 (Interpreted frame)
- org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(java.lang.String) @bci=212, line=413 (Interpreted frame)
- org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(java.io.DataInputStream) @bci=13, line=172 (Interpreted frame)
- org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(org.apache.hadoop.hdfs.protocol.datatransfer.Op) @bci=149, line=92 (Compiled frame)
- org.apache.hadoop.hdfs.server.datanode.DataXceiver.run() @bci=510, line=232 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)

Has anybody seen this before?

Issue Links

• duplicates: HADOOP-10404 Some accesses to DomainSocketWatcher#closed are not protected by lock
• is related to: HADOOP-11333 Fix deadlock in DomainSocketWatcher when the notification pipe is full
• relates to: HDFS-7999 FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time