[HDFS-9668] Optimize the locking in FsDatasetImpl

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: datanode
• Assignee: Jingcheng Du
• Reporter: Jingcheng Du
• Created: 20/Jan/16 07:40
• Updated: 18/Feb/20 15:12

Description

During the HBase test on a tiered storage of HDFS (WAL is stored in SSD/RAMDISK, and all other files are stored in HDD), we observe many long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part of the jstack result:

"DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at /192.168.50.16:48521 [Receiving block BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread t@93336
   java.lang.Thread.State: BLOCKED
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1111)
	- waiting to lock <18324c9> (a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at /192.168.50.16:48520 [Receiving block BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:183)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None
	
"DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at /192.168.50.16:48520 [Receiving block BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread t@93335
   java.lang.Thread.State: RUNNABLE
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createNewFile(File.java:1012)
	at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
	- locked <18324c9> (a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:183)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

We measured the execution of some operations in FsDatasetImpl during the test. Here following is the result.

The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy load take a really long time.
It means one slow operation of finalizeBlock, addBlock and createRbw in a slow storage can block all the other same operations in the same DataNode, especially in HBase when many wal/flusher/compactor are configured.
We need a finer grained lock mechanism in a new FsDatasetImpl implementation and users can choose the implementation by configuring "dfs.datanode.fsdataset.factory" in DataNode.
We can implement the lock by either storage level or block-level.

Issue Links

• is blocked by: HADOOP-13702 Add a new instrumented read-write lock
• Is contained by: HDFS-15180 DataNode FsDatasetImpl Fine-Grained Locking via BlockPool.
• is related to: HDFS-15150 Introduce read write lock to Datanode
• is related to: HDFS-15000 Improve FsDatasetImpl to avoid IO operation in datasetLock
• relates to: HDFS-7060 Avoid taking locks when sending heartbeats from the DataNode
• relates to: HDFS-10682 Replace FsDatasetImpl object lock with a separate lock object