[HDFS-4851] Deadlock in pipeline recovery

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.4-alpha, 3.0.0-alpha1
• Component/s: datanode
• Target Version/s: 2.1.0-beta
• Assignee: Andrew Wang
• Reporter: Andrew Wang
• Created: 25/May/13 01:13
• Updated: 12/May/16 18:14
• Resolved: 26/Jul/13 06:39

Description

Here's a deadlock scenario that cropped up during pipeline recovery, debugged through jstacks. Todd tipped me off to this one.

1. Pipeline fails, client initiates recovery. We have the old leftover DataXceiver, and a new one doing recovery.
2. New DataXceiver does recoverRbw, grabbing the FsDatasetImpl lock
3. Old DataXceiver is in BlockReceiver#computePartialChunkCrc, calls FsDatasetImpl#getTmpInputStreams and blocks on the FsDatasetImpl lock.
4. New DataXceiver ReplicaInPipeline#stopWriter, interrupting the old DataXceiver and then joining on it.
5. Boom, deadlock. New DX holds the FsDatasetImpl lock and is joining on the old DX, which is in turn waiting on the FsDatasetImpl lock.

Issue Links

• duplicates: HDFS-3655 Datanode recoverRbw could hang sometime