[HDFS-9137] DeadLock between DataNode#refreshVolumes and BPOfferService#registrationSucceeded

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1, 3.0.0-alpha1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: datanode
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Uma Maheswara Rao G
• Reporter: Uma Maheswara Rao G
• Created: 24/Sep/15 05:55
• Updated: 30/Aug/16 01:26
• Resolved: 08/Oct/15 02:38

Description

I can see this code flows between DataNode#refreshVolumes and BPOfferService#registrationSucceeded could cause deadLock.
In practice situation may be rare as user calling refreshVolumes at the time DN registration with NN. But seems like issue can happen.

Reason for deadLock:

1) refreshVolumes will be called with DN lock and after at the end it will also trigger Block report. In the Block report call, BPServiceActor#triggerBlockReport calls toString on bpos. Here it takes readLock on bpos.
DN lock then boos lock

2) BPOfferSetrvice#registrationSucceeded call is taking writeLock on bpos and calling dn.bpRegistrationSucceeded which is again synchronized call on DN.
bpos lock and then DN lock.

So, this can clearly create dead lock.
I think simple fix could be to move triggerBlockReport call outside out DN lock and I feel that call may not be really needed inside DN lock.

Thoughts?

Issue Links

• is duplicated by: HDFS-9310 TestDataNodeHotSwapVolumes fails occasionally