[HDFS-15068] DataNode could meet deadlock if invoke refreshVolumes when register

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Mingxiang Li
• Reporter: Xiaoqiao He
• Created: 17/Dec/19 03:57
• Updated: 29/Nov/21 08:11
• Resolved: 04/Jan/20 09:09

Description

DataNode could meet deadlock when invoke `dfsadmin -reconfig datanode ip:host start` to trigger #refreshVolumes.
1. DataNod#refreshVolumes hold datanode instance ownable synchronizer when enter this method first, then try to hold BPOfferService readlock when `bpos.getNamespaceInfo()` in following code segment.

        for (BPOfferService bpos : blockPoolManager.getAllNamenodeThreads()) {
          nsInfos.add(bpos.getNamespaceInfo());
        }

2. BPOfferService#registrationSucceeded (which is invoked by #register when DataNode start or #reregister when processCommandFromActor) hold BPOfferService writelock first, then try to hold datanode instance ownable synchronizer in following method.

  synchronized void bpRegistrationSucceeded(DatanodeRegistration bpRegistration,
      String blockPoolId) throws IOException {
    id = bpRegistration;

    if(!storage.getDatanodeUuid().equals(bpRegistration.getDatanodeUuid())) {
      throw new IOException("Inconsistent Datanode IDs. Name-node returned "
          + bpRegistration.getDatanodeUuid()
          + ". Expecting " + storage.getDatanodeUuid());
    }
    
    registerBlockPoolWithSecretManager(bpRegistration, blockPoolId);
  }