Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15068

DataNode could meet deadlock if invoke refreshVolumes when register

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, 3.1.4, 3.2.2
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DataNode could meet deadlock when invoke `dfsadmin -reconfig datanode ip:host start` to trigger #refreshVolumes.
      1. DataNod#refreshVolumes hold datanode instance ownable synchronizer when enter this method first, then try to hold BPOfferService readlock when `bpos.getNamespaceInfo()` in following code segment.

              for (BPOfferService bpos : blockPoolManager.getAllNamenodeThreads()) {
                nsInfos.add(bpos.getNamespaceInfo());
              }
      

      2. BPOfferService#registrationSucceeded (which is invoked by #register when DataNode start or #reregister when processCommandFromActor) hold BPOfferService writelock first, then try to hold datanode instance ownable synchronizer in following method.

        synchronized void bpRegistrationSucceeded(DatanodeRegistration bpRegistration,
            String blockPoolId) throws IOException {
          id = bpRegistration;
      
          if(!storage.getDatanodeUuid().equals(bpRegistration.getDatanodeUuid())) {
            throw new IOException("Inconsistent Datanode IDs. Name-node returned "
                + bpRegistration.getDatanodeUuid()
                + ". Expecting " + storage.getDatanodeUuid());
          }
          
          registerBlockPoolWithSecretManager(bpRegistration, blockPoolId);
        }
      

        Attachments

        1. HDFS-15068.005.patch
          12 kB
          Aiphago
        2. HDFS-15068.004.patch
          12 kB
          Aiphago
        3. HDFS-15068.003.patch
          12 kB
          Aiphago
        4. HDFS-15068.002.patch
          11 kB
          Aiphago
        5. HDFS-15068.001.patch
          7 kB
          Aiphago

          Activity

            People

            • Assignee:
              Aiphag0 Aiphago
              Reporter:
              hexiaoqiao Xiaoqiao He
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: