Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15068

DataNode could meet deadlock if invoke refreshVolumes when register

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0, 3.1.4, 3.2.2
    • datanode
    • None
    • Reviewed

    Description

      DataNode could meet deadlock when invoke `dfsadmin -reconfig datanode ip:host start` to trigger #refreshVolumes.
      1. DataNod#refreshVolumes hold datanode instance ownable synchronizer when enter this method first, then try to hold BPOfferService readlock when `bpos.getNamespaceInfo()` in following code segment.

              for (BPOfferService bpos : blockPoolManager.getAllNamenodeThreads()) {
                nsInfos.add(bpos.getNamespaceInfo());
              }
      

      2. BPOfferService#registrationSucceeded (which is invoked by #register when DataNode start or #reregister when processCommandFromActor) hold BPOfferService writelock first, then try to hold datanode instance ownable synchronizer in following method.

        synchronized void bpRegistrationSucceeded(DatanodeRegistration bpRegistration,
            String blockPoolId) throws IOException {
          id = bpRegistration;
      
          if(!storage.getDatanodeUuid().equals(bpRegistration.getDatanodeUuid())) {
            throw new IOException("Inconsistent Datanode IDs. Name-node returned "
                + bpRegistration.getDatanodeUuid()
                + ". Expecting " + storage.getDatanodeUuid());
          }
          
          registerBlockPoolWithSecretManager(bpRegistration, blockPoolId);
        }
      

      Attachments

        1. HDFS-15068.001.patch
          7 kB
          Mingxiang Li
        2. HDFS-15068.002.patch
          11 kB
          Mingxiang Li
        3. HDFS-15068.003.patch
          12 kB
          Mingxiang Li
        4. HDFS-15068.004.patch
          12 kB
          Mingxiang Li
        5. HDFS-15068.005.patch
          12 kB
          Mingxiang Li

        Activity

          People

            Aiphag0 Mingxiang Li
            hexiaoqiao Xiaoqiao He
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: