Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15113

Missing IBR when NameNode restart if open processCommand async feature

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:

      Description

      Recently, I meet one case that NameNode missing block after restart which is related with HDFS-14997.
      a. during NameNode restart, it will return command `DNA_REGISTER` to DataNode when receive some RPC request from DataNode.
      b. when DataNode receive `DNA_REGISTER` command, it will run #reRegister async.

        void reRegister() throws IOException {
          if (shouldRun()) {
            // re-retrieve namespace info to make sure that, if the NN
            // was restarted, we still match its version (HDFS-2120)
            NamespaceInfo nsInfo = retrieveNamespaceInfo();
            // and re-register
            register(nsInfo);
            scheduler.scheduleHeartbeat();
            // HDFS-9917,Standby NN IBR can be very huge if standby namenode is down
            // for sometime.
            if (state == HAServiceState.STANDBY || state == HAServiceState.OBSERVER) {
              ibrManager.clearIBRs();
            }
          }
        }
      

      c. As we know, #register will trigger BR immediately.
      d. because #reRegister run async, so we could not make sure which one run first between send FBR and clear IBR. If clean IBR run first, it will be OK. But if send FBR first then clear IBR, it will missing some blocks received between these two time point until next FBR.

        Attachments

        1. HDFS-15113.001.patch
          1 kB
          Xiaoqiao He
        2. HDFS-15113.002.patch
          4 kB
          Xiaoqiao He
        3. HDFS-15113.003.patch
          4 kB
          Xiaoqiao He
        4. HDFS-15113.004.patch
          8 kB
          Xiaoqiao He
        5. HDFS-15113.005.patch
          8 kB
          Xiaoqiao He
        6. HDFS-15113.addendum.patch
          4 kB
          Xiaoqiao He

          Issue Links

            Activity

              People

              • Assignee:
                hexiaoqiao Xiaoqiao He
                Reporter:
                hexiaoqiao Xiaoqiao He
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: