[HDFS-16484] [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.2.4, 3.3.5
• Fix Version/s: 3.4.0, 3.2.4, 3.3.5
• Component/s: sps
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.2.4, 3.3.5
• Hadoop Flags: Reviewed
• Assignee: qinyuren
• Reporter: qinyuren
• Created: 25/Feb/22 06:34
• Updated: 12/Feb/24 06:40
• Resolved: 13/Apr/22 02:54

Description

Currently, we ran SPS in our cluster and found this log. The SPSPathIdProcessor thread enters an infinite loop and prints the same log all the time.

In SPSPathIdProcessor thread, if it get a inodeId which path does not exist, then the SPSPathIdProcessor thread entry infinite loop and can't work normally. 

The reason is that #ctxt.getNextSPSPath() get a inodeId which path does not exist. The inodeId will not be set to null, causing the thread hold this inodeId forever.

public void run() {
  LOG.info("Starting SPSPathIdProcessor!.");
  Long startINode = null;
  while (ctxt.isRunning()) {
    try {
      if (!ctxt.isInSafeMode()) {
        if (startINode == null) {
          startINode = ctxt.getNextSPSPath();
        } // else same id will be retried
        if (startINode == null) {
          // Waiting for SPS path
          Thread.sleep(3000);
        } else {
          ctxt.scanAndCollectFiles(startINode);
          // check if directory was empty and no child added to queue
          DirPendingWorkInfo dirPendingWorkInfo =
              pendingWorkForDirectory.get(startINode);
          if (dirPendingWorkInfo != null
              && dirPendingWorkInfo.isDirWorkDone()) {
            ctxt.removeSPSHint(startINode);
            pendingWorkForDirectory.remove(startINode);
          }
        }
        startINode = null; // Current inode successfully scanned.
      }
    } catch (Throwable t) {
      String reClass = t.getClass().getName();
      if (InterruptedException.class.getName().equals(reClass)) {
        LOG.info("SPSPathIdProcessor thread is interrupted. Stopping..");
        break;
      }
      LOG.warn("Exception while scanning file inodes to satisfy the policy",
          t);
      try {
        Thread.sleep(3000);
      } catch (InterruptedException e) {
        LOG.info("Interrupted while waiting in SPSPathIdProcessor", t);
        break;
      }
    }
  }
} 

 

 

Issue Links

• links to: GitHub Pull Request #4032