[HDFS-15273] CacheReplicationMonitor hold lock for long time and lead to NN out of service

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.4.0
• Component/s: caching, namenode
• Assignee: Xiaoqiao He
• Reporter: Xiaoqiao He
• Created: 12/Apr/20 06:45
• Updated: 26/Oct/23 17:59
• Resolved: 26/Oct/23 17:59

Description

CacheReplicationMonitor scan Cache Directives and Cached BlockMap periodically. If we add more and more cache directives, CacheReplicationMonitor will cost very long time to rescan all of cache directives and cache blocks. Meanwhile, scan operation hold global write lock, during scan period, NameNode could not process other request.
So I think we should warn this risk to end user who turn on CacheManager feature before improve this implement.

  private void rescan() throws InterruptedException {
    scannedDirectives = 0;
    scannedBlocks = 0;
    try {
      namesystem.writeLock();
      try {
        lock.lock();
        if (shutdown) {
          throw new InterruptedException("CacheReplicationMonitor was " +
              "shut down.");
        }
        curScanCount = completedScanCount + 1;
      } finally {
        lock.unlock();
      }

      resetStatistics();
      rescanCacheDirectives();
      rescanCachedBlockMap();
      blockManager.getDatanodeManager().resetLastCachingDirectiveSentTime();
    } finally {
      namesystem.writeUnlock();
    }
  }