HDFS-4995 added a config dfs.content-summary.limit which allows for an administrator to set a limit on the number of entries processed during a single acquisition of the FSNamesystemLock during the creation of a content summary. This is useful to prevent very long (multiple seconds) pauses on the NameNode when getContentSummary is called on large directories.
However, even on versions with
HDFS-4995, we have seen warnings like:
happen quite consistently when getContentSummary was called on a large directory on a heavily-loaded NameNode. Such long pauses completely destroy the performance of the NameNode. We have the limit set to its default of 5000; if it was respected, clearly there would not be a 10-second pause.
The current yield() code within ContentSummaryComputationContext looks like:
We believe that this check in particular is the culprit:
The content summary computation will only relinquish the lock if it is currently the only holder of the lock. Given the high volume of read requests on a heavily loaded NameNode, especially when unfair locking is enabled, it is likely there may be another holder of the read lock performing some short-lived operation. By refusing to give up the lock in this case, the content summary computation ends up never relinquishing the lock.
We propose to simply remove the readHoldCount checks from this yield(). This should alleviate the case described above by giving up the read lock and allowing other short-lived operations to complete (while the content summary thread sleeps) so that the lock can finally be given up completely. This has the drawback that sometimes, the content summary may give up the lock unnecessarily, if the read lock is never actually released by the time the thread continues again. The only negative impact from this is to make some large content summary operations slightly slower, with the tradeoff of reducing NameNode-wide performance impact.