[HDFS-8046] Allow better control of getContentSummary

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.1, 2.8.0, 2.7.2, 3.0.0-alpha1
• Labels: 2.6.1-candidate 2.7.2-candidate
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Kihwal Lee
• Reporter: Kihwal Lee
• Created: 02/Apr/15 17:15
• Updated: 06/Jan/17 01:34
• Resolved: 08/Apr/15 20:41

Description

On busy clusters, users performing quota checks against a big directory structure can affect the namenode performance. It has become a lot better after HDFS-4995, but as clusters get bigger and busier, it is apparent that we need finer grain control to avoid long read lock causing throughput drop.

Even with unfair namesystem lock setting, a long read lock (10s of milliseconds) can starve many readers and especially writers. So the locking duration should be reduced, which can be done by imposing a lower count-per-iteration limit in the existing implementation. But HDFS-4995 came with a fixed amount of sleep between locks. This needs to be made configurable, so that getContentSummary() doesn't get exceedingly slow.

Issue Links

• relates to: ACCUMULO-3957 Consider moving off getContentSummary in the monitor
• relates to: HDFS-8898 Create API and command-line argument to get quota and quota usage without detailed content summary