Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
On busy clusters, users performing quota checks against a big directory structure can affect the namenode performance. It has become a lot better after HDFS-4995, but as clusters get bigger and busier, it is apparent that we need finer grain control to avoid long read lock causing throughput drop.
Even with unfair namesystem lock setting, a long read lock (10s of milliseconds) can starve many readers and especially writers. So the locking duration should be reduced, which can be done by imposing a lower count-per-iteration limit in the existing implementation. But HDFS-4995 came with a fixed amount of sleep between locks. This needs to be made configurable, so that getContentSummary() doesn't get exceedingly slow.
Attachments
Attachments
Issue Links
- relates to
-
ACCUMULO-3957 Consider moving off getContentSummary in the monitor
- Resolved
-
HDFS-8898 Create API and command-line argument to get quota and quota usage without detailed content summary
- Resolved
I will post a patch that we applied in our systems. Many clusters have been running with it for some time now.