Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8046

Allow better control of getContentSummary

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      On busy clusters, users performing quota checks against a big directory structure can affect the namenode performance. It has become a lot better after HDFS-4995, but as clusters get bigger and busier, it is apparent that we need finer grain control to avoid long read lock causing throughput drop.

      Even with unfair namesystem lock setting, a long read lock (10s of milliseconds) can starve many readers and especially writers. So the locking duration should be reduced, which can be done by imposing a lower count-per-iteration limit in the existing implementation. But HDFS-4995 came with a fixed amount of sleep between locks. This needs to be made configurable, so that getContentSummary() doesn't get exceedingly slow.

        Attachments

        1. HDFS-8046.v1.patch
          6 kB
          Kihwal Lee
        2. HDFS-8046-branch-2.6.1.txt
          6 kB
          Vinod Kumar Vavilapalli

          Issue Links

            Activity

              People

              • Assignee:
                kihwal Kihwal Lee
                Reporter:
                kihwal Kihwal Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: