Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18083

Make large/small file clean thread number configurable in HFileCleaner

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      After HBASE-18083 we could configure HFileCleaner to use multiple threads for large/small (archived) hfile cleaning with hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count, both default to 1. These properties support online configuration change.
      Show
      After HBASE-18083 we could configure HFileCleaner to use multiple threads for large/small (archived) hfile cleaning with hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count, both default to 1. These properties support online configuration change.

      Description

      Currently we have only one thread for both large and small file cleaning, but when write pressure is huge we might need more cleaner threads, so we need to make the thread number configurable.

      We observed more than 1.8PB data in archive directory online due to business access rate change, and this proposal is one of the necessary changes required.

      Default value of the configurations would still be left to 1 to keep low pressure to NN for normal case.

        Attachments

        1. HBASE-18083.patch
          8 kB
          Yu Li
        2. HBASE-18083.v2.patch
          10 kB
          Yu Li
        3. HBASE-18083.v3.patch
          13 kB
          Yu Li

          Issue Links

            Activity

              People

              • Assignee:
                liyu Yu Li
                Reporter:
                liyu Yu Li
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: