Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-17215

Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      After HBASE-17215 we change to use two threads for (archived) hfile cleaning. The size throttling for large/small files could be set through "hbase.regionserver.thread.hfilecleaner.throttle" and default to 67108864 (64M). It supports online configuration change, just find the active master address through zookeeper dump and use it in update_config command, e.g. update_config 'hbasem1.et2.tbsite.net,60100,1488038696741'
      Show
      After HBASE-17215 we change to use two threads for (archived) hfile cleaning. The size throttling for large/small files could be set through "hbase.regionserver.thread.hfilecleaner.throttle" and default to 67108864 (64M). It supports online configuration change, just find the active master address through zookeeper dump and use it in update_config command, e.g. update_config 'hbasem1.et2.tbsite.net,60100,1488038696741'

      Description

      When using PCIe-SSD the flush speed will be really quick, and although we have per CF flush, we still have the hbase.regionserver.optionalcacheflushinterval setting and some other mechanism to avoid data kept in memory for too long to flush small hfiles. In our online environment we found the single thread cleaner kept cleaning earlier flushed small files while large files got no chance, which caused disk full then many other problems.

      Deleting hfiles in parallel with too many threads will also increase the workload of namenode, so here we propose to separate large/small hfile cleaner threads just like we do for compaction, and it turned out to work well in our cluster.

        Attachments

        1. HBASE-17215.patch
          22 kB
          Yu Li
        2. HBASE-17215.v2.patch
          23 kB
          Yu Li
        3. HBASE-17215.v3.patch
          23 kB
          Yu Li

          Issue Links

            Activity

              People

              • Assignee:
                liyu Yu Li
                Reporter:
                liyu Yu Li
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: