Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18309

Support multi threads in CleanerChore

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Resolved
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-beta-1, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      After HBASE-18309,
      1. we could use multiple threads to scan archive directories (including data and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning the concrete size, but no more than available cpu cores) and double (between >0.0 and <=1.0, meaning ratio of available cpu cores) value and defaults to 0.25. Pay attention that 1.0 is different from 1, the former indicates it will use 100% of cores, while the latter will use only 1 thread for chore to scan dir.
      2. We also support using multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size, 2 by default.
      3. In addition, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size support online re-configuration as hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count does.
      4. Please take hbase.cleaner.scan.dir.concurrent.size, hbase.regionserver.hfilecleaner.large.thread.count, hbase.regionserver.hfilecleaner.small.thread.count and hbase.oldwals.cleaner.thread.size into account when setting this config to avoid thread flooding.
      Show
      After HBASE-18309 , 1. we could use multiple threads to scan archive directories (including data and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning the concrete size, but no more than available cpu cores) and double (between >0.0 and <=1.0, meaning ratio of available cpu cores) value and defaults to 0.25. Pay attention that 1.0 is different from 1, the former indicates it will use 100% of cores, while the latter will use only 1 thread for chore to scan dir. 2. We also support using multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size, 2 by default. 3. In addition, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size support online re-configuration as hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count does. 4. Please take hbase.cleaner.scan.dir.concurrent.size, hbase.regionserver.hfilecleaner.large.thread.count, hbase.regionserver.hfilecleaner.small.thread.count and hbase.oldwals.cleaner.thread.size into account when setting this config to avoid thread flooding.

      Description

      There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find this is not enough. The number of files under oldWALs reach the max-directory-items limit of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash not happened any more.

      What's more, currently there's only one thread iterating the archive directory, and we could use multiple threads cleaning sub directories in parallel to speed it up.

        Attachments

        1. HBASE-18309.addendum.patch
          1 kB
          Chia-Ping Tsai
        2. HBASE-18309.branch-1.001.patch
          42 kB
          Reid Chan
        3. HBASE-18309.branch-1.002.patch
          42 kB
          Reid Chan
        4. HBASE-18309.branch-1.003.patch
          40 kB
          Reid Chan
        5. HBASE-18309.branch-1.004.patch
          40 kB
          Reid Chan
        6. HBASE-18309.branch-1.005.patch
          40 kB
          Reid Chan
        7. HBASE-18309.branch-1.006.patch
          40 kB
          Zach York
        8. HBASE-18309.master.001.patch
          15 kB
          Reid Chan
        9. HBASE-18309.master.002.patch
          16 kB
          Reid Chan
        10. HBASE-18309.master.004.patch
          31 kB
          Reid Chan
        11. HBASE-18309.master.005.patch
          32 kB
          Reid Chan
        12. HBASE-18309.master.006.patch
          32 kB
          Reid Chan
        13. HBASE-18309.master.007.patch
          32 kB
          Reid Chan
        14. HBASE-18309.master.008.patch
          32 kB
          Reid Chan
        15. HBASE-18309.master.009.patch
          32 kB
          Reid Chan
        16. HBASE-18309.master.010.patch
          32 kB
          Reid Chan
        17. HBASE-18309.master.011.patch
          33 kB
          Reid Chan
        18. HBASE-18309.master.012.patch
          33 kB
          Reid Chan
        19. space_consumption_in_archive.png
          202 kB
          Yu Li

          Issue Links

            Activity

              People

              • Assignee:
                reidchan Reid Chan
                Reporter:
                binlijin Lijin Bin
              • Votes:
                0 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: