Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20352

[Chore] Backport HBASE-18309 to branch-1

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.3.3, 1.4.4
    • None
    • None
    • Reviewed
    • Hide
      HBASE-20352 aims to speed up process of cleaning oldWALs in two aspects,
      1. Use multiple threads to scan oldWALs directories through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning the concrete size, but no more than available cpu cores) and double (between >0.0 and <=1.0, meaning ratio of available cpu cores) value, 0.25 by default. Pay attention that 1.0 is different from 1, the former indicates it will use 100% of cores, while the latter will use only 1 thread for chore to scan dir.
      2. Use multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size, 2 by default.
      3. In addition, two configs, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size both support online re-configuration.
      Show
      HBASE-20352 aims to speed up process of cleaning oldWALs in two aspects, 1. Use multiple threads to scan oldWALs directories through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning the concrete size, but no more than available cpu cores) and double (between >0.0 and <=1.0, meaning ratio of available cpu cores) value, 0.25 by default. Pay attention that 1.0 is different from 1, the former indicates it will use 100% of cores, while the latter will use only 1 thread for chore to scan dir. 2. Use multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size, 2 by default. 3. In addition, two configs, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size both support online re-configuration.

    Description

      Using multiple threads to scan directory and to clean old WALs

      Attachments

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-18309 Support multi threads in CleanerChore

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HBASE-20877 Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-20401 Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link Review Board

          Activity

            People

              reidchan Reid Chan
              reidchan Reid Chan
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: