Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19528

Major Compaction Tool

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0, 2.0.0-beta-2, 2.0.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Tool allows you to compact a cluster with given concurrency of regionservers compacting at a given time. If tool completes successfully everything requested for compaction will be compacted, regardless of region moves, splits and merges.

      Description

      The basic overview of how this tool works is:

      Parameters:

      Table

      Stores

      ClusterConcurrency

      Timestamp

      So you input a table, desired concurrency and the list of stores you wish to major compact. The tool first checks the filesystem to see which stores need compaction based on the timestamp you provide (default is current time). It takes that list of stores that require compaction and executes those requests concurrently with at most N distinct RegionServers compacting at a given time. Each thread waits for the compaction to complete before moving to the next queue. If a region split, merge or move happens this tool ensures those regions get major compacted as well.

      This helps us in two ways, we can limit how much I/O bandwidth we are using for major compaction cluster wide and we are guaranteed after the tool completes that all requested compactions complete regardless of moves, merges and splits.

        Attachments

        1. HBASE-19528.v8.patch
          41 kB
          churro morales
        2. HBASE-19528.v2.branch-1 (1).patch
          42 kB
          Michael Stack
        3. HBASE-19528.v2.branch-1 (1).patch
          42 kB
          Michael Stack
        4. HBASE-19528.v2.branch-1.patch
          42 kB
          churro morales
        5. HBASE-19528.v2.branch-1.patch
          42 kB
          Michael Stack
        6. HBASE-19528.v2.branch-1.patch
          42 kB
          Michael Stack
        7. HBASE-19528.v2.branch-1.patch
          42 kB
          Michael Stack
        8. HBASE-19528.v1.patch
          39 kB
          churro morales
        9. HBASE-19528.v1.branch-1.patch
          42 kB
          churro morales
        10. HBASE-19528.patch
          35 kB
          churro morales
        11. HBASE-19528.branch-1.patch
          42 kB
          churro morales
        12. 0001-HBASE-19528-Major-Compaction-Tool-ADDENDUM.patch
          6 kB
          Michael Stack
        13. 0001-HBASE-19528-Do-nothing-patch.patch
          0.9 kB
          Michael Stack
        14. 0001-HBASE-19528-Do-nothing-patch.patch
          0.9 kB
          Michael Stack

          Issue Links

            Activity

              People

              • Assignee:
                churromorales churro morales
                Reporter:
                churromorales churro morales
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: