Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-1476

scaling compaction with multiple threads

    Details

      Description

      Was thinking we should build in support to be able to handle more then one thread for compactions this will allow us to keep up with compactions when we get to the point where we store Tb's of data per node and may regions
      Maybe a configurable setting to set how many threads a region server can use for compactions.

      With compression turned on my compactions are limited by cpu speed with multi cores then it would be nice to be able to scale compactions to 2 or more cores.

        Issue Links

          Activity

          Hide
          streamy Jonathan Gray added a comment -

          This is amongst the many threading-related issues. Trying to keep this tied together.

          Show
          streamy Jonathan Gray added a comment - This is amongst the many threading-related issues. Trying to keep this tied together.
          Hide
          streamy Jonathan Gray added a comment -

          Would like to get this in to 0.20.5. Working on this actively now.

          Show
          streamy Jonathan Gray added a comment - Would like to get this in to 0.20.5. Working on this actively now.
          Hide
          stack stack added a comment -

          Bulk move of 0.20.5 issues into 0.21.0 after vote that we merge branch into TRUNK up on list.

          Show
          stack stack added a comment - Bulk move of 0.20.5 issues into 0.21.0 after vote that we merge branch into TRUNK up on list.
          Hide
          jdcryans Jean-Daniel Cryans added a comment -

          This sounds like HBASE-2405, can we close as dupe?

          Show
          jdcryans Jean-Daniel Cryans added a comment - This sounds like HBASE-2405 , can we close as dupe?
          Hide
          streamy Jonathan Gray added a comment -

          HBASE-2405 deals with open/close of regions not compactions? This is more of a dupe of HBASE-2832 which deals with multi-threading of flushing. The solutions will be very much the same (using the new EventHandlers) for flushing and compacting. And again, I'm hoping to do this for 0.90.

          Show
          streamy Jonathan Gray added a comment - HBASE-2405 deals with open/close of regions not compactions? This is more of a dupe of HBASE-2832 which deals with multi-threading of flushing. The solutions will be very much the same (using the new EventHandlers) for flushing and compacting. And again, I'm hoping to do this for 0.90.
          Hide
          streamy Jonathan Gray added a comment -

          Punting to 0.92 for now. The bigger compaction/flush improvements should happen in that version.

          Show
          streamy Jonathan Gray added a comment - Punting to 0.92 for now. The bigger compaction/flush improvements should happen in that version.
          Hide
          nspiegelberg Nicolas Spiegelberg added a comment -

          This is a decently-complicated fix. Up on Review Board at:

          https://review.cloudera.org/r/1664/

          Show
          nspiegelberg Nicolas Spiegelberg added a comment - This is a decently-complicated fix. Up on Review Board at: https://review.cloudera.org/r/1664/
          Hide
          stack stack added a comment -

          I did a first pass review N over on RB. Good stuff.

          Show
          stack stack added a comment - I did a first pass review N over on RB. Good stuff.
          Hide
          nspiegelberg Nicolas Spiegelberg added a comment -

          FYI: responded to stack's commentary on RB

          Show
          nspiegelberg Nicolas Spiegelberg added a comment - FYI: responded to stack's commentary on RB
          Hide
          stack stack added a comment -

          OK. I'm good w/ committing this to trunk with your extra comments. Good stuff N.

          Show
          stack stack added a comment - OK. I'm good w/ committing this to trunk with your extra comments. Good stuff N.
          Hide
          nspiegelberg Nicolas Spiegelberg added a comment -

          I did some minor refactoring in an internal peer review that's ongoing. I think it's all wrapped up, but waiting to confirm. I will re-post the patch and commit once done.

          Show
          nspiegelberg Nicolas Spiegelberg added a comment - I did some minor refactoring in an internal peer review that's ongoing. I think it's all wrapped up, but waiting to confirm. I will re-post the patch and commit once done.
          Hide
          nspiegelberg Nicolas Spiegelberg added a comment -

          Introducing per-store locking in HBASE-3797 will allow us to make compaction decisions at enqueue time & massively simplify this fix.

          Show
          nspiegelberg Nicolas Spiegelberg added a comment - Introducing per-store locking in HBASE-3797 will allow us to make compaction decisions at enqueue time & massively simplify this fix.
          Hide
          nspiegelberg Nicolas Spiegelberg added a comment -

          bundled with HBASE-3797 changes. See that JIRA for details.

          Show
          nspiegelberg Nicolas Spiegelberg added a comment - bundled with HBASE-3797 changes. See that JIRA for details.
          Hide
          lars_francke Lars Francke added a comment -

          This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

          Show
          lars_francke Lars Francke added a comment - This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

            People

            • Assignee:
              nspiegelberg Nicolas Spiegelberg
              Reporter:
              viper799 Billy Pearson
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development