Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Fix Version/s: 0.8 beta 1
    • Component/s: Core
    • Labels:

      Description

      This ticket overlaps with CASSANDRA-1876 to a degree, but the approaches and reasoning are different enough to open a separate issue.

      The problem with compactions currently is that they compact the set of sstables that existed the moment the compaction started. This means that for longer running compactions (even when running as fast as possible on the hardware), a very large number of new sstables might be created in the meantime. We have observed this proliferation of sstables killing performance during major/high-bucketed compactions.

      One approach would be to pause compactions in upper buckets (containing larger files) when compactions in lower buckets become possible. While this would likely solve the problem with read performance, it does not actually help us perform compaction any faster, which is a reasonable requirement for other situations.

      Instead, we need to be able to perform any compactions that are currently required in parallel, independent of what bucket they might be in.

        Issue Links

          Activity

          Stu Hood created issue -
          Stu Hood made changes -
          Field Original Value New Value
          Link This issue supercedes CASSANDRA-1876 [ CASSANDRA-1876 ]
          Stu Hood made changes -
          Link This issue relates to CASSANDRA-2156 [ CASSANDRA-2156 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-sstabletracker.txt [ 12471363 ]
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12471364 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-sstabletracker.txt [ 12471363 ]
          Stu Hood made changes -
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12471364 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-sstabletracker.txt [ 12473448 ]
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12473449 ]
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt [ 12473450 ]
          Stu Hood made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-sstabletracker.txt [ 12473448 ]
          Stu Hood made changes -
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12473449 ]
          Stu Hood made changes -
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt [ 12473450 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-sstabletracker.txt [ 12474327 ]
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12474328 ]
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt [ 12474329 ]
          Stu Hood made changes -
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt [ 12474329 ]
          Stu Hood made changes -
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12474328 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-sstabletracker.txt [ 12474327 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-DataTracker.txt [ 12475069 ]
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12475070 ]
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt [ 12475071 ]
          Attachment 0004-Try-harder-to-close-scanners-in-compaction-close.txt [ 12475072 ]
          Jonathan Ellis made changes -
          Assignee Stu Hood [ stuhood ]
          Reviewer jbellis slebresne
          Stu Hood made changes -
          Attachment 0004-Try-harder-to-close-scanners-in-compaction-close.txt [ 12475072 ]
          Stu Hood made changes -
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt [ 12475071 ]
          Stu Hood made changes -
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12475070 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-DataTracker.txt [ 12475069 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-DataTracker.txt [ 12475791 ]
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12475792 ]
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-a-concrete-ser.txt [ 12475793 ]
          Attachment 0004-Allow-multithread-compaction-to-be-disabled.txt [ 12475794 ]
          Attachment 0005-Add-a-harness-to-allow-compaction-tasks-that-need-to-a.txt [ 12475795 ]
          Pavel Yaskevich made changes -
          Link This issue blocks CASSANDRA-1740 [ CASSANDRA-1740 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-DataTracker.txt [ 12475791 ]
          Stu Hood made changes -
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12475792 ]
          Stu Hood made changes -
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-a-concrete-ser.txt [ 12475793 ]
          Stu Hood made changes -
          Attachment 0004-Allow-multithread-compaction-to-be-disabled.txt [ 12475794 ]
          Stu Hood made changes -
          Attachment 0005-Add-a-harness-to-allow-compaction-tasks-that-need-to-a.txt [ 12475795 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-DataTracker.txt [ 12475906 ]
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12475907 ]
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-a-concrete-ser.txt [ 12475908 ]
          Attachment 0004-Allow-multithread-compaction-to-be-disabled.txt [ 12475909 ]
          Attachment 0005-Add-a-harness-to-allow-compaction-tasks-that-need-to-a.txt [ 12475910 ]
          Stu Hood made changes -
          Attachment 0001-Add-a-compacting-set-to-DataTracker.txt [ 12475906 ]
          Stu Hood made changes -
          Attachment 0005-Add-a-harness-to-allow-compaction-tasks-that-need-to-a.txt [ 12475910 ]
          Stu Hood made changes -
          Attachment 0004-Allow-multithread-compaction-to-be-disabled.txt [ 12475909 ]
          Stu Hood made changes -
          Attachment 0003-Expose-multiple-compactions-via-JMX-and-a-concrete-ser.txt [ 12475908 ]
          Stu Hood made changes -
          Attachment 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt [ 12475907 ]
          Sylvain Lebresne made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Gavin made changes -
          Workflow no-reopen-closed, patch-avail [ 12599269 ] patch-available, re-open possible [ 12749357 ]
          Gavin made changes -
          Workflow patch-available, re-open possible [ 12749357 ] reopen-resolved, no closed status, patch-avail, testing [ 12756958 ]

            People

            • Assignee:
              Stu Hood
              Reporter:
              Stu Hood
              Reviewer:
              Sylvain Lebresne
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development