Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-8329

LeveledCompactionStrategy should split large files across data directories when compacting

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: 2.0.12, 2.1.3
    • Component/s: Compaction
    • Labels:

      Description

      Because we fall back to STCS for L0 when LCS gets behind, the sstables in L0 can get quite large during sustained periods of heavy writes. This can result in large imbalances between data volumes when using JBOD support.

      Eventually these large files get broken up as L0 sstables are moved up into higher levels; however, because LCS only chooses a single volume on which to write all of the sstables created during a single compaction, the imbalance is persisted.

      1. 0001-get-new-sstable-directory-for-every-new-file-during-.patch
        10 kB
        Marcus Eriksson
      2. test_no_patch_2.0.jpg
        103 kB
        Alan Boudreault
      3. test_with_patch_2.0.jpg
        109 kB
        Alan Boudreault

        Issue Links

          Activity

          Hide
          snazy Robert Stupp added a comment -

          Related to CASSANDRA-7386 ?

          Show
          snazy Robert Stupp added a comment - Related to CASSANDRA-7386 ?
          Hide
          krummas Marcus Eriksson added a comment -

          Attaching patch that gets a new directory for every new file during compaction.

          Show
          krummas Marcus Eriksson added a comment - Attaching patch that gets a new directory for every new file during compaction.
          Hide
          jbellis Jonathan Ellis added a comment -

          Related to CASSANDRA-7386?

          Yes.

          Show
          jbellis Jonathan Ellis added a comment - Related to CASSANDRA-7386 ? Yes.
          Hide
          yukim Yuki Morishita added a comment -

          Patch LGTM.
          Though if we can test similar to the one that Alan Boudreault did in CASSANDRA-7386, it would be perfect.

          Show
          yukim Yuki Morishita added a comment - Patch LGTM. Though if we can test similar to the one that Alan Boudreault did in CASSANDRA-7386 , it would be perfect.
          Hide
          aboudreault Alan Boudreault added a comment - - edited

          Let me know if I can help here. I'm already set up for my tests in CASSANDRA-7386, so I can probably test this one too. CC Ryan McGuire unless CASSANDRA-8316 has a higher priority.

          Show
          aboudreault Alan Boudreault added a comment - - edited Let me know if I can help here. I'm already set up for my tests in CASSANDRA-7386 , so I can probably test this one too. CC Ryan McGuire unless CASSANDRA-8316 has a higher priority.
          Hide
          yukim Yuki Morishita added a comment -

          Alan Boudreault Thanks for your help. We need to check if LCS will write to multiple disks when compacting large SSTables in L0 after the patch.

          Show
          yukim Yuki Morishita added a comment - Alan Boudreault Thanks for your help. We need to check if LCS will write to multiple disks when compacting large SSTables in L0 after the patch.
          Hide
          aboudreault Alan Boudreault added a comment -

          Marcus Eriksson Is there a patch for cassandra 2.1? I assume that this issue affects also 2.1 and master.

          Show
          aboudreault Alan Boudreault added a comment - Marcus Eriksson Is there a patch for cassandra 2.1? I assume that this issue affects also 2.1 and master.
          Hide
          krummas Marcus Eriksson added a comment -

          Alan Boudreault yes, it does, I think testing on 2.0 is enough here, unless you really want to test 2.1+?

          Show
          krummas Marcus Eriksson added a comment - Alan Boudreault yes, it does, I think testing on 2.0 is enough here, unless you really want to test 2.1+?
          Hide
          aboudreault Alan Boudreault added a comment - - edited

          Devs, here my test results.

          Test

          • 12 disks of 2G of size
          • Cassandra use default values for concurrent_compactors and compaction_throughput_mb_per_sec.
          • Goal: Stress the server with many concurent writes for 45-50 minutes.
          Results - No Patch

          We can see the peak the LCS compaction of big sstables.

          test_no_patch_2.0.jpg

          Results - With Patch

          Success. There is no more peak during the compaction.

          test_with_patch_2.0.jpg

          Let me know if I can do anything else.

          Show
          aboudreault Alan Boudreault added a comment - - edited Devs, here my test results. Test 12 disks of 2G of size Cassandra use default values for concurrent_compactors and compaction_throughput_mb_per_sec. Goal: Stress the server with many concurent writes for 45-50 minutes. Results - No Patch We can see the peak the LCS compaction of big sstables. test_no_patch_2.0.jpg Results - With Patch Success. There is no more peak during the compaction. test_with_patch_2.0.jpg Let me know if I can do anything else.
          Hide
          krummas Marcus Eriksson added a comment -

          ok, thanks guys, committed!

          Show
          krummas Marcus Eriksson added a comment - ok, thanks guys, committed!

            People

            • Assignee:
              krummas Marcus Eriksson
              Reporter:
              jblangston@datastax.com J.B. Langston
              Reviewer:
              Yuki Morishita
              Tester:
              Alan Boudreault
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development