Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-5547

Multi-threaded scrub, cleanup and upgradesstables

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: 2.1 beta2
    • Component/s: Tools
    • Labels:

      Description

      Scrub (especially offline) could benefit from being multi-threaded, especially in the case where the SSTables are compressed.

      1. 0001-5547.patch
        18 kB
        Marcus Eriksson
      2. cassandra-2.0-5547.txt
        16 kB
        Russell Spitzer

        Issue Links

          Activity

          Hide
          jbellis Jonathan Ellis added a comment -

          Could generalize to cleanup and upgrade too.

          Show
          jbellis Jonathan Ellis added a comment - Could generalize to cleanup and upgrade too.
          Hide
          rspitzer Russell Spitzer added a comment -

          Thought I would make an attempt at this. Made all three operations request a thread per file. I would be glad to make any changes/improvements requested.

          Example of multithreaded scrubbing

          INFO [CompactionExecutor:29] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-38-Data.db') (18061938 bytes)
           INFO [CompactionExecutor:26] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-40-Data.db') (110006268 bytes)
           INFO [CompactionExecutor:28] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-32-Data.db') (99237382 bytes)
           INFO [CompactionExecutor:25] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-23-Data.db') (76191460 bytes)
           INFO [CompactionExecutor:32] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-27-Data.db') (189539010 bytes)
           INFO [CompactionExecutor:31] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-37-Data.db') (27822796 bytes)
           INFO [CompactionExecutor:30] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-39-Data.db') (29473004 bytes)
           INFO [CompactionExecutor:29] 2014-03-22 14:29:37,391 OutputHandler.java (line 42) Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-38-Data.db') complete: 64971 rows in new sstable and 0 empty (tombstoned) rows dropped
           INFO [CompactionExecutor:31] 2014-03-22 14:29:40,882 OutputHandler.java (line 42) Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-37-Data.db') complete: 100082 rows in new sstable and 0 empty (tombstoned) rows dropped
           INFO [CompactionExecutor:30] 2014-03-22 14:29:41,383 OutputHandler.java (line 42) Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-39-Data.db') complete: 106018 rows in new sstable and 0 empty (tombstoned) rows dropped
           INFO [CompactionExecutor:25] 2014-03-22 14:29:52,562 OutputHandler.java (line 42) Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-23-Data.db') complete: 274070 rows in new sstable and 0 empty (tombstoned) rows dropped
           INFO [CompactionExecutor:28] 2014-03-22 14:29:56,753 OutputHandler.java (line 42) Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-32-Data.db') complete: 356969 rows in new sstable and 0 empty (tombstoned) rows dropped
           INFO [CompactionExecutor:26] 2014-03-22 14:29:58,021 OutputHandler.java (line 42) Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-40-Data.db') complete: 395706 rows in new sstable and 0 empty (tombstoned) rows dropped
           INFO [CompactionExecutor:32] 2014-03-22 14:30:02,841 OutputHandler.java (line 42) Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-27-Data.db') complete: 681795 rows in new sstable and 0 empty (tombstoned) rows dropped
          
          Show
          rspitzer Russell Spitzer added a comment - Thought I would make an attempt at this. Made all three operations request a thread per file. I would be glad to make any changes/improvements requested. Example of multithreaded scrubbing INFO [CompactionExecutor:29] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-38-Data.db') (18061938 bytes) INFO [CompactionExecutor:26] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-40-Data.db') (110006268 bytes) INFO [CompactionExecutor:28] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-32-Data.db') (99237382 bytes) INFO [CompactionExecutor:25] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-23-Data.db') (76191460 bytes) INFO [CompactionExecutor:32] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-27-Data.db') (189539010 bytes) INFO [CompactionExecutor:31] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-37-Data.db') (27822796 bytes) INFO [CompactionExecutor:30] 2014-03-22 14:29:30,783 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-39-Data.db') (29473004 bytes) INFO [CompactionExecutor:29] 2014-03-22 14:29:37,391 OutputHandler.java (line 42) Scrub of SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-38-Data.db') complete: 64971 rows in new sstable and 0 empty (tombstoned) rows dropped INFO [CompactionExecutor:31] 2014-03-22 14:29:40,882 OutputHandler.java (line 42) Scrub of SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-37-Data.db') complete: 100082 rows in new sstable and 0 empty (tombstoned) rows dropped INFO [CompactionExecutor:30] 2014-03-22 14:29:41,383 OutputHandler.java (line 42) Scrub of SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-39-Data.db') complete: 106018 rows in new sstable and 0 empty (tombstoned) rows dropped INFO [CompactionExecutor:25] 2014-03-22 14:29:52,562 OutputHandler.java (line 42) Scrub of SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-23-Data.db') complete: 274070 rows in new sstable and 0 empty (tombstoned) rows dropped INFO [CompactionExecutor:28] 2014-03-22 14:29:56,753 OutputHandler.java (line 42) Scrub of SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-32-Data.db') complete: 356969 rows in new sstable and 0 empty (tombstoned) rows dropped INFO [CompactionExecutor:26] 2014-03-22 14:29:58,021 OutputHandler.java (line 42) Scrub of SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-40-Data.db') complete: 395706 rows in new sstable and 0 empty (tombstoned) rows dropped INFO [CompactionExecutor:32] 2014-03-22 14:30:02,841 OutputHandler.java (line 42) Scrub of SSTableReader(path='/ var /lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-27-Data.db') complete: 681795 rows in new sstable and 0 empty (tombstoned) rows dropped
          Hide
          jbellis Jonathan Ellis added a comment -

          Thanks; will have a look soon.

          Show
          jbellis Jonathan Ellis added a comment - Thanks; will have a look soon.
          Hide
          jbellis Jonathan Ellis added a comment -

          Sorry, I want to review this but I'm overcommitted. Can you take review, Marcus?

          Show
          jbellis Jonathan Ellis added a comment - Sorry, I want to review this but I'm overcommitted. Can you take review, Marcus?
          Hide
          krummas Marcus Eriksson added a comment -

          Did a bit more refactoring on top of your commit

          now we only have to implement the performXYZ and then the xyzOne methods for these operations, what do you think?

          Show
          krummas Marcus Eriksson added a comment - Did a bit more refactoring on top of your commit now we only have to implement the performXYZ and then the xyzOne methods for these operations, what do you think?
          Hide
          rspitzer Russell Spitzer added a comment -

          Looks like a great idea to me, definitely more useful for future development.

          one nit
          *parallell*AllSSTableOperation -> *parallel*AllSSTableOperation

          Show
          rspitzer Russell Spitzer added a comment - Looks like a great idea to me, definitely more useful for future development. one nit *parallell*AllSSTableOperation -> *parallel*AllSSTableOperation
          Hide
          krummas Marcus Eriksson added a comment -

          We should probably only target 2.1 for this, patch pushed here: https://github.com/krummas/cassandra/commits/marcuse/5547-2, we changed things quite a bit in 2.1 so would be nice if you could have another look Russell Spitzer

          Show
          krummas Marcus Eriksson added a comment - We should probably only target 2.1 for this, patch pushed here: https://github.com/krummas/cassandra/commits/marcuse/5547-2 , we changed things quite a bit in 2.1 so would be nice if you could have another look Russell Spitzer
          Hide
          rspitzer Russell Spitzer added a comment -

          Looks good to me +1, I hope this will end up being useful for folks.

          Show
          rspitzer Russell Spitzer added a comment - Looks good to me +1, I hope this will end up being useful for folks.
          Hide
          krummas Marcus Eriksson added a comment -

          committed!

          Show
          krummas Marcus Eriksson added a comment - committed!
          Hide
          rcoli Robert Coli added a comment -

          This is super helpful for operators; thank you!

          Show
          rcoli Robert Coli added a comment - This is super helpful for operators; thank you!

            People

            • Assignee:
              rspitzer Russell Spitzer
              Reporter:
              bcoverston Benjamin Coverston
              Reviewer:
              Marcus Eriksson
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development