Cassandra
  1. Cassandra
  2. CASSANDRA-3399

Truncate disregards running compactions when deleting sstables

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 0.8.8, 1.0.2
    • Component/s: Core
    • Labels:
      None

      Description

      All truncation do is `cfs.markCompacted(truncatedSSTables)` without holding any lock or anything. Which have the effect of actually deleting sstables that may be compacting. More precisely there is three problems:

      1. It removes those compacting sstables from the current set of active sstables for the cfs. But when they are done compacting, DataTracker.replaceCompactedSSTables() will be called and it assumes that the compacted sstable are parts of the current set of active sstables. In other words, we'll get an exception looking like the one of CASSANDRA-3306.
      2. The result of the compaction will be added as a new active sstable (actually no, because the code will throw an exception before because of the preceding point, but that's something we should probably deal with).
      3. Currently, compaction don't 'acquire references' on SSTR. That's because the code assumes we won't compact twice the same sstable and that compaction is the only mean to delete an sstable. With these two assumption, acquiring references is not necessary, but truncate break that first assumption.

      As for solution, I see two possibilities:

      1. make the compaction lock be per-cf instead of global (which I think is easy and a good idea anyway) and grab the write lock to do the markCompacted call. The big downside is that truncation will potentially take much longer.
      2. had two phases: mark the sstable that are not compacting as compacted and set the dataTracker as 'truncated at', and let it deal with the other sstable when their compaction is done. A bit like what is proposed for CASSANDRA-3116
      1. 3399.txt
        2 kB
        Jonathan Ellis

        Activity

        Gavin made changes -
        Workflow patch-available, re-open possible [ 12749073 ] reopen-resolved, no closed status, patch-avail, testing [ 12756811 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12639204 ] patch-available, re-open possible [ 12749073 ]
        Hide
        Hudson added a comment -

        Integrated in Cassandra-0.8 #391 (See https://builds.apache.org/job/Cassandra-0.8/391/)
        acquire compactionlock during truncate
        patch by jbellis; reviewed by slebresne for CASSANDRA-3399

        jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195695
        Files :

        • /cassandra/branches/cassandra-0.8/CHANGES.txt
        • /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
        Show
        Hudson added a comment - Integrated in Cassandra-0.8 #391 (See https://builds.apache.org/job/Cassandra-0.8/391/ ) acquire compactionlock during truncate patch by jbellis; reviewed by slebresne for CASSANDRA-3399 jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195695 Files : /cassandra/branches/cassandra-0.8/CHANGES.txt /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
        Jonathan Ellis made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Jonathan Ellis added a comment -

        committed

        Show
        Jonathan Ellis added a comment - committed
        Hide
        Sylvain Lebresne added a comment -

        +1

        Show
        Sylvain Lebresne added a comment - +1
        Jonathan Ellis made changes -
        Attachment 3399.txt [ 12501633 ]
        Hide
        Jonathan Ellis added a comment -

        new patch w/ less crack smoking

        Show
        Jonathan Ellis added a comment - new patch w/ less crack smoking
        Jonathan Ellis made changes -
        Attachment 3399.txt [ 12501631 ]
        Jonathan Ellis made changes -
        Assignee Jonathan Ellis [ jbellis ]
        Hide
        Jonathan Ellis added a comment -

        (patch is against 1.0, will also commit to 0.8)

        Show
        Jonathan Ellis added a comment - (patch is against 1.0, will also commit to 0.8)
        Jonathan Ellis made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.7.0 [ 12316026 ]
        Affects Version/s 1.0.0 [ 12316349 ]
        Reviewer slebresne
        Fix Version/s 0.8.8 [ 12318351 ]
        Jonathan Ellis made changes -
        Field Original Value New Value
        Attachment 3399.txt [ 12501631 ]
        Hide
        Jonathan Ellis added a comment -

        big lock patch attached.

        Show
        Jonathan Ellis added a comment - big lock patch attached.
        Hide
        Jonathan Ellis added a comment -
        Show
        Jonathan Ellis added a comment - created CASSANDRA-3429 and CASSANDRA-3430 .
        Hide
        Jonathan Ellis added a comment -

        I think we should:

        • grab the Big Lock for this ticket
        • create another ticket to take a 3116 approach in 1.1
        • create another ticket to break the Big Lock apart to a per-CF lock in 1.0.2
        Show
        Jonathan Ellis added a comment - I think we should: grab the Big Lock for this ticket create another ticket to take a 3116 approach in 1.1 create another ticket to break the Big Lock apart to a per-CF lock in 1.0.2
        Sylvain Lebresne created issue -

          People

          • Assignee:
            Jonathan Ellis
            Reporter:
            Sylvain Lebresne
            Reviewer:
            Sylvain Lebresne
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development