Cassandra
  1. Cassandra
  2. CASSANDRA-1074

check bloom filters to make minor compaction able to delete (some) tombstones

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 0.6.6, 0.7 beta 2
    • Component/s: Core
    • Labels:
      None

      Description

      Given a tombstoned key which is older than GCGraceSeconds, current (0.6.1) compaction implementation still requires a major compaction for the key to actually be deleted. The major compaction is required is because we must know whether there is a version of the key inside all SSTables associated with the columnfamily, including ones not involved in minor compactions. However, as we have bloom filters into each one of these SSTables, minor compaction can relatively inexpensively check for existence of this key in SSTable files not involved in the current minor compaction, and thereby delete the key, assuming all bloom filters return negative. If the filter returns positive, a major compaction would of course still be required.

      For use cases like CASSANDRA-1041 where users are strongly motivated to not do a major compaction, this seems to answer the jbellis objection :

      "You don't want to skip large files in major compactions, since the definition of major is "compact everything so it is safe to remove tombstones." "

      The above described improvement appears to provide "safe to remove (some) tombstones" without requiring "compact everything", and so may be a useful optimization.

      =Rob

        Activity

        Hide
        Jonathan Ellis added a comment -

        committed, thanks!

        Show
        Jonathan Ellis added a comment - committed, thanks!
        Hide
        Sylvain Lebresne added a comment -

        Attaching patch for 0.6 (sorry for the delay)

        Show
        Sylvain Lebresne added a comment - Attaching patch for 0.6 (sorry for the delay)
        Hide
        Jonathan Ellis added a comment -

        committed w/ minor changes.

        happy to merge backport to 0.6 as well.

        Thanks Sylvain!

        Show
        Jonathan Ellis added a comment - committed w/ minor changes. happy to merge backport to 0.6 as well. Thanks Sylvain!
        Hide
        Sylvain Lebresne added a comment -

        I think that's a neat optimisation. It may not remove the need for major compaction, but it will quite probably make it less frequent.

        Attaching a patch against trunk. If it gets some love and there is a willingness to include
        this in the 0.6 branch, I'll port the patch.

        Show
        Sylvain Lebresne added a comment - I think that's a neat optimisation. It may not remove the need for major compaction, but it will quite probably make it less frequent. Attaching a patch against trunk. If it gets some love and there is a willingness to include this in the 0.6 branch, I'll port the patch.
        Hide
        Robert Coli added a comment -

        fixed -> unknown.

        Show
        Robert Coli added a comment - fixed -> unknown.
        Hide
        Jonathan Ellis added a comment -

        Are you tagging this 0.6.2 because you plan to address it relatively quickly, or are you just being optimistic?

        Show
        Jonathan Ellis added a comment - Are you tagging this 0.6.2 because you plan to address it relatively quickly, or are you just being optimistic?

          People

          • Assignee:
            Sylvain Lebresne
            Reporter:
            Robert Coli
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development