Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15327

Deleted data can re-appear if range movement streaming time exceeds gc_grace_seconds

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Correctness
    • Normal
    • Normal
    • None

    Description

      Hey,

      We've come across a scenario in production (noticed on Cassandra 2.2.14) where data that is deleted from Cassandra at consistency ALL can be resurrected.  I've added a reproduction in a comment.

      If a delete is issued during a range movement (i.e. bootstrap, decommission, move), and gc_grace_seconds is surpassed before the stream is finished, then the tombstones from the delete can be purged from the recipient node before the data is streamed. Once the move is complete, the data now exists on the recipient node without a tombstone.

      We noticed this because our bootstrapping time occasionally exceeds our configured gc_grace_seconds, so we lose the consistency guarantee.  As an operator, it would be great to not have to worry about this edge case.

      I've attached a patch that we have tested and successfully used in production, and haven't noticed any ill effects.  Happy to submit patches for more recent versions, I'm not sure how cleanly this will actually merge since there was some refactoring to this logic in 3.x.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            leonz Leon Zaruvinsky
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment