Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15369

Fake row deletions and range tombstones, causing digest mismatch and sstable growth

    XMLWordPrintableJSON

    Details

    • Bug Category:
      Correctness - Consistency
    • Severity:
      Normal
    • Complexity:
      Impossible
    • Discovered By:
      Code Inspection
    • Platform:
      All
    • Impacts:
      None

      Description

      As assessed in CASSANDRA-15363, we generate fake row deletions and fake tombstone markers under various circumstances:

      • If we perform a clustering key query (or select a compact column):
      • Serving from a Memtable, we will generate fake row deletions
      • Serving from an sstable, we will generate fake row tombstone markers
      • If we perform a slice query, we will generate only fake row tombstone markers for any range tombstone that begins or ends outside of the limit of the requested slice
      • If we perform a multi-slice or IN query, this will occur for each slice/clustering

      Unfortunately, these different behaviours can lead to very different data stored in sstables until a full repair is run.  When we read-repair, we only send these fake deletions or range tombstones.  A fake row deletion, clustering RT and slice RT, each produces a different digest.  So for each single point lookup we can produce a digest mismatch twice, and until a full repair is run we can encounter an unlimited number of digest mismatches across different overlapping queries.

      Relatedly, this seems a more problematic variant of our atomicity failures caused by our monotonic reads, since RTs can have an atomic effect across (up to) the entire partition, whereas the propagation may happen on an arbitrarily small portion.  If the RT exists on only one node, this could plausibly lead to fairly problematic scenario if that node fails before the range can be repaired. 

      At the very least, this behaviour can lead to an almost unlimited amount of extraneous data being stored until the range is repaired and compaction happens to overwrite the sub-range RTs and row deletions.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              benedict Benedict Elliott Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: