Cassandra
  1. Cassandra
  2. CASSANDRA-6285

2.0 HSHA server introduces corrupt data

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Fix Version/s: 2.0.8
    • Component/s: Core
    • Labels:
      None
    • Environment:

      4 nodes, shortly updated from 1.2.11 to 2.0.2

      Description

      After altering everything to LCS the table OpsCenter.rollups60 amd one other none OpsCenter-Table got stuck with everything hanging around in L0.
      The compaction started and ran until the logs showed this:
      ERROR [CompactionExecutor:111] 2013-11-01 19:14:53,865 CassandraDaemon.java (line 187) Exception in thread Thread[CompactionExecutor:111,1,RMI Runtime]
      java.lang.RuntimeException: Last written key DecoratedKey(1326283851463420237, 37382e34362e3132382e3139382d6a7576616c69735f6e6f72785f696e6465785f323031335f31305f30382d63616368655f646f63756d656e74736c6f6f6b75702d676574426c6f6f6d46696c746572537061636555736564) >= current key DecoratedKey(954210699457429663, 37382e34362e3132382e3139382d6a7576616c69735f6e6f72785f696e6465785f323031335f31305f30382d63616368655f646f63756d656e74736c6f6f6b75702d676574546f74616c4469736b5370616365557365640b0f) writing into /var/lib/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-tmp-jb-58656-Data.db
      at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:141)
      at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:164)
      at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:160)
      at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
      at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
      at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
      at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:296)
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:724)

      Moving back to STC worked to keep the compactions running.
      Especialy my own Table i would like to move to LCS.
      After a major compaction with STC the move to LCS fails with the same Exception.

      1. enable_reallocate_buffers.txt
        1 kB
        Brandon Williams
      2. disruptor-memory-corruption.patch
        0.6 kB
        Viktor Kuzmin
      3. disruptor-high-cpu.patch
        2 kB
        Viktor Kuzmin
      4. compaction_test.py
        3 kB
        Russ Hatch
      5. cassandra-attack-src.zip
        8.56 MB
        Viktor Kuzmin
      6. CASSANDRA-6285-disruptor-heap.patch
        1 kB
        Pavel Yaskevich
      7. 6285_testnotes1.txt
        115 kB
        Michael Shuler

        Activity

          People

          • Assignee:
            Pavel Yaskevich
            Reporter:
            David Sauer
            Reviewer:
            Brandon Williams
            Tester:
            Rick Branson
          • Votes:
            4 Vote for this issue
            Watchers:
            34 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development