Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-4538

Strange CorruptedBlockException when massive insert binary data

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Urgent
    • Resolution: Cannot Reproduce
    • Fix Version/s: None
    • Component/s: None
    • Environment:

      Debian sequeeze 32bit

    • Severity:
      Critical

      Description

      After inserting ~ 10000 records, here is the error log

      INFO 10:53:33,543 Compacted to [/var/lib/cassandra/data/ST/company/ST-company.company_acct_no_idx-he-13-Data.db,]. 407,681 to 409,133 (~100% of original) bytes for 9,250 keys at 0.715926MB/s. Time: 545ms.
      ERROR 10:53:35,445 Exception in thread Thread[CompactionExecutor:3,1,main]
      java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption detected, chunk at 7530128 of length 19575.
      at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116)
      at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:99)
      at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176)
      at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83)
      at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68)
      at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
      at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
      at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
      at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
      at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
      at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
      at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
      at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173)
      at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption detected, chunk at 7530128 of length 19575.
      at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:98)
      at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
      at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
      at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
      at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
      at org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95)
      at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
      at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363)
      at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:119)
      at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
      at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
      at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234)
      at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:112)
      ... 20 more

      Here is the startup of cassandra
      root@cassandra-desktop:~# cassandra -f
      xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms496M -Xmx496M -Xmn124M -XX:+HeapDumpOnOutOfMemoryError -Xss128k
      INFO 10:56:37,113 Logging initialized
      INFO 10:56:37,122 JVM vendor/version: Java HotSpot(TM) Client VM/1.6.0_26
      INFO 10:56:37,123 Heap size: 507117568/507117568
      INFO 10:56:37,123 Classpath: /etc/cassandra:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.3.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.3.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
      INFO 10:56:37,126 JNA not found. Native methods will be disabled.
      INFO 10:56:37,143 Loading settings from file:/etc/cassandra/cassandra.yaml

      Attached is the test case

        Attachments

        1. cassandra-stresstest.zip
          41 kB
          Tommy Cheng

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tomcheng76 Tommy Cheng
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: