Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-4538

Strange CorruptedBlockException when massive insert binary data

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Urgent
    • Resolution: Cannot Reproduce
    • None
    • None
    • Debian sequeeze 32bit

    • Critical

    Description

      After inserting ~ 10000 records, here is the error log

      INFO 10:53:33,543 Compacted to [/var/lib/cassandra/data/ST/company/ST-company.company_acct_no_idx-he-13-Data.db,]. 407,681 to 409,133 (~100% of original) bytes for 9,250 keys at 0.715926MB/s. Time: 545ms.
      ERROR 10:53:35,445 Exception in thread Thread[CompactionExecutor:3,1,main]
      java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption detected, chunk at 7530128 of length 19575.
      at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116)
      at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:99)
      at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176)
      at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83)
      at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68)
      at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
      at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
      at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
      at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
      at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
      at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
      at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
      at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173)
      at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption detected, chunk at 7530128 of length 19575.
      at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:98)
      at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
      at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
      at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
      at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
      at org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95)
      at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
      at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363)
      at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:119)
      at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
      at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
      at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234)
      at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:112)
      ... 20 more

      Here is the startup of cassandra
      root@cassandra-desktop:~# cassandra -f
      xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms496M -Xmx496M -Xmn124M -XX:+HeapDumpOnOutOfMemoryError -Xss128k
      INFO 10:56:37,113 Logging initialized
      INFO 10:56:37,122 JVM vendor/version: Java HotSpot(TM) Client VM/1.6.0_26
      INFO 10:56:37,123 Heap size: 507117568/507117568
      INFO 10:56:37,123 Classpath: /etc/cassandra:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.3.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.3.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
      INFO 10:56:37,126 JNA not found. Native methods will be disabled.
      INFO 10:56:37,143 Loading settings from file:/etc/cassandra/cassandra.yaml

      Attached is the test case

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            tomcheng76 Tommy Cheng
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment