Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10478

Seek position is not within mmap segment

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Urgent
    • Resolution: Fixed
    • Fix Version/s: 2.1.11, 2.2.3
    • Component/s: None
    • Labels:
      None
    • Environment:

      Cassandra 2.2.2
      Java 1.8.0.60

    • Severity:
      Critical
    • Since Version:

      Description

      After upgrading to 2.2.2 we started seeing timeouts accompanied by the following error in the log. Disabling mmap (by using "disk_access_mode: standard") completely solves the problem.

      We did not experience this problem in 2.2.1.
      The change to src/java/org/apache/cassandra/io/util/ByteBufferDataInput.java in the following commit seems interesting as it changes the calculation of the mmap boundaries (and moves from <= to <) https://github.com/apache/cassandra/commit/25de92e321604626d6c098233082904832c07814

      WARN  [SharedPool-Worker-1] 2015-10-07 03:40:39,771 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
      java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Seek position 717680 is not within mmap segment (seg offs: 0, length: 717680)
      	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2187) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
      	at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.2.2.jar:2.2.2]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
      Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Seek position 717680 is not within mmap segment (seg offs: 0, length: 717680)
      	at org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigTableReader.java:250) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.sstable.format.SSTableReader.getPosition(SSTableReader.java:1558) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.sstable.format.big.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:75) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:246) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:270) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:2004) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1808) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:360) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1537) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2183) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	... 4 common frames omitted
      Caused by: java.io.IOException: Seek position 717680 is not within mmap segment (seg offs: 0, length: 717680)
      	at org.apache.cassandra.io.util.ByteBufferDataInput.seek(ByteBufferDataInput.java:47) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.util.AbstractDataInput.skipBytes(AbstractDataInput.java:33) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:405) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.RowIndexEntry$Serializer.skipPromotedIndex(RowIndexEntry.java:164) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.RowIndexEntry$Serializer.skip(RowIndexEntry.java:155) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigTableReader.java:244) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	... 17 common frames omitted
      

        Attachments

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              benedict Benedict Elliott Smith Assign to me
              Reporter:
              mroi Omri Iluz
              Authors:
              Benedict Elliott Smith
              Reviewers:
              T Jake Luciani

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment