Uploaded image for project: 'Apache Cassandra'
  1. Apache Cassandra
  2. CASSANDRA-10478

Seek position is not within mmap segment

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Urgent
    • Resolution: Fixed
    • 2.1.11, 2.2.3
    • None
    • None
    • Cassandra 2.2.2
      Java 1.8.0.60

    • Critical

    Description

      After upgrading to 2.2.2 we started seeing timeouts accompanied by the following error in the log. Disabling mmap (by using "disk_access_mode: standard") completely solves the problem.

      We did not experience this problem in 2.2.1.
      The change to src/java/org/apache/cassandra/io/util/ByteBufferDataInput.java in the following commit seems interesting as it changes the calculation of the mmap boundaries (and moves from <= to <) https://github.com/apache/cassandra/commit/25de92e321604626d6c098233082904832c07814

      WARN  [SharedPool-Worker-1] 2015-10-07 03:40:39,771 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
      java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Seek position 717680 is not within mmap segment (seg offs: 0, length: 717680)
      	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2187) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
      	at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.2.2.jar:2.2.2]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
      Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Seek position 717680 is not within mmap segment (seg offs: 0, length: 717680)
      	at org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigTableReader.java:250) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.sstable.format.SSTableReader.getPosition(SSTableReader.java:1558) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.sstable.format.big.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:75) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:246) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:270) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:2004) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1808) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:360) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1537) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2183) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	... 4 common frames omitted
      Caused by: java.io.IOException: Seek position 717680 is not within mmap segment (seg offs: 0, length: 717680)
      	at org.apache.cassandra.io.util.ByteBufferDataInput.seek(ByteBufferDataInput.java:47) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.util.AbstractDataInput.skipBytes(AbstractDataInput.java:33) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:405) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.RowIndexEntry$Serializer.skipPromotedIndex(RowIndexEntry.java:164) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.db.RowIndexEntry$Serializer.skip(RowIndexEntry.java:155) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	at org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigTableReader.java:244) ~[apache-cassandra-2.2.2.jar:2.2.2]
      	... 17 common frames omitted
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            benedict Benedict Elliott Smith Assign to me
            mroi Omri Iluz
            Benedict Elliott Smith
            T Jake Luciani
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment