Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-13008

Add vm.max_map_count StartupCheck

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Low
    • Resolution: Fixed
    • Fix Version/s: 3.0.11, 3.11.0
    • Component/s: Local/Config
    • Labels:
      None

      Description

      It's recommended to set vm.max_map_count to 1048575 (CASSANDRA-3563)
      When the max_map_count is low, it throws OOM exception, which is hard to link to the real issue of vm.max_map_count.
      The problem happened when we tried to remove one node, all the other nodes in cluster crashed. As each node was trying to load more local SSTable files for streaming.

      I would suggest to add a StartupCheck for max_map_count, at least it could give a warning message to help the debug.

      ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to be unstable.  Exiting forcefully due to:
      java.lang.OutOfMemoryError: Map failed
          at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
          at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_112]
          at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296) ~[apache-cassandra-3.0.10.jar:3.0.10]
          at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
      

        Attachments

        1. 13008-3.0.txt
          4 kB
          Jay Zhuang

          Activity

            People

            • Assignee:
              jay.zhuang Jay Zhuang
              Reporter:
              jay.zhuang Jay Zhuang
              Authors:
              Jay Zhuang
              Reviewers:
              Stefania Alborghetti
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: