[CASSANDRA-12009] Nodetool scrub crashes: java.lang.OutOfMemoryError: Map failed - ASF JIRA

Details

Type: Bug
Status: Open
Priority: Normal
Resolution: Unresolved
Fix Version/s: None
Component/s: Tool/nodetool
Labels:
None
Environment:
Hide

java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

uname -a
Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

rpm -qa datastax-ddc
datastax-ddc-3.5.0-1.noarch
Show
java -version openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) uname -a Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux rpm -qa datastax-ddc datastax-ddc-3.5.0-1.noarch

Severity:
Normal

Description

Running nodetool scrub fails after a day or two. Originally, the instance had 8GB of RAM with 4GB of Xmx/Xms. After seeing the OOM error several times, I upgraded the box to a 24GB RAM box, giving 20GB to Xms/Xmx to see if it would finish the scrub operation. Both cases failed with the same error as below (tried on both the CMS and G1 GC collectors):

INFO  [Service Thread] 2016-06-14 15:01:17,968 StatusLogger.java:106 - system_traces.events                      0,0
DEBUG [GossipTasks:1] 2016-06-14 15:01:18,019 Gossiper.java:336 - Convicting /10.240.0.4 with status removed - alive false
WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,151 OutputHandler.java:57 - Error reading row (stacktrace follows):
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314) [apache-cassandra-3.5.0.jar:3.5.0]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_91]
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.5.0.jar:3.5.0]
        ... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_91]
        ... 24 common frames omitted
WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,152 OutputHandler.java:52 - Row starting at position 31397042159 is unreadable; skipping to next
WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,257 OutputHandler.java:57 - Error reading row (stacktrace follows):
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314) [apache-cassandra-3.5.0.jar:3.5.0]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_91]
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.5.0.jar:3.5.0]
        ... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_91]
        ... 24 common frames omitted
WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,258 OutputHandler.java:52 - Row starting at position 31397303324 is unreadable; skipping to next
WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:57 - Error reading row (stacktrace follows):
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) ~[apache-cassandra-3.5.0.jar:3.5Gq
        at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224) ~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314) [apache-cassandra-3.5.0.jar:3.5.0]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_91]
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.5.0.jar:3.5.0]
        ... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_91]
        ... 24 common frames omitted
WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:52 - Row starting at position 31397313498 is unreadable; skipping to next

Nodetool scrub crashes: java.lang.OutOfMemoryError: Map failed

Details

Description

Attachments

Activity

People

Dates