[CASSANDRA-7385] sstableloader OutOfMemoryError: Java heap space

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Problem
• Component/s: Legacy/Tools
• Severity: Normal
• Assignee: Unassigned
• Reporter: Mike Heffner
• Created: 12/Jun/14 17:59
• Updated: 16/Apr/19 09:31
• Resolved: 12/Jun/14 20:29

Description

We hit the following exception with sstableloader while attempting to bulk load about 100GB of SSTs. We are now employing this workaround before starting an sstableloader run:

sed -i -e 's/-Xmx256M/-Xmx8G/g' /usr/bin/sstableloader

ERROR 19:25:45,060 Error in ThreadPoolExecutor
java.lang.OutOfMemoryError: Java heap space
	at org.apache.cassandra.io.util.FastByteArrayOutputStream.expand(FastByteArrayOutputStream.java:104)
	at org.apache.cassandra.io.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:235)
	at java.io.DataOutputStream.writeInt(DataOutputStream.java:199)
	at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:412)
	at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:407)
	at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:59)
	at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:46)
	at org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.serialize(PendingFile.java:142)
	at org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:67)
	at org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:58)
	at org.apache.cassandra.net.MessagingService.constructStreamHeader(MessagingService.java:782)
	at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:65)
	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "Streaming to /10.167.a.b:1" java.lang.OutOfMemoryError: Java heap space
	at org.apache.cassandra.io.util.FastByteArrayOutputStream.expand(FastByteArrayOutputStream.java:104)
	at org.apache.cassandra.io.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:235)
	at java.io.DataOutputStream.writeInt(DataOutputStream.java:199)
	at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:412)
	at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:407)
	at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:59)
	at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:46)
	at org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.serialize(PendingFile.java:142)
	at org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:67)
	at org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:58)
	at org.apache.cassandra.net.MessagingService.constructStreamHeader(MessagingService.java:782)
	at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:65)
	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.apache.cassandra.io.compress.CompressionMetadata.getChunksForSections(CompressionMetadata.java:210)
	at org.apache.cassandra.streaming.StreamOut.createPendingFiles(StreamOut.java:182)
	at org.apache.cassandra.streaming.StreamOut.transferSSTables(StreamOut.java:157)
	at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:145)
	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:67)

Issue Links

• is superceded by: CASSANDRA-5969 Allow JVM_OPTS to be passed to sstablescrub