[CASSANDRA-9398] Failed during snapshot creation while repairing

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate

Environment

Cassandra 2.1.5
Debian Wheezy

• Severity: Normal
• Since Version: 2.1.5
• Assignee: Yuki Morishita
• Reporter: Rafał Furmański

Authors

Yuki Morishita

• Created: 15/May/15 07:19
• Updated: 16/Apr/19 09:31
• Resolved: 15/May/15 23:03

Description

Hi,

during nodetool repair I'm always getting a lot of these errors on every node in my cluster:

ERROR [RepairJobTask:6] 2015-05-15 07:14:32,004 RepairJob.java:145 - Error occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.195.15.166
	at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77) ~[apache-cassandra-2.1.5.jar:2.1.5]
	at org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:349) ~[apache-cassandra-2.1.5.jar:2.1.5]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_80]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
ERROR [AntiEntropySessions:349] 2015-05-15 07:14:32,006 RepairSession.java:303 - [repair #cc9e41f0-fad1-11e4-9359-4963524a8bde] session completed with the following error
java.io.IOException: Failed during snapshot creation.
	at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.5.jar:2.1.5]
	at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.5.jar:2.1.5]
	at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
INFO  [AntiEntropySessions:350] 2015-05-15 07:14:32,006 RepairSession.java:260 - [repair #0876ff50-fad2-11e4-9359-4963524a8bde] new session: will sync /10.195.15.162, /10.195.15.167, /10.195.15.166 on range (-3856177530820600213,-3825198336844278876] for sync.[user_stats, entity, user_store, user_device, entity_by_id, user_device_progress, user_quota]
ERROR [AntiEntropySessions:349] 2015-05-15 07:14:32,007 CassandraDaemon.java:223 - Exception in thread Thread[AntiEntropySessions:349,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.5.jar:2.1.5]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_80]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
Caused by: java.io.IOException: Failed during snapshot creation.
	at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.5.jar:2.1.5]
	at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.5.jar:2.1.5]
	at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]
	... 3 common frames omitted

Any ideas what could be wrong?

Issue Links

• is duplicated by: CASSANDRA-8696 nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot