[CASSANDRA-8333] Streaming Error during repair

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate

Environment

Windows-7-32 bit, 3GB RAM, Java 1.7.0_55

• Severity: Normal
• Since Version: 2.1.2
• Assignee: Unassigned Assign to me
• Reporter: Andreas Schnitzerling
• Created: 18/Nov/14 11:55
• Updated: 16/Apr/19 09:31
• Resolved: 01/Dec/14 18:05

Description

During repair, connections are closing and throwing exceptions. CPU is running on 100%, when error occurs. My test-configuration is one node w/ 2.1.2 and 11 nodes w/ 2.0.11. If I make repair either on 2.1 or 2.0 I get such an error. But if I have 2.0 everywhere istalled, no error. 2.0 nodes make "endless" repair in that circumstance. Seems to be incompatibility between 2.0 and 2.1.

system.log

ERROR [STREAM-OUT-/10.6.8.212] 2014-11-18 12:28:34,948 StreamSession.java:472 - Stream #0866dc80-6f16-11e4-bc5c-5fe413b6852c Streaming error occurred
java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert
durch den Hostcomputer abgebrochen
at sun.nio.ch.SocketDispatcher.write0(Native Method) ~[na:1.7.0_55]
at sun.nio.ch.SocketDispatcher.write(Unknown Source) ~[na:1.7.0_55]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source) ~[na:1.7.0_55]
at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.7.0_55]
at sun.nio.ch.SocketChannelImpl.write(Unknown Source) ~[na:1.7.0_55]
at org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:346) [apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:326) [apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]
ERROR [AntiEntropySessions:1] 2014-11-18 12:28:34,948 RepairSession.java:303 - repair #e10d0240-6f15-11e4-bc5c-5fe413b6852c session completed with the following error
org.apache.cassandra.exceptions.RepairException: repair #e10d0240-6f15-11e4-bc5c-5fe413b6852c on logdata/onlinedata, (-143721749331492309,-139544903266258032] Sync failed between /10.9.9.241 and /10.6.8.212
at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:389) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:126) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_55]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]

Since in windows only parallel repair is possible, is there a way to throttle CPU-consumption? I reduced rpc_X_threads to 4 and concurrent_reads/writes to 4. But no change. On other nodes is C* 2.0.10 and nothing in their system.log.

Issue Links

• duplicates: CASSANDRA-8110 Make streaming forward & backwards compatible
• duplicates: CASSANDRA-7530 Repair should be allowed in mixed version clusters