[CASSANDRA-9702] Repair running really slow

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce
• Component/s: Legacy/Streaming and Messaging

Environment

C* 2.1.7, Debian Wheezy

• Severity: Normal
• Assignee: Unassigned Assign to me
• Reporter: mlowicki
• Created: 01/Jul/15 20:36
• Updated: 16/Apr/19 09:31
• Resolved: 22/Jul/16 17:05

Description

We're using 2.1.x since the very beginning and we always had problem with failing or slow repair. In one data center we aren't able to finish repair for many weeks (partially because CASSANDRA-9681 as we needed to reboot nodes periodically).

I've launched it today morning (12 hours now) and monitor using https://github.com/spotify/cassandra-opstools/blob/master/bin/spcassandra-repairstats. For the first hour it progressed to 9.43% but then it took ~10 hours to reach 9.44%. I see very rarely logs related to repair (each 15-20 minutes but sometimes nothing new for 1 hour).

Repair launched with:

nodetool repair --partitioner-range --parallel --in-local-dc {keyspace}

Attached log file from today.

We've ~4.1TB of data in 12 nodes with RF set to 3 (2 DC with 6 nodes each).