Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Duplicate
-
None
-
None
-
None
-
Reproduced using ccm and Cassandra 2.1.12
-
Normal
Description
When repairing, it is impossible to repair using subranges and a specific table at the same time.
When running this:
date && echo "Repairing standard1 on 127.0.0.1" && time nodetool -h localhost -p 7100 repair -dc datacenter1 -local -par -- keyspace1 standard1
Without -st / -et options, I have the following output:
MacBook-Pro:~ alain$ tail -100f ~/.ccm/test-2.1.12/node1/logs/system.log INFO [Thread-33] 2016-06-09 14:18:52,193 StorageService.java:2939 - Starting repair command #8, repairing 3 ranges for keyspace keyspace1 (parallelism=PARALLEL, full=true) INFO [AntiEntropySessions:12] 2016-06-09 14:18:52,194 RepairSession.java:260 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] new session: will sync /127.0.0.1, /127.0.0.2, /127.0.0.3 on range (3074457345618258602,-9223372036854775808] for keyspace1.[standard1] INFO [AntiEntropySessions:12] 2016-06-09 14:18:52,195 RepairJob.java:163 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] requesting merkle trees for standard1 (to [/127.0.0.2, /127.0.0.3, /127.0.0.1]) INFO [AntiEntropyStage:1] 2016-06-09 14:18:57,433 RepairSession.java:171 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.2 INFO [AntiEntropyStage:1] 2016-06-09 14:18:57,436 RepairSession.java:171 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.3 INFO [AntiEntropyStage:1] 2016-06-09 14:18:57,439 RepairSession.java:171 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.1 INFO [AntiEntropySessions:13] 2016-06-09 14:18:57,439 RepairSession.java:260 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] new session: will sync /127.0.0.1, /127.0.0.2, /127.0.0.3 on range (-9223372036854775808,-3074457345618258603] for keyspace1.[standard1] INFO [RepairJobTask:1] 2016-06-09 14:18:57,440 Differencer.java:67 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.3 are consistent for standard1 INFO [RepairJobTask:3] 2016-06-09 14:18:57,440 Differencer.java:67 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.3 and /127.0.0.1 are consistent for standard1 INFO [RepairJobTask:2] 2016-06-09 14:18:57,440 Differencer.java:67 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.1 are consistent for standard1 INFO [AntiEntropySessions:13] 2016-06-09 14:18:57,440 RepairJob.java:163 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] requesting merkle trees for standard1 (to [/127.0.0.2, /127.0.0.3, /127.0.0.1]) INFO [AntiEntropyStage:1] 2016-06-09 14:18:57,440 RepairSession.java:237 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] standard1 is fully synced INFO [AntiEntropySessions:12] 2016-06-09 14:18:57,440 RepairSession.java:299 - [repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] session completed successfully INFO [AntiEntropyStage:1] 2016-06-09 14:19:03,676 RepairSession.java:171 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.2 INFO [AntiEntropyStage:1] 2016-06-09 14:19:03,684 RepairSession.java:171 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.3 INFO [AntiEntropyStage:1] 2016-06-09 14:19:03,758 RepairSession.java:171 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.1 INFO [AntiEntropySessions:14] 2016-06-09 14:19:03,759 RepairSession.java:260 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] new session: will sync /127.0.0.1, /127.0.0.2, /127.0.0.3 on range (-3074457345618258603,3074457345618258602] for keyspace1.[standard1] INFO [RepairJobTask:1] 2016-06-09 14:19:03,759 Differencer.java:67 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.3 are consistent for standard1 INFO [AntiEntropySessions:14] 2016-06-09 14:19:03,759 RepairJob.java:163 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] requesting merkle trees for standard1 (to [/127.0.0.2, /127.0.0.3, /127.0.0.1]) INFO [RepairJobTask:3] 2016-06-09 14:19:03,760 Differencer.java:67 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.3 and /127.0.0.1 are consistent for standard1 INFO [RepairJobTask:2] 2016-06-09 14:19:03,760 Differencer.java:67 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.1 are consistent for standard1 INFO [AntiEntropyStage:1] 2016-06-09 14:19:03,760 RepairSession.java:237 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] standard1 is fully synced INFO [AntiEntropySessions:13] 2016-06-09 14:19:03,760 RepairSession.java:299 - [repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] session completed successfully INFO [AntiEntropyStage:1] 2016-06-09 14:19:09,922 RepairSession.java:171 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.2 INFO [AntiEntropyStage:1] 2016-06-09 14:19:09,925 RepairSession.java:171 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.3 INFO [AntiEntropyStage:1] 2016-06-09 14:19:10,237 RepairSession.java:171 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.1 INFO [Thread-33] 2016-06-09 14:19:10,237 StorageService.java:3026 - Repair session 53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e for range (3074457345618258602,-9223372036854775808] finished INFO [Thread-33] 2016-06-09 14:19:10,238 StorageService.java:3026 - Repair session 57074af0-2e3c-11e6-95ae-d1beb0ba4c9e for range (-9223372036854775808,-3074457345618258603] finished INFO [RepairJobTask:1] 2016-06-09 14:19:10,238 Differencer.java:67 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.3 are consistent for standard1 INFO [RepairJobTask:2] 2016-06-09 14:19:10,238 Differencer.java:67 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.1 are consistent for standard1 INFO [RepairJobTask:3] 2016-06-09 14:19:10,238 Differencer.java:67 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.3 and /127.0.0.1 are consistent for standard1 INFO [AntiEntropyStage:1] 2016-06-09 14:19:10,238 RepairSession.java:237 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] standard1 is fully synced INFO [AntiEntropySessions:14] 2016-06-09 14:19:10,239 RepairSession.java:299 - [repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] session completed successfully INFO [Thread-33] 2016-06-09 14:19:10,239 StorageService.java:3026 - Repair session 5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e for range (-3074457345618258603,3074457345618258602] finished
MacBook-Pro:~ alain$ for i in $(echo "SELECT columnfamily_name FROM system.schema_columns WHERE keyspace_name = 'keyspace1';" | cqlsh | uniq | grep standard); do echo Session synced for $i: $(grep -i "$i is fully synced" ~/.ccm/test-2.1.12/node1/logs/system.log* | wc -l); done Session synced for standard1: 3 Session synced for standard2: 0
Using -st and -et options, as follow:
date && echo "Repairing standard1 on 127.0.0.1" && time nodetool -h localhost -p 7100 repair -dc datacenter1 -local -par -st '-3074457345618258603' -et 3074457345618258602 -- keyspace1 standard1
I have this output:
INFO [RMI TCP Connection(67)-127.0.0.1] 2016-06-09 14:23:10,937 StorageService.java:2846 - starting user-requested repair of range [(-3074457345618258603,3074457345618258602]] for keyspace keyspace1 and column families [] INFO [Thread-35] 2016-06-09 14:23:10,937 StorageService.java:2939 - Starting repair command #9, repairing 1 ranges for keyspace keyspace1 (parallelism=PARALLEL, full=true) INFO [AntiEntropySessions:15] 2016-06-09 14:23:10,938 RepairSession.java:260 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] new session: will sync /127.0.0.1, /127.0.0.2, /127.0.0.3 on range (-3074457345618258603,3074457345618258602] for keyspace1.[standard1, standard2] INFO [AntiEntropySessions:15] 2016-06-09 14:23:10,938 RepairJob.java:163 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] requesting merkle trees for standard1 (to [/127.0.0.2, /127.0.0.3, /127.0.0.1]) INFO [AntiEntropyStage:1] 2016-06-09 14:23:16,184 RepairSession.java:171 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.3 INFO [AntiEntropyStage:1] 2016-06-09 14:23:16,190 RepairSession.java:171 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.2 INFO [AntiEntropyStage:1] 2016-06-09 14:23:16,417 RepairSession.java:171 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard1 from /127.0.0.1 INFO [AntiEntropyStage:1] 2016-06-09 14:23:16,418 RepairJob.java:163 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] requesting merkle trees for standard2 (to [/127.0.0.2, /127.0.0.3, /127.0.0.1]) INFO [RepairJobTask:1] 2016-06-09 14:23:16,418 Differencer.java:67 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.3 and /127.0.0.2 are consistent for standard1 INFO [RepairJobTask:3] 2016-06-09 14:23:16,419 Differencer.java:67 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.1 are consistent for standard1 INFO [RepairJobTask:2] 2016-06-09 14:23:16,419 Differencer.java:67 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.3 and /127.0.0.1 are consistent for standard1 INFO [AntiEntropyStage:1] 2016-06-09 14:23:16,419 RepairSession.java:237 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] standard1 is fully synced INFO [AntiEntropyStage:1] 2016-06-09 14:23:22,439 RepairSession.java:171 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard2 from /127.0.0.2 INFO [AntiEntropyStage:1] 2016-06-09 14:23:22,440 RepairSession.java:171 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard2 from /127.0.0.3 INFO [AntiEntropyStage:1] 2016-06-09 14:23:22,748 RepairSession.java:171 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for standard2 from /127.0.0.1 INFO [RepairJobTask:2] 2016-06-09 14:23:22,748 Differencer.java:67 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.3 are consistent for standard2 INFO [RepairJobTask:3] 2016-06-09 14:23:22,749 Differencer.java:67 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.1 are consistent for standard2 INFO [RepairJobTask:1] 2016-06-09 14:23:22,749 Differencer.java:67 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.3 and /127.0.0.1 are consistent for standard2 INFO [AntiEntropyStage:1] 2016-06-09 14:23:22,749 RepairSession.java:237 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] standard2 is fully synced INFO [AntiEntropySessions:15] 2016-06-09 14:23:22,749 RepairSession.java:299 - [repair #ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e] session completed successfully INFO [Thread-35] 2016-06-09 14:23:22,749 StorageService.java:3026 - Repair session ee202ba0-2e3c-11e6-95ae-d1beb0ba4c9e for range (-3074457345618258603,3074457345618258602] finished
All the data from all the tables has been repaired in this range, when I expected to only repair one table.
MacBook-Pro:~ alain$ for i in $(echo "SELECT columnfamily_name FROM system.schema_columns WHERE keyspace_name = 'keyspace1';" | cqlsh | uniq | grep standard); do echo Session synced for $i: $(grep -i "$i is fully synced" ~/.ccm/MAS-2530-2.1.12/node1/logs/system.log* | wc -l); done Session synced for standard1: 1 Session synced for standard2: 1
This work in latest 2.0, for what it is worth. More info here: https://github.com/BrianGallew/cassandra_range_repair/issues/31