[CASSANDRA-4022] Compaction of hints can get stuck in a loop

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 1.2.0 beta 1
• Severity: Critical
• Assignee: Yuki Morishita
• Reporter: Brandon Williams

Authors

Yuki Morishita

Reviewers

Jonathan Ellis

• Created: 08/Mar/12 17:46
• Updated: 16/Apr/19 09:32
• Resolved: 31/Mar/12 04:45

Description

Not exactly sure how I caused this as I was working on something else in trunk, but:

 INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
 INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes 
for 1 keys at 5.912220MB/s.  Time: 748ms.
 INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
 INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes 
for 1 keys at 5.479976MB/s.  Time: 807ms.
 INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
 INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes 
for 1 keys at 4.786083MB/s.  Time: 924ms.
 INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
 INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
 INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes 
for 1 keys at 4.485132MB/s.  Time: 986ms.
 INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
 INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
 INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes 
for 1 keys at 4.223821MB/s.  Time: 1,047ms.
 INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
 INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
 INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes 
for 1 keys at 4.526449MB/s.  Time: 977ms.
 INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
 INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes 
for 1 keys at 6.263938MB/s.  Time: 706ms.
 INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
 INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s.  Time: 729ms.
 INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
 INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s.  Time: 758ms.
 INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
 INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s.  Time: 929ms.
 INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
 INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
 INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,].  4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s.  Time: 1,006ms.
 INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
 INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
 INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,].  4,637,160 to 4,637

I suspect we broke something subtle in CASSANDRA-3955.