[CASSANDRA-4446] nodetool drain sometimes doesn't mark commitlog fully flushed

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 1.2.1
• Component/s: Tool/nodetool

Environment

ubuntu 10.04 64bit
Linux HOSTNAME 2.6.32-345-ec2 #48-Ubuntu SMP Wed May 2 19:29:55 UTC 2012 x86_64 GNU/Linux
sun JVM
cassandra 1.0.10 installed from apache deb

• Severity: Low
• Assignee: Jonathan Ellis
• Reporter: Robert Coli

Authors

Jonathan Ellis

Reviewers

Yuki Morishita

• Created: 18/Jul/12 21:30
• Updated: 16/Apr/19 09:32
• Resolved: 15/Jan/13 15:14

Description

I recently wiped a customer's QA cluster. I drained each node and verified that they were drained. When I restarted the nodes, I saw the commitlog replay create a memtable and then flush it. I have attached a sanitized log snippet from a representative node at the time.

It appears to show the following :
1) Drain begins
2) Drain triggers flush
3) Flush triggers compaction
4) StorageService logs DRAINED message
5) compaction thread excepts
6) on restart, same CF creates a memtable
7) and then flushes it [1]

The columnfamily involved in the replay in 7) is the CF for which the compaction thread excepted in 5). This seems to suggest a timing issue whereby the exception in 5) prevents the flush in 3) from marking all the segments flushed, causing them to replay after restart.

In case it might be relevant, I did an online change of compaction strategy from Leveled to SizeTiered during the uptime period preceding this drain.

[1] Isn't commitlog replay not supposed to automatically trigger a flush in modern cassandra?

Issue Links

• is related to: CASSANDRA-5911 Commit logs are not removed after nodetool flush or nodetool drain