[CASSANDRA-4782] Commitlog not replayed after restart

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 1.1.6
• Severity: Critical
• Assignee: Jonathan Ellis
• Reporter: Fabien Rousseau

Authors

Jonathan Ellis

Reviewers

Fabien Rousseau

• Created: 09/Oct/12 16:51
• Updated: 16/Apr/19 09:32
• Resolved: 11/Oct/12 13:59

Description

It seems that there are two corner cases where commitlog is not replayed after a restart :

• After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
• After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5

This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )

But this assertion can be broken :
In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)

This could explain the following tickets : #4741 and #4481