[CASSANDRA-16842] Allow CommitLogSegmentReader to optionally skip sync marker CRC checks

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1-alpha1, 4.1
• Component/s: Local/Commit Log
• Change Category: Operability
• Complexity: Normal
• Platform: All

Source Control Link

https://github.com/apache/cassandra/commit/f9b7c1e6984f5b81aae1e3a2191d4e9599db15ae

Test and Documentation Plan

additional tests in CommitLogTest to ensure the new switch behaves correctly with and without compression and encryption on the CommitLog

• Assignee: Caleb Rackliffe
• Reporter: Caleb Rackliffe

Authors

Caleb Rackliffe, Marcus Eriksson

Reviewers

Josh McKenzie

• Created: 10/Aug/21 21:13
• Updated: 27/May/22 19:25
• Resolved: 25/Aug/21 20:30

Description

CommitLog sync markers are written in two phases. In the first, zeroes are written for the position of the next sync marker and the sync marker CRC value. In the second, when the next sync marker is written, the actual position and CRC values are written. If the process shuts down in a disorderly fashion, it is entirely possible for a valid next marker position to be written to our memory mapped file but not the final CRC value. Later, when we attempt to replay the segment, we will fail without recovering any of the perfectly valid mutations it contains. (This assumes we’re confining ourselves to the case where there is no compression or encryption.)

ERROR 2020-11-18T10:55:23,888 [main] org.apache.cassandra.utils.JVMStabilityInspector:102 - Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 23091775 of commit log …/CommitLog-6-1605699607608.log, with invalid CRC. The end of segment marker should be zero.
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:731)
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.readSyncMarker(CommitLogReplayer.java:274)
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:436)
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:189)
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:170
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:151)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:332)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:656)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:808)

It may be useful to provide an option that would allow us to override the default/strict behavior here and skip the CRC check if a non-zero end position is present, allowing valid mutations to be recovered and startup to proceed.