[KUDU-521] Consider splitting the WAL in two, one for replicated operations and one for local operations

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Backlog
• Component/s: consensus, log
• Labels: roadmap-candidate scalability
• Target Version/s: Backlog
• Assignee: Unassigned
• Reporter: David Alves
• Created: 25/Sep/14 16:16
• Updated: 28/May/20 16:02

Description

Currently we rewrite the wal on replay because it has both local (the commits) and replicated operations (the replicates). But we could potentially split this in two separate WALs, one for the consensus replicated stuff and one for the local stuff.

Keeping 2 different wals would have the following advantages:

• Only replicates would be fsynced for writes.
• On bootstrap the replicates WAL would not have to be rewritten
• Replicate WAL segments could be checksummed across machines to make sure they are the same.
• The order of commits w.r.t. replicates would not matter.
• Flushing the WAL queue before we write the meta would only concern the "local" WAL.

... and the following disadvantages:

• On bootstrap replaying would be slightly more complex. One would have to read from the replicated WAL and then advance the local wal until we find the commits we want.
• GC would be more complex (but not as complex as dealing with commits before replicates in the current version though).