Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-560 Consensus/WAL/Transactions Optimizations and tests
  3. KUDU-521

Consider splitting the WAL in two, one for replicated operations and one for local operations

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: Backlog
    • Fix Version/s: None
    • Component/s: consensus, log
    • Labels:
      None

      Description

      Currently we rewrite the wal on replay because it has both local (the commits) and replicated operations (the replicates). But we could potentially split this in two separate WALs, one for the consensus replicated stuff and one for the local stuff.

      Keeping 2 different wals would have the following advantages:

      • Only replicates would be fsynced for writes.
      • On bootstrap the replicates WAL would not have to be rewritten
      • Replicate WAL segments could be checksummed across machines to make sure they are the same.
      • The order of commits w.r.t. replicates would not matter.
      • Flushing the WAL queue before we write the meta would only concern the "local" WAL.

      ... and the following disadvantages:

      • On bootstrap replaying would be slightly more complex. One would have to read from the replicated WAL and then advance the local wal until we find the commits we want.
      • GC would be more complex (but not as complex as dealing with commits before replicates in the current version though).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dralves David Alves
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: