Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1516

FileChannel Write Dual Checkpoints to avoid replays

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.0
    • 1.4.0
    • Channel, File Channel
    • None

    Description

      Per the LFS paper (http://www.cs.berkeley.edu/~brewer/cs262/LFS.pdf) we can write two checkpoints to avoid replaying the logs in the case we crash/shutdown while writing a checkpoint.

      Section 4:

      "In order to handle a crash during a checkpoint operation there are actually two checkpoint regions, and checkpoint operations alternate between them. The checkpoint time is in the last block of the checkpoint so if the checkpoint fails the time will not be updated. During reboot, the system reads both checkpoint regions and uses the one with the most recent time."

      Attachments

        1. DualCheckpoints.pdf
          96 kB
          Hari Shreedharan
        2. DualCheckpointsv2.pdf
          106 kB
          Hari Shreedharan
        3. FLUME-1516.patch
          44 kB
          Hari Shreedharan
        4. FLUME-1516-1.patch
          44 kB
          Hari Shreedharan
        5. DualCheckpointsv3.pdf
          79 kB
          Hari Shreedharan
        6. FLUME-1516-2.patch
          70 kB
          Hari Shreedharan
        7. FLUME-1516-3.patch
          76 kB
          Hari Shreedharan
        8. FLUME-1516-4.patch
          76 kB
          Hari Shreedharan
        9. FLUME-1516-5.patch
          83 kB
          Hari Shreedharan
        10. FLUME-1516-6.patch
          83 kB
          Hari Shreedharan
        11. FLUME-1516-7.patch
          92 kB
          Hari Shreedharan
        12. FLUME-1516-8.patch
          97 kB
          Hari Shreedharan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hshreedharan Hari Shreedharan
            brocknoland Brock Noland
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment