Flume
  1. Flume
  2. FLUME-1516

FileChannel Write Dual Checkpoints to avoid replays

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.3.0
    • Fix Version/s: v1.4.0
    • Component/s: Channel, File Channel
    • Labels:
      None

      Description

      Per the LFS paper (http://www.cs.berkeley.edu/~brewer/cs262/LFS.pdf) we can write two checkpoints to avoid replaying the logs in the case we crash/shutdown while writing a checkpoint.

      Section 4:

      "In order to handle a crash during a checkpoint operation there are actually two checkpoint regions, and checkpoint operations alternate between them. The checkpoint time is in the last block of the checkpoint so if the checkpoint fails the time will not be updated. During reboot, the system reads both checkpoint regions and uses the one with the most recent time."

      1. FLUME-1516-8.patch
        97 kB
        Hari Shreedharan
      2. FLUME-1516-7.patch
        92 kB
        Hari Shreedharan
      3. FLUME-1516-6.patch
        83 kB
        Hari Shreedharan
      4. FLUME-1516-5.patch
        83 kB
        Hari Shreedharan
      5. FLUME-1516-4.patch
        76 kB
        Hari Shreedharan
      6. FLUME-1516-3.patch
        76 kB
        Hari Shreedharan
      7. FLUME-1516-2.patch
        70 kB
        Hari Shreedharan
      8. FLUME-1516-1.patch
        44 kB
        Hari Shreedharan
      9. FLUME-1516.patch
        44 kB
        Hari Shreedharan
      10. DualCheckpointsv3.pdf
        79 kB
        Hari Shreedharan
      11. DualCheckpointsv2.pdf
        106 kB
        Hari Shreedharan
      12. DualCheckpoints.pdf
        96 kB
        Hari Shreedharan

        Issue Links

          Activity

          Brock Noland created issue -
          Hari Shreedharan made changes -
          Field Original Value New Value
          Assignee Hari Shreedharan [ hshreedharan ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516.patch [ 12563475 ]
          Hari Shreedharan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s v1.4.0 [ 12323372 ]
          Hari Shreedharan made changes -
          Attachment DualCheckpoints.pdf [ 12563505 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-1.patch [ 12563508 ]
          Hari Shreedharan made changes -
          Remote Link This issue links to "Review (Web Link)" [ 11953 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-1-rebased.patch [ 12563802 ]
          Hari Shreedharan made changes -
          Issue Type Improvement [ 4 ] New Feature [ 2 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-2.patch [ 12563867 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516.patch [ 12563475 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-1.patch [ 12563508 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-1-rebased.patch [ 12563802 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-2.patch [ 12563867 ]
          Hari Shreedharan made changes -
          Attachment DualCheckpointsv2.pdf [ 12563882 ]
          Hari Shreedharan made changes -
          Remote Link This issue links to "Review (Web Link)" [ 11953 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516.patch [ 12563895 ]
          Hari Shreedharan made changes -
          Remote Link This issue links to "Review (Web Link)" [ 11958 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1821-1.patch [ 12563959 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1821-1.patch [ 12563959 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-1.patch [ 12563960 ]
          Hari Shreedharan made changes -
          Attachment DualCheckpointv3.pdf [ 12565655 ]
          Hari Shreedharan made changes -
          Attachment DualCheckpointv3.pdf [ 12565655 ]
          Hari Shreedharan made changes -
          Attachment DualCheckpointsv3.pdf [ 12565733 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-2.patch [ 12565852 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-2.patch [ 12565852 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-2.patch [ 12565883 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-3.patch [ 12571949 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-4.patch [ 12573690 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-5.patch [ 12573799 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-6.patch [ 12575386 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-7.patch [ 12575602 ]
          Brock Noland made changes -
          Summary Write Dual Checkpoints to avoid replays FileChannel Write Dual Checkpoints to avoid replays
          Component/s File Channel [ 12320407 ]
          Brock Noland made changes -
          Link This issue is related to FLUME-1968 [ FLUME-1968 ]
          Hari Shreedharan made changes -
          Attachment FLUME-1516-8.patch [ 12577246 ]
          Brock Noland made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development