Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-905

Host Affinity - State restore doesn't work if the previous shutdown was uncontrolled (continuous offset)

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.1
    • Component/s: None
    • Labels:
      None

      Description

      The host affinity feature writes an OFFSET file during shutdown which records the changelog offset for the persisted state store. Because the file is written at shutdown, it depends on a clean shutdown. Otherwise the file will not be there when the task restarts and the state will not be restored.

      This ticket is to add a continuous checkpointing feature so the OFFSET file is written at regular intervals and the most recent update can be used to restore state.

      Note: this feature assumes all changelog entries are idempotent as warned in https://issues.apache.org/jira/secure/attachment/12696372/DESIGN-SAMZA-516-2.pdf

      1. SAMZA-905_3.patch
        21 kB
        Jake Maes
      2. SAMZA-905_2.patch
        21 kB
        Jake Maes
      3. SAMZA-905.patch
        12 kB
        Jake Maes

        Activity

        Hide
        nickpan47 Yi Pan (Data Infrastructure) added a comment -

        Merged and submitted. Thanks!

        Show
        nickpan47 Yi Pan (Data Infrastructure) added a comment - Merged and submitted. Thanks!
        Hide
        jmakes Jake Maes added a comment -

        Patch SAMZA-905_3.patch incorporates Yi's feedback.

        Thanks!

        Show
        jmakes Jake Maes added a comment - Patch SAMZA-905 _3.patch incorporates Yi's feedback. Thanks!
        Hide
        jmakes Jake Maes added a comment -

        SAMZA-905_2.patch includes documentation updates.

        Show
        jmakes Jake Maes added a comment - SAMZA-905 _2.patch includes documentation updates.
        Hide
        jmakes Jake Maes added a comment -
        Show
        jmakes Jake Maes added a comment - Attaching patch RB: https://reviews.apache.org/r/45136/

          People

          • Assignee:
            jmakes Jake Maes
            Reporter:
            jmakes Jake Maes
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development