Apache S4
  1. Apache S4
  2. S4-11

add checkpointing mechanism to s4-piper

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.5.0
    • Labels:
      None

      Description

      There is already such a mechanism in S4.

      I'll port it to S4-piper, once the API is more stable, especially regarding triggers.

        Issue Links

          Activity

          Hide
          Matthieu Morel added a comment -

          Patch available in branch S4-11.

          Provides a checkpointing mechanism with the following characteristics:

          • based either on event count or on time intervals
          • configurable per PE prototype
          • parallel with event processing flow
          • default backend implementation relying on a file system is provided
          • backend configuration through Guice modules
          • configurable timeout on fetches (S4-44)

          Patch also updates the twitter trending example by adding checkpointing capabilities.

          Note that checkpointing, to be enabled, requires both:

          • the application to specify the checkpointing configuration per prototype
          • the S4 node configuration to enable checkpointing (configured through a module, a default one with a file system backend is available in s4-core)
          Show
          Matthieu Morel added a comment - Patch available in branch S4-11 . Provides a checkpointing mechanism with the following characteristics: based either on event count or on time intervals configurable per PE prototype parallel with event processing flow default backend implementation relying on a file system is provided backend configuration through Guice modules configurable timeout on fetches ( S4-44 ) Patch also updates the twitter trending example by adding checkpointing capabilities. Note that checkpointing, to be enabled, requires both: the application to specify the checkpointing configuration per prototype the S4 node configuration to enable checkpointing (configured through a module, a default one with a file system backend is available in s4-core)
          Hide
          Matthieu Morel added a comment -

          Just updated the patch with a clarification of the API for evaluating the "dirty" state of the API, following a suggestion from Daniel. See commit 726580ba162ffaa4107ee4c89c16b139cfa520d8

          Show
          Matthieu Morel added a comment - Just updated the patch with a clarification of the API for evaluating the "dirty" state of the API, following a suggestion from Daniel. See commit 726580ba162ffaa4107ee4c89c16b139cfa520d8
          Hide
          Daniel Gómez Ferro added a comment -

          This is a great feature and patch looks good, +1!

          Show
          Daniel Gómez Ferro added a comment - This is a great feature and patch looks good, +1!
          Hide
          Matthieu Morel added a comment -

          merged into piper branch commit 54c5fa232b6bead60f2c2f27286cf2c81f18d4ea

          Show
          Matthieu Morel added a comment - merged into piper branch commit 54c5fa232b6bead60f2c2f27286cf2c81f18d4ea

            People

            • Assignee:
              Matthieu Morel
              Reporter:
              Matthieu Morel
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development