Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1064

Standalone Samza with Zookeeper for Coordination

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      In this use-case, we propose using Zookeeper for coordinating processor liveness and task distribution in a Samza job. Additionally, it opens up the possibility of allowing a flexible number of participating processors (no fixed container count).

        Attachments

        1. image_0.png
          5 kB
          Navina Ramesh
        2. image_1.png
          62 kB
          Navina Ramesh
        3. Samza Standalone-0.md
          30 kB
          Navina Ramesh
        4. Samza Standalone-0.pdf
          322 kB
          Navina Ramesh
        5. SamzaStandalone1.md
          30 kB
          Boris Shkolnik
        6. SamzaStandalone1.pdf
          75 kB
          Boris Shkolnik

        Issue Links

        depends upon

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1203 Use ProcessorId instead of Zk node Id in JobModel generation

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1147 ZkJobCoordinator does not honor the interface contract of JobCoordinator

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1173 Evaluate CoordinationService design and resolve unresolved questions from PR#91

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1150 Handling Error propagation between ZkJobCoordinator & DebounceTimer

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1193 Fix the flaky test TestZkBarrierForVersionUpgrade.testZkBarrierForVersionUpgrade

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1201 Zk path setup for leader elector and barrier should be separated

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1212 Refactor interaction between StreamProcessor, JobCoordinator and SamzaContainer

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1241 ScheduleAfterDebounceTime thread is not shutdown when ZkJobCoordinator shutsdown

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1272 ZkCoordinationUtils deletes the entire Zk tree on reset

        • Major - Major loss of function.
        • Resolved
        is related to

        Sub-task - The sub-task of the issue SAMZA-555 Write a ZK-based samza-standalone JobRunner

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1279 list of children in onProcessorsChange throws NPE when there an exception in ZK

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1282 Spinning up more containers than the number of tasks kills leader

        • Major - Major loss of function.
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. SAMZA-1125 Container.shutdown() only stops the runloop

        • Major - Major loss of function.
        • Resolved

        Test - A new unit, integration or system test. SAMZA-1343 Modify the existing failure-testing script for Zookeeper based deployment

        • Major - Major loss of function.
        • Open
        is required by

        New Feature - A new feature of the product, which has yet to be developed. SAMZA-1063 Samza Standalone Project

        • Major - Major loss of function.
        • In Progress
        relates to

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1344 Add retry in ApplicationRunner for failed StreamProcessors

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1348 Run stop() in the StreamProcessor in the same thread as other events.

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1301 Investigate ZKClient exceptions in ZkJobCoordinator.

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1382 Support for ZK communication version protocol.

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1302 Improve Zookeeper error handling

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1304 Improve duplicate processors detection

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1338 Number of processors greater then number of tasks results in an incorrect behavior.

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1368 make sure that a new job model will be generated if zk barrier times out.

        • Major - Major loss of function.
        • Closed

        New Feature - A new feature of the product, which has yet to be developed. SAMZA-1041 Multi-stage feature for Samza

        • Major - Major loss of function.
        • In Progress

        Improvement - An improvement or enhancement to an existing feature or task. SAMZA-1349 Fix stop() sequence in the StreamProcessor

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. SAMZA-1303 Request for waitForFinish(time, TimeUnit) api in LocalApplicationRunner

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed
        requires

        Bug - A problem which impairs or prevents the functions of the product. SAMZA-1084 User thread does not see errors from the processor thread when using the StreamProcessor API

        • Major - Major loss of function.
        • Resolved
        links to

        Web Link GitHub Pull Request #64

        Web Link GitHub Pull Request #180

        1.
        		 Implement Leader Election using ZK Sub-task Resolved Navina Ramesh Actions
        2.
        		 Implement Simple grouper by container, with support for arbitrary container ids Sub-task Resolved Boris Shkolnik Actions
        3.
        		 Implement debounce scheduler Sub-task Resolved Boris Shkolnik Actions
        4.
        		 ZkController for communication with the ZK Sub-task Resolved Boris Shkolnik Actions
        5.
        		 Barrier for job model update Sub-task Resolved Boris Shkolnik Actions
        6.
        		 Create test utils for Zk based Standalone testing Sub-task Resolved Boris Shkolnik Actions
        7.
        		 add utils for publishing job model and job model version. Sub-task Resolved Boris Shkolnik Actions
        8.
        		 Implement startup and shutdown sequence of jobs in ZK environment Sub-task Open Navina Ramesh Actions
        9.
        		 create ZK based JobCoordinator Sub-task Resolved Boris Shkolnik Actions
        10.
        		 add a time out for ZkVersionUpgradeBarrier Sub-task Resolved Boris Shkolnik Actions
        11.
        		 Semantics of ProcessorId in Samza Sub-task Resolved Navina Ramesh Actions
        12.
        		 verify that SamzaContainerController.stop shuts down the container completely. Sub-task Resolved Navina Ramesh Actions
        13.
        		 simplify ZK barrier for version upgrade. Sub-task Resolved Boris Shkolnik Actions
        14.
        		 create ZkCoordination service Sub-task Resolved Boris Shkolnik Actions
        15.
        		 List integration tests needed for the StandAlone project Sub-task Resolved Boris Shkolnik Actions
        16.
        		 Metrics should be added for ZK based JobCoordinator Sub-task In Progress Navina Ramesh Actions
        17.
        		 Review tryBecomeLeader implementation to see if it can be simplified. Sub-task Open Unassigned Actions
        18.
        		 ZK cleanup Sub-task Closed Boris Shkolnik Actions
        19.
        		 Locality should be used in standalone execution (host-affinity) Sub-task Open Navina Ramesh Actions
        20.
        		 Fix ZK path issues + container IDs generation. Sub-task Resolved Boris Shkolnik Actions
        21.
        		 ZkController should close zk connection on stop() Sub-task Resolved Boris Shkolnik Actions
        22.
        		 add new StandAlone configs to the configuration doc Sub-task Closed Boris Shkolnik Actions
        23.
        		 if ZkConnect string contains extra path, it needs to be created on the ZK. Sub-task Closed Boris Shkolnik Actions
        24.
        		 update samza configs with StandAlone tutorial link, when it is available. Sub-task Resolved Boris Shkolnik Actions
        25.
        		 HappyPathTesting. Single and multiple processors happy path. Sub-task Resolved Boris Shkolnik Actions
        26.
        		 HappyPath Testing. Rolling Upgrades. Sub-task Open Shanthoosh Venkataraman Actions
        27.
        		 Error Testing. ZkUnavailable. Sub-task Resolved Boris Shkolnik Actions
        28.
        		 Error Testing. Failing processor. Sub-task Resolved Boris Shkolnik Actions
        29.
        		 Failure Testing. A processor dies while waiting for a barrier to complete. Sub-task Open Shanthoosh Venkataraman Actions
        30.
        		 add a config for de-bounce time. Sub-task Resolved Boris Shkolnik Actions
        31.
        		 If LocalAppRunner is used - use ZkJobCoordinator by default. Sub-task Resolved Unassigned Actions
        32.
        		 HappyPath Testing. Using local storage. Sub-task Open Unassigned Actions
        33.
        		 renam StandaloneJobCoordinator to PassthroughJobCoordinator Sub-task Resolved Boris Shkolnik Actions
        34.
        		 Stand alone integration tests. Sub-task Resolved Shanthoosh Venkataraman Actions
        35.
        		 HappyPathTesting. Verify JM generated only once for multiple added SP Sub-task Open Shanthoosh Venkataraman Actions
        36.
        		 ERror Testing. If all processor die, LAR should shutdown. Sub-task Open Shanthoosh Venkataraman Actions
        37.
        		 session expiration propagation Sub-task Resolved Boris Shkolnik Actions
        38.
        		 LocalApplicationRunner needs to support StreamTask Sub-task Resolved Boris Shkolnik Actions

          Activity
          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              navina Navina Ramesh
              Reporter:
              navina Navina Ramesh

              Dates

              • Created:
                Updated:

                Issue deployment