[SAMZA-1064] Standalone Samza with Zookeeper for Coordination

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.13.0
• Assignee: Navina Ramesh
• Reporter: Navina Ramesh
• Created: 16/Dec/16 02:02
• Updated: 01/Aug/17 02:52

Description

In this use-case, we propose using Zookeeper for coordinating processor liveness and task distribution in a Samza job. Additionally, it opens up the possibility of allowing a flexible number of participating processors (no fixed container count).

Issue Links

• depends upon: SAMZA-1203 Use ProcessorId instead of Zk node Id in JobModel generation
• depends upon: SAMZA-1147 ZkJobCoordinator does not honor the interface contract of JobCoordinator
• depends upon: SAMZA-1173 Evaluate CoordinationService design and resolve unresolved questions from PR#91
• depends upon: SAMZA-1150 Handling Error propagation between ZkJobCoordinator & DebounceTimer
• depends upon: SAMZA-1193 Fix the flaky test TestZkBarrierForVersionUpgrade.testZkBarrierForVersionUpgrade
• depends upon: SAMZA-1201 Zk path setup for leader elector and barrier should be separated
• depends upon: SAMZA-1212 Refactor interaction between StreamProcessor, JobCoordinator and SamzaContainer
• depends upon: SAMZA-1241 ScheduleAfterDebounceTime thread is not shutdown when ZkJobCoordinator shutsdown
• depends upon: SAMZA-1272 ZkCoordinationUtils deletes the entire Zk tree on reset
• is related to: SAMZA-555 Write a ZK-based samza-standalone JobRunner
• is related to: SAMZA-1279 list of children in onProcessorsChange throws NPE when there an exception in ZK
• is related to: SAMZA-1282 Spinning up more containers than the number of tasks kills leader
• is related to: SAMZA-1125 Container.shutdown() only stops the runloop
• is related to: SAMZA-1343 Modify the existing failure-testing script for Zookeeper based deployment
• is required by: SAMZA-1063 Samza Standalone Project
• relates to: SAMZA-1344 Add retry in ApplicationRunner for failed StreamProcessors
• relates to: SAMZA-1348 Run stop() in the StreamProcessor in the same thread as other events.
• relates to: SAMZA-1301 Investigate ZKClient exceptions in ZkJobCoordinator.
• relates to: SAMZA-1382 Support for ZK communication version protocol.
• relates to: SAMZA-1302 Improve Zookeeper error handling
• relates to: SAMZA-1304 Improve duplicate processors detection
• relates to: SAMZA-1338 Number of processors greater then number of tasks results in an incorrect behavior.
• relates to: SAMZA-1368 make sure that a new job model will be generated if zk barrier times out.
• relates to: SAMZA-1041 Multi-stage feature for Samza
• relates to: SAMZA-1349 Fix stop() sequence in the StreamProcessor
• relates to: SAMZA-1303 Request for waitForFinish(time, TimeUnit) api in LocalApplicationRunner
• requires: SAMZA-1084 User thread does not see errors from the processor thread when using the StreamProcessor API
• links to: GitHub Pull Request #64
• links to: GitHub Pull Request #180

Sub-Tasks

• Implement Leader Election using ZK: Resolved
• Implement Simple grouper by container, with support for arbitrary container ids: Resolved
• Implement debounce scheduler: Resolved
• ZkController for communication with the ZK: Resolved
• Barrier for job model update: Resolved
• Create test utils for Zk based Standalone testing: Resolved
• add utils for publishing job model and job model version.: Resolved
• Implement startup and shutdown sequence of jobs in ZK environment: Open
• create ZK based JobCoordinator: Resolved
• add a time out for ZkVersionUpgradeBarrier: Resolved
• Semantics of ProcessorId in Samza: Resolved
• verify that SamzaContainerController.stop shuts down the container completely.: Resolved
• simplify ZK barrier for version upgrade.: Resolved
• create ZkCoordination service: Resolved
• List integration tests needed for the StandAlone project: Resolved
• Metrics should be added for ZK based JobCoordinator: In Progress
• Review tryBecomeLeader implementation to see if it can be simplified.: Open
• ZK cleanup: Closed
• Locality should be used in standalone execution (host-affinity): Open
• Fix ZK path issues + container IDs generation.: Resolved
• ZkController should close zk connection on stop(): Resolved
• add new StandAlone configs to the configuration doc: Closed
• if ZkConnect string contains extra path, it needs to be created on the ZK.: Closed
• update samza configs with StandAlone tutorial link, when it is available.: Resolved
• HappyPathTesting. Single and multiple processors happy path.: Resolved
• HappyPath Testing. Rolling Upgrades.: Open
• Error Testing. ZkUnavailable.: Resolved
• Error Testing. Failing processor.: Resolved
• Failure Testing. A processor dies while waiting for a barrier to complete.: Open
• add a config for de-bounce time.: Resolved
• If LocalAppRunner is used - use ZkJobCoordinator by default.: Resolved
• HappyPath Testing. Using local storage.: Open
• renam StandaloneJobCoordinator to PassthroughJobCoordinator: Resolved
• Stand alone integration tests.: Resolved
• HappyPathTesting. Verify JM generated only once for multiple added SP: Open
• ERror Testing. If all processor die, LAR should shutdown.: Open
• session expiration propagation: Resolved
• LocalApplicationRunner needs to support StreamTask: Resolved