[SAMZA-1373] Write Coordination Services for Samza Standalone using Azure

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Assignee: Pawas Chhokra
• Reporter: Pawas Chhokra
• Created: 31/Jul/17 23:01
• Updated: 02/Jan/18 19:03

Description

The current coordination primitives for Samza Standalone have been implemented using Zookeeper. Zookeeper is tedious to deploy on Azure and increases customer dependency on infrastructure. Writing coordination primitives in Azure will help identify issues with the current job coordinator design, validate the functionalities Samza Standalone claims it provides, and provide a platform for our customers to host their applications in the cloud. It will also provide more incentive for non-users of Samza to switch to Samza, and enable relatively smooth onboarding for customers that run their apps on Azure.

Issue Links

• incorporates: SAMZA-1374 Implement Leader Election using Lease Blob in Azure
• incorporates: SAMZA-1379 Create Azure Client
• incorporates: SAMZA-1380 Create Utility Class for interacting with Azure Blob Storage
• incorporates: SAMZA-1381 Create Utility Class for interacting with Azure Table Storage
• incorporates: SAMZA-1377 Implement Azure Job Coordinator
• incorporates: SAMZA-1402 Refactor Azure Scheduler Interface
• relates to: SAMZA-1549 Synchronization issue between heartbeat scheduler and job coordinator main thread during processor shutdown
• relates to: SAMZA-1523 Cleanup table entries before shutting down the processor