[SAMZA-1786] Introduce metadata store abstraction.

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Shanthoosh Venkataraman
• Reporter: Shanthoosh Venkataraman
• Created: 26/Jul/18 21:30
• Updated: 29/Aug/18 22:38

Description

As a part of SEP-11, this JIRA introduces MetadataStore interface to store task and container locality for both yarn and standalone deployment models. Refer to SEP-11 design doc for more details. 

 

Few important points to note:

1. As a part of this changes, LocalityManager/TaskAsssignemtnManager alone will be updated to use this interface(subsequently in upcoming future RB's other util classes will be moved to use this interface as well).
2. In an immediate followup RB, ZkMetadataStore(storing metadata information in zookeeper) will be added. It will be used in standalone to read/write locality(through LocalityManager & other standard util classes).
3. In future, executionPlan, other job related metadata(with respect to running containers, stream graph) can be stored in any store through the same abstraction. In future, if we want to deploy standalone jobs in azure, only we need to implement this interface(CosmosDBMetadataStore to store samza metadata in cosmos).

Issue Links

• Dependency: SAMZA-1554 Host affinity in standalone.
• links to: GitHub Pull Request #583