Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1786

Introduce metadata store abstraction.

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As a part of SEP-11, this JIRA introduces MetadataStore interface to store task and container locality for both yarn and standalone deployment models. Refer to SEP-11 design doc for more details. 

       

      Few important points to note:

      1. As a part of this changes, LocalityManager/TaskAsssignemtnManager alone will be updated to use this interface(subsequently in upcoming future RB's other util classes will be moved to use this interface as well).
      2. In an immediate followup RB, ZkMetadataStore(storing metadata information in zookeeper) will be added. It will be used in standalone to read/write locality(through LocalityManager & other standard util classes).
      3. In future, executionPlan, other job related metadata(with respect to running containers, stream graph) can be stored in any store through the same abstraction. In future, if we want to deploy standalone jobs in azure, only we need to implement this interface(CosmosDBMetadataStore to store samza metadata in cosmos).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                spvenkat Shanthoosh Venkataraman
                Reporter:
                spvenkat Shanthoosh Venkataraman
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: