[SAMZA-1893] JobNodeConfigurationGenerator should only generate configurations for operators, streams, stores, and tables that are reachable by a JobNode

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Yi Pan
• Created: 20/Sep/18 22:28
• Updated: 20/Sep/18 22:28

Description

Currently, the planner does not generate multi-job ExecutionPlan yet. And hence, the current implementation of JobNodeConfigurationGenerator does not strictly follow the rule to only generate the configuration for input/output/intermediate streams, operators, stores, and tables reachable by the current JobNode yet (i.e. for single JobNode plan, everything is reachable by the only JobNode).

When we extend it to multi-job plan, we need to generate the configurations only for streams, operators, stores, and tables that are reachable by a JobNode. If two JobNodes collide on the configuration for those configuration, it will result in the following problems:
1) input streams are consumed multiple times, unnecessarily
2) stores' changelog will be written by multiple jobs, creating consistency issue in recovery
3) tables will be accessed (read or write) by multiple jobs, also creating consistency issues

We need to make sure that JobNodes in multi-job plans don't create collision in configuration for input/output/intermediate streams, state stores, and tables.

Issue Links

• is a child of: SAMZA-1041 Multi-stage feature for Samza