[SAMZA-1842] Need consistent logic to calculate input, output, and intermediate streams in high and low-level API applications

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Yi Pan
• Created: 06/Sep/18 01:26
• Updated: 06/Sep/18 23:28

Description

Currently, the input and output descriptors for input, output, and intermediate streams are explicitly set in TaskApplication, while only the external input and output descriptors are created and set in high-level StreamApplication.

This creates a discrepancy in how we compute the input, output, and intermediate streams in the planner for TaskApplication and StreamApplication (i.e. intermediate streams don't have descriptors in StreamApplication, but have both input and output descriptors in TaskApplication). We should make the logic consistent between TaskApplication and StreamApplication.

Issue Links

• is related to: SAMZA-1804 Allow configuring input and output system and stream properties in code
• is related to: SAMZA-1811 Consolidate the ExecutionPlanner to handle both high-level DAG and low-level single node jobs
• is related to: SAMZA-1789 Refactor the application runtime classes so that we have a unified pattern to write and launch applications in high- and low-level APIs in YARN and standalone