[NEMO-235] Refactor TaskExecutor's data processing logic

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.1
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Tae-Geon Um
• Created: 24/Oct/18 02:06
• Updated: 25/Nov/18 23:30
• Resolved: 26/Oct/18 00:09

Description

TaskExecutor has complex logic to process data, additional output, side inputs and so on. This is because the TaskExecutor manages data flow within a Task. The downside of this approach is that it increases complexity of TaskExecutor if we add other feature of data (e.g., watermark) and it transfers data between Transform by pull-based approach, which may occur additional enqueue/dequeue overhead for each element processing. 

 

We can simplify the TaskExecutor's data processing logic and implement push-based element processing by delegating the data flow in a Task to OutputCollector. We can implement various OutputCollectors such as push-based output collector that emits data to downstream operator directly and sink output collector that writes data to Writer. 

Issue Links

• causes: NEMO-238 Optimize processElement in TaskExecutor
• is related to: NEMO-234 Implement TransformChaining
• is related to: NEMO-229 Support watermark
• links to: GitHub Pull Request #128