[TEZ-1166] Support for pipelined execution

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Bikas Saha
• Reporter: Bikas Saha
• Created: 02/Jun/14 06:38
• Updated: 12/Mar/15 20:35

Description

Vertices in a DAG may perform non-blocking operations. This enables data to be pipelined across a chain of producer/consumer vertex tasks until it hits a blocking operation or is completely processed. This umbrella jira tracks improvements needed in Tez (eg fault tolerance/scheduling) to support this.