[OOZIE-1178] Workflow Application Master in YARN

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Bo Wang
• Created: 30/Jul/12 20:57
• Updated: 25/May/18 13:35

Description

It is useful to have a workflow application master, which will be capable of running a DAG of jobs. The workflow client submits a DAG request to the AM and then the AM will manage the life cycle of this application in terms of requesting the needed resources from the RM, and starting, monitoring and retrying the application's individual tasks.

Compared to running Oozie with the current MapReduce Application Master, these are some of the advantages:

• Less number of consumed resources, since only one application master will be spawned for the whole workflow.
• Reuse of resources, since the same resources can be used by multiple consecutive jobs in the workflow (no need to request/wait for resources for every individual job from the central RM).
• More optimization opportunities in terms of collective resource requests.
• Optimization opportunities in terms of rewriting and composing jobs in the workflow (e.g. pushing down Mappers).
• This Application Master can be reused/extended by higher systems like Pig and hive to provide an optimized way of running their workflows.

Issue Links

• is related to: OOZIE-593 WF lite2 implementation with testcases
• is related to: HIVE-1107 Generic parallel execution framework for Hive (and Pig, and ...)
• relates to: OOZIE-1770 Create Oozie Application Master for YARN
• relates to: PIG-1734 Pig needs a more efficient DAG execution