[TEZ-684] Uber/Local modes for Tez

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Assignee: Chen He
• Reporter: Chen He
• Created: 23/Dec/13 23:24
• Updated: 18/Oct/18 03:36
• Resolved: 18/Dec/14 02:03

Description

Similarly to MapReduce Uber-mode in Yarn, we plan to create the Uber-mode for Tez. It runs all tasks in local in one process.

Our target is to start DAGAppMaster in local JVM and let it run all tasks in one process.

Here is my design:

Once user submits a DAG, Tez starts a instance of DAGAppMaster. This DAGAppMaster will check TezConfiguration before instantiate ContainerLauncher. If "is_Uber" is true, DAGAppMaster creates a LocalContainerLauncher. LocalTaskScheduler and LocalTaskSchedulerEventHandler will call LocalContainerLauncher to run all tasks one by one in a single JVM. Communications between ResourceManager and local classes (DAGAppMaster, LocalContainerLauncher, LocalTaskScheduler, and LocalTaskSchedulerEventHandler) are muted.

Issue Links

• is blocked by: TEZ-1343 Bypass the Fetcher and read directly from the local filesystem if source vertex ran on the same host
• is duplicated by: TEZ-235 Tez Local Mode
• is related to: TEZ-1844 Shouldn't invoke system.exit in local mode when AM is failed to start
• is related to: PIG-4008 Pig code change to enable Tez Local mode
• is related to: TEZ-1837 Restrict usage of Environment variables to main methods
• requires: TEZ-1191 DAGAppMaster shuts down JVM resulting in shutdown propagation up the stack.
• supercedes: TEZ-72 Support running of single-task vertices inlined in the DAG AM