[TEZ-684] Uber/Local modes for Tez

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Assignee: Chen He
• Reporter: Chen He
• Created: 23/Dec/13 23:24
• Updated: 18/Oct/18 03:36
• Resolved: 18/Dec/14 02:03

Description

Similarly to MapReduce Uber-mode in Yarn, we plan to create the Uber-mode for Tez. It runs all tasks in local in one process.

Our target is to start DAGAppMaster in local JVM and let it run all tasks in one process.

Here is my design:

Once user submits a DAG, Tez starts a instance of DAGAppMaster. This DAGAppMaster will check TezConfiguration before instantiate ContainerLauncher. If "is_Uber" is true, DAGAppMaster creates a LocalContainerLauncher. LocalTaskScheduler and LocalTaskSchedulerEventHandler will call LocalContainerLauncher to run all tasks one by one in a single JVM. Communications between ResourceManager and local classes (DAGAppMaster, LocalContainerLauncher, LocalTaskScheduler, and LocalTaskSchedulerEventHandler) are muted.

Issue Links

• is blocked by: TEZ-1343 Bypass the Fetcher and read directly from the local filesystem if source vertex ran on the same host
• is duplicated by: TEZ-235 Tez Local Mode
• is related to: TEZ-1844 Shouldn't invoke system.exit in local mode when AM is failed to start
• is related to: PIG-4008 Pig code change to enable Tez Local mode
• is related to: TEZ-1837 Restrict usage of Environment variables to main methods
• requires: TEZ-1191 DAGAppMaster shuts down JVM resulting in shutdown propagation up the stack.
• supercedes: TEZ-72 Support running of single-task vertices inlined in the DAG AM

Sub-Tasks

• Create LocalContainerLauncher: Closed
• Avoid asking for new containers in uber-local mode: Closed
• LocalFetcher for the Uber-mode: Closed
• Get Tez Local-mode work: Resolved
• Client changes to allow local mode DAG submission: Closed
• LocalContainerLauncher improvement - error handling, multiple tasks, Tez events: Closed
• Tez Configuration changes for TezLocalRunner: Closed
• Add Utility for Modifying Environment Variables: Closed
• Add parameterized constructor capabilities in ReflectionUtils: Closed
• Create execution underlying framework abstraction layer: Closed
• Support submission of multiple applications with LocalRunner from within the same JVM: Closed
• Add documentation for LocalMode usage: Closed
• Session mode and ignore-lib-uris needs to be set for session mode: Closed
• Avoid sysexit in the DAGAM in case of local mode: Closed
• Additional supplement for TEZ local mode document: Closed
• Disable node blacklisting and ATS in AM for local mode: Closed
• Tez Local Mode fails if "fs.defaultFS" is not set to be "file:///' in core-site.xml: Resolved
• Local Mode crashes on concurrentModificationException: Closed
• Add multiDAG session test and move TestLocalMode to tez-tests: Closed
• LocalTaskSchedulerService.getTotalResources() and getAvailableResources() can get negative if JVM memory is larger than 2GB: Closed