Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-684

Uber/Local modes for Tez

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Similarly to MapReduce Uber-mode in Yarn, we plan to create the Uber-mode for Tez. It runs all tasks in local in one process.

      Our target is to start DAGAppMaster in local JVM and let it run all tasks in one process.

      Here is my design:

      Once user submits a DAG, Tez starts a instance of DAGAppMaster. This DAGAppMaster will check TezConfiguration before instantiate ContainerLauncher. If "is_Uber" is true, DAGAppMaster creates a LocalContainerLauncher. LocalTaskScheduler and LocalTaskSchedulerEventHandler will call LocalContainerLauncher to run all tasks one by one in a single JVM. Communications between ResourceManager and local classes (DAGAppMaster, LocalContainerLauncher, LocalTaskScheduler, and LocalTaskSchedulerEventHandler) are muted.

        Attachments

        1. TEZ-684.patch
          142 kB
          Chen He
        2. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        3. TEZ-684.patch
          115 kB
          Chen He
        4. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        5. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        6. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        7. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        8. TEZ-684.patch
          121 kB
          Jonathan Turner Eagles
        9. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        10. TEZ-684.patch
          134 kB
          Jonathan Turner Eagles
        11. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        12. TEZ-684.patch
          151 kB
          Chen He
        13. TEZ-684-2014-7-21.patch
          118 kB
          Chen He
        14. TezUberModeDesignDraft.png
          74 kB
          Chen He

          Issue Links

          1.
          Create LocalContainerLauncher Sub-task Closed Chen He
          2.
          Avoid asking for new containers in uber-local mode Sub-task Closed Jonathan Turner Eagles
          3.
          LocalFetcher for the Uber-mode Sub-task Closed Chen He
          4.
          Get Tez Local-mode work Sub-task Resolved Chen He
          5.
          Client changes to allow local mode DAG submission Sub-task Closed Jonathan Turner Eagles
          6.
          LocalContainerLauncher improvement - error handling, multiple tasks, Tez events Sub-task Closed Siddharth Seth
          7.
          Tez Configuration changes for TezLocalRunner Sub-task Closed Chen He
          8.
          Add Utility for Modifying Environment Variables Sub-task Closed Jonathan Turner Eagles
          9.
          Add parameterized constructor capabilities in ReflectionUtils Sub-task Closed Jonathan Turner Eagles
          10.
          Create execution underlying framework abstraction layer Sub-task Closed Jonathan Turner Eagles
          11.
          Support submission of multiple applications with LocalRunner from within the same JVM Sub-task Closed Siddharth Seth
          12.
          Add documentation for LocalMode usage Sub-task Closed Siddharth Seth
          13.
          Session mode and ignore-lib-uris needs to be set for session mode Sub-task Closed Siddharth Seth
          14.
          Avoid sysexit in the DAGAM in case of local mode Sub-task Closed Siddharth Seth
          15.
          Additional supplement for TEZ local mode document Sub-task Closed Chen He
          16.
          Disable node blacklisting and ATS in AM for local mode Sub-task Closed Jonathan Turner Eagles
          17.
          Tez Local Mode fails if "fs.defaultFS" is not set to be "file:///' in core-site.xml Sub-task Resolved Chen He
          18.
          Local Mode crashes on concurrentModificationException Sub-task Closed Chen He
          19.
          Add multiDAG session test and move TestLocalMode to tez-tests Sub-task Closed Chen He
          20.
          LocalTaskSchedulerService.getTotalResources() and getAvailableResources() can get negative if JVM memory is larger than 2GB Sub-task Closed Chen He

            Activity

              People

              • Assignee:
                airbots Chen He
                Reporter:
                airbots Chen He
              • Votes:
                6 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: