Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-684

Uber/Local modes for Tez

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Similarly to MapReduce Uber-mode in Yarn, we plan to create the Uber-mode for Tez. It runs all tasks in local in one process.

      Our target is to start DAGAppMaster in local JVM and let it run all tasks in one process.

      Here is my design:

      Once user submits a DAG, Tez starts a instance of DAGAppMaster. This DAGAppMaster will check TezConfiguration before instantiate ContainerLauncher. If "is_Uber" is true, DAGAppMaster creates a LocalContainerLauncher. LocalTaskScheduler and LocalTaskSchedulerEventHandler will call LocalContainerLauncher to run all tasks one by one in a single JVM. Communications between ResourceManager and local classes (DAGAppMaster, LocalContainerLauncher, LocalTaskScheduler, and LocalTaskSchedulerEventHandler) are muted.

        Attachments

        1. TEZ-684-2014-7-21.patch
          118 kB
          Chen He
        2. TEZ-684.patch
          142 kB
          Chen He
        3. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        4. TEZ-684.patch
          115 kB
          Chen He
        5. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        6. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        7. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        8. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        9. TEZ-684.patch
          121 kB
          Jonathan Turner Eagles
        10. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        11. TEZ-684.patch
          134 kB
          Jonathan Turner Eagles
        12. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        13. TEZ-684.patch
          151 kB
          Chen He
        14. TezUberModeDesignDraft.png
          74 kB
          Chen He

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                airbots Chen He
                Reporter:
                airbots Chen He
              • Votes:
                6 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: