Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-684

Uber/Local modes for Tez

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.0
    • None
    • None

    Description

      Similarly to MapReduce Uber-mode in Yarn, we plan to create the Uber-mode for Tez. It runs all tasks in local in one process.

      Our target is to start DAGAppMaster in local JVM and let it run all tasks in one process.

      Here is my design:

      Once user submits a DAG, Tez starts a instance of DAGAppMaster. This DAGAppMaster will check TezConfiguration before instantiate ContainerLauncher. If "is_Uber" is true, DAGAppMaster creates a LocalContainerLauncher. LocalTaskScheduler and LocalTaskSchedulerEventHandler will call LocalContainerLauncher to run all tasks one by one in a single JVM. Communications between ResourceManager and local classes (DAGAppMaster, LocalContainerLauncher, LocalTaskScheduler, and LocalTaskSchedulerEventHandler) are muted.

      Attachments

        1. TEZ-684-2014-7-21.patch
          118 kB
          Chen He
        2. TEZ-684.patch
          142 kB
          Chen He
        3. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        4. TEZ-684.patch
          115 kB
          Chen He
        5. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        6. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        7. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        8. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        9. TEZ-684.patch
          121 kB
          Jonathan Turner Eagles
        10. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        11. TEZ-684.patch
          134 kB
          Jonathan Turner Eagles
        12. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        13. TEZ-684.patch
          151 kB
          Chen He
        14. TezUberModeDesignDraft.png
          74 kB
          Chen He

        Issue Links

        is blocked by

        Task - A task that needs to be done. TEZ-1343 Bypass the Fetcher and read directly from the local filesystem if source vertex ran on the same host

        • Major - Major loss of function.
        • Closed
        is duplicated by

        Improvement - An improvement or enhancement to an existing feature or task. TEZ-235 Tez Local Mode

        • Major - Major loss of function.
        • Resolved
        is related to

        Bug - A problem which impairs or prevents the functions of the product. TEZ-1844 Shouldn't invoke system.exit in local mode when AM is failed to start

        • Major - Major loss of function.
        • Closed

        New Feature - A new feature of the product, which has yet to be developed. PIG-4008 Pig code change to enable Tez Local mode

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. TEZ-1837 Restrict usage of Environment variables to main methods

        • Major - Major loss of function.
        • Closed
        requires

        Sub-task - The sub-task of the issue TEZ-1191 DAGAppMaster shuts down JVM resulting in shutdown propagation up the stack.

        • Major - Major loss of function.
        • Reopened
        supercedes

        Bug - A problem which impairs or prevents the functions of the product. TEZ-72 Support running of single-task vertices inlined in the DAG AM

        • Major - Major loss of function.
        • Resolved
        1.
        		 Create LocalContainerLauncher Sub-task Closed Chen He Actions
        2.
        		 Avoid asking for new containers in uber-local mode Sub-task Closed Jonathan Turner Eagles Actions
        3.
        		 LocalFetcher for the Uber-mode Sub-task Closed Chen He Actions
        4.
        		 Get Tez Local-mode work Sub-task Resolved Chen He Actions
        5.
        		 Client changes to allow local mode DAG submission Sub-task Closed Jonathan Turner Eagles Actions
        6.
        		 LocalContainerLauncher improvement - error handling, multiple tasks, Tez events Sub-task Closed Siddharth Seth Actions
        7.
        		 Tez Configuration changes for TezLocalRunner Sub-task Closed Chen He Actions
        8.
        		 Add Utility for Modifying Environment Variables Sub-task Closed Jonathan Turner Eagles Actions
        9.
        		 Add parameterized constructor capabilities in ReflectionUtils Sub-task Closed Jonathan Turner Eagles Actions
        10.
        		 Create execution underlying framework abstraction layer Sub-task Closed Jonathan Turner Eagles Actions
        11.
        		 Support submission of multiple applications with LocalRunner from within the same JVM Sub-task Closed Siddharth Seth Actions
        12.
        		 Add documentation for LocalMode usage Sub-task Closed Siddharth Seth Actions
        13.
        		 Session mode and ignore-lib-uris needs to be set for session mode Sub-task Closed Siddharth Seth Actions
        14.
        		 Avoid sysexit in the DAGAM in case of local mode Sub-task Closed Siddharth Seth Actions
        15.
        		 Additional supplement for TEZ local mode document Sub-task Closed Chen He Actions
        16.
        		 Disable node blacklisting and ATS in AM for local mode Sub-task Closed Jonathan Turner Eagles Actions
        17.
        		 Tez Local Mode fails if "fs.defaultFS" is not set to be "file:///' in core-site.xml Sub-task Resolved Chen He Actions
        18.
        		 Local Mode crashes on concurrentModificationException Sub-task Closed Chen He Actions
        19.
        		 Add multiDAG session test and move TestLocalMode to tez-tests Sub-task Closed Chen He Actions
        20.
        		 LocalTaskSchedulerService.getTotalResources() and getAvailableResources() can get negative if JVM memory is larger than 2GB Sub-task Closed Chen He Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            airbots Chen He
            airbots Chen He
            Votes:
            6 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment