Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-2003

[Umbrella] Allow Tez to co-ordinate execution to external services

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0-alpha
    • Component/s: None
    • Labels:
      None

      Description

      The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc.
      This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild).

      The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources.

      A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable.
      A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf).
      This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks.

      Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers.

        Attachments

        1. 2003_20150728.1.txt
          940 kB
          Siddharth Seth
        2. 2003_20150807.1.txt
          1.01 MB
          Siddharth Seth
        3. 2003_20150807.2.txt
          1.04 MB
          Siddharth Seth
        4. 2003_20150812.1.txt
          1.19 MB
          Siddharth Seth
        5. 2003_20150812.2.txt
          1.19 MB
          Siddharth Seth
        6. 2003_20150814.1.txt
          1.03 MB
          Siddharth Seth
        7. 2003_20150814.2.txt
          1.03 MB
          Siddharth Seth
        8. 2003_20150820.1.txt
          1.04 MB
          Siddharth Seth
        9. 2003_20150821.1.txt
          1.04 MB
          Siddharth Seth
        10. Tez With External Services.pdf
          96 kB
          Siddharth Seth

          Issue Links

          1.
          Define basic interface for pluggable ContainerLaunchers Sub-task Closed Siddharth Seth
          2.
          Define basic interface for pluggable TaskSchedulers Sub-task Closed Siddharth Seth
          3.
          Task communication plane needs to be pluggable Sub-task Closed Siddharth Seth
          4.
          Temporarily allow the scheduler and launcher to be specified via configuration Sub-task Closed Siddharth Seth
          5.
          Add some tests for jobs running in external services Sub-task Closed Siddharth Seth
          6.
          Add a manager for ContainerLaunchers running in the AM Sub-task Closed Siddharth Seth
          7.
          Allow for the scheduler, launcher, task communicator to be specified per vertex Sub-task Closed Siddharth Seth
          8.
          Setup pluggable components at AM/Vertex level Sub-task Closed Siddharth Seth
          9.
          Fix component managers to use pluggable components Sub-task Closed Siddharth Seth
          10.
          Change Node tracking to work per external container source Sub-task Closed Siddharth Seth
          11.
          Create a task communicator for local mode. Allow tasks to run in the AM Sub-task Closed Siddharth Seth
          12.
          Add unit tests for verifying multiple schedulers, launchers, communicators Sub-task Closed Siddharth Seth
          13.
          Add additional tests for tasks running in the AM Sub-task Closed Siddharth Seth
          14.
          Fix minor bugs in adding default scheduler, getting launchers Sub-task Closed Siddharth Seth
          15.
          Change version in branch TEZ-2003 Sub-task Closed Siddharth Seth
          16.
          Task priority should be available to the TaskCommunicator plugin Sub-task Closed Siddharth Seth
          17.
          Allow the TaskCommunicator to report failed / killed attempts Sub-task Closed Siddharth Seth
          18.
          Miscellaneous fixes after last reabse Sub-task Closed Siddharth Seth
          19.
          Fixes after rebase 04/07 Sub-task Closed Siddharth Seth
          20.
          Separate TaskReporter into an interface Sub-task Closed Siddharth Seth
          21.
          Allow TaskCommunicators to indicate task/container liveness Sub-task Closed Siddharth Seth
          22.
          Allow TaskCommunicators to subscribe for Vertex updates Sub-task Closed Siddharth Seth
          23.
          Expose additional information in TaskCommunicatorContext Sub-task Closed Siddharth Seth
          24.
          Propagate dag completion to TaskCommunicator Sub-task Closed Siddharth Seth
          25.
          Fixes after rebase 04/28 Sub-task Closed Siddharth Seth
          26.
          Send dag identifier as part of the fetcher request string Sub-task Closed Siddharth Seth
          27.
          LogicalIOProcessorRuntimeTask, RuntimeTask, TezTaskRunner should handle interrupts & carry out necessary cleanups Sub-task Closed Rajesh Balamohan
          28.
          TaskRunner returning before executing the task Sub-task Closed Siddharth Seth
          29.
          rebase 05/08 Sub-task Closed Siddharth Seth
          30.
          Allow tasks to be killed Sub-task Closed Siddharth Seth
          31.
          tez-tools version in the branch is incorrect Sub-task Closed Siddharth Seth
          32.
          Add tests for TezTaskRunner2 Sub-task Closed Siddharth Seth
          33.
          TaskRunner2 should call abort, NPEs while cleaning up tasks Sub-task Closed Siddharth Seth
          34.
          Retrun the status of a kill request in TaskRunner2 Sub-task Closed Siddharth Seth
          35.
          NPE in LogicalIOProcessorRuntimeTask while printing thread info Sub-task Closed Siddharth Seth
          36.
          Inform TaskCommunicaor about Task and Container termination reasons Sub-task Closed Siddharth Seth
          37.
          TezTaskRunner2 not killing tasks properly in all situations Sub-task Closed Siddharth Seth
          38.
          rebase 06/01 Sub-task Closed Siddharth Seth
          39.
          TEZ-2003 build is broken Sub-task Closed Sergey Shelukhin
          40.
          rebase 07/14 Sub-task Closed Siddharth Seth
          41.
          Fix log lines with DEBUG in messages, consolidate TEZ-2003 TODOs Sub-task Closed Siddharth Seth
          42.
          Pluggable services should not extend AbstractService Sub-task Closed Siddharth Seth
          43.
          Cleanup the way services are specified for an AM and vertices Sub-task Closed Siddharth Seth
          44.
          Change service contexts to expose a user specified payload instead of the AM configuration Sub-task Closed Siddharth Seth
          45.
          Add tests for client side changes - specifying plugins, etc Sub-task Closed Siddharth Seth
          46.
          Propagation of errors from plugins to the AM for error reporting Sub-task Closed Siddharth Seth
          47.
          Remove TaskAttempt holder used within TezTaskCommunicator Sub-task Closed Siddharth Seth
          48.
          Add javadocs for new pluggable components, fix problems reported by jenkins Sub-task Closed Siddharth Seth
          49.
          Fix comments from reviews - part 1 Sub-task Closed Siddharth Seth
          50.
          rebase 08/05 Sub-task Closed Siddharth Seth
          51.
          TEZ-2003 build fails Sub-task Closed Jeff Zhang
          52.
          Fix version on tez job analyzer Sub-task Resolved Unassigned
          53.
          Fix comments from reviews - part 2 Sub-task Closed Siddharth Seth
          54.
          renames for tez-2003 changes Sub-task Closed Siddharth Seth
          55.
          Add tests for node handling when there's multiple schedulers Sub-task Closed Siddharth Seth
          56.
          Fix comments from review - part 3 Sub-task Closed Siddharth Seth
          57.
          rebase 08/14 Sub-task Closed Siddharth Seth
          58.
          Fix findbugs warnings Sub-task Closed Siddharth Seth
          59.
          rebase 08/21 Sub-task Closed Siddharth Seth
          60.
          Pre-merge: Update CHANGES.txt and version in branch Sub-task Closed Siddharth Seth
          61.
          Add an onError method to service plugin contexts Sub-task Closed Siddharth Seth
          62.
          Rename TaskAttemptListener and associated classes to TaskCommunicatorManager Sub-task Resolved Siddharth Seth

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                sseth Siddharth Seth
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: