[TEZ-2003] [Umbrella] Allow Tez to co-ordinate execution to external services

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0-alpha
• Assignee: Unassigned
• Reporter: Siddharth Seth
• Created: 28/Jan/15 00:25
• Updated: 18/Aug/16 23:05
• Resolved: 18/Aug/16 23:05

Description

The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc.
This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild).

The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources.

A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable.
A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf).
This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks.

Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers.

Issue Links

• is related to: TEZ-1876 [Umbrella] Polish up support for local mode in Tez
• is related to: TEZ-1184 [Umbrella] Support for multiple tasks running in the same JVM
• is related to: TEZ-2212 Notify components on DAG completion
• is related to: TEZ-2130 Send the sessionToken as part of the AM CLC
• relates to: TEZ-2664 Remove YARN constructs from various APIs
• relates to: TEZ-2169 Add NDC context to various threads and pools
• relates to: TEZ-2045 TaskAttemptListener should not pull Tasks from AMContainer
• requires: TEZ-1233 Allow configuration of framework parameters per vertex

Sub-Tasks

• Define basic interface for pluggable ContainerLaunchers: Closed
• Define basic interface for pluggable TaskSchedulers: Closed
• Task communication plane needs to be pluggable: Closed
• Temporarily allow the scheduler and launcher to be specified via configuration: Closed
• Add some tests for jobs running in external services: Closed
• Add a manager for ContainerLaunchers running in the AM: Closed
• Allow for the scheduler, launcher, task communicator to be specified per vertex: Closed
• Setup pluggable components at AM/Vertex level: Closed
• Fix component managers to use pluggable components: Closed
• Change Node tracking to work per external container source: Closed
• Create a task communicator for local mode. Allow tasks to run in the AM: Closed
• Add unit tests for verifying multiple schedulers, launchers, communicators: Closed
• Add additional tests for tasks running in the AM: Closed
• Fix minor bugs in adding default scheduler, getting launchers: Closed
• Change version in branch TEZ-2003: Closed
• Task priority should be available to the TaskCommunicator plugin: Closed
• Allow the TaskCommunicator to report failed / killed attempts: Closed
• Miscellaneous fixes after last reabse: Closed
• Fixes after rebase 04/07: Closed
• Separate TaskReporter into an interface: Closed
• Allow TaskCommunicators to indicate task/container liveness: Closed
• Allow TaskCommunicators to subscribe for Vertex updates: Closed
• Expose additional information in TaskCommunicatorContext: Closed
• Propagate dag completion to TaskCommunicator: Closed
• Fixes after rebase 04/28: Closed
• Send dag identifier as part of the fetcher request string: Closed
• LogicalIOProcessorRuntimeTask, RuntimeTask, TezTaskRunner should handle interrupts & carry out necessary cleanups: Closed
• TaskRunner returning before executing the task: Closed
• rebase 05/08: Closed
• Allow tasks to be killed: Closed
• tez-tools version in the branch is incorrect: Closed
• Add tests for TezTaskRunner2: Closed
• TaskRunner2 should call abort, NPEs while cleaning up tasks: Closed
• Retrun the status of a kill request in TaskRunner2: Closed
• NPE in LogicalIOProcessorRuntimeTask while printing thread info: Closed
• Inform TaskCommunicaor about Task and Container termination reasons: Closed
• TezTaskRunner2 not killing tasks properly in all situations: Closed
• rebase 06/01: Closed
• TEZ-2003 build is broken: Closed
• rebase 07/14: Closed
• Fix log lines with DEBUG in messages, consolidate TEZ-2003 TODOs: Closed
• Pluggable services should not extend AbstractService: Closed
• Cleanup the way services are specified for an AM and vertices: Closed
• Change service contexts to expose a user specified payload instead of the AM configuration: Closed
• Add tests for client side changes - specifying plugins, etc: Closed
• Propagation of errors from plugins to the AM for error reporting: Closed
• Remove TaskAttempt holder used within TezTaskCommunicator: Closed
• Add javadocs for new pluggable components, fix problems reported by jenkins: Closed
• Fix comments from reviews - part 1: Closed
• rebase 08/05: Closed
• TEZ-2003 build fails: Closed
• Fix version on tez job analyzer: Resolved
• Fix comments from reviews - part 2: Closed
• renames for tez-2003 changes: Closed
• Add tests for node handling when there's multiple schedulers: Closed
• Fix comments from review - part 3: Closed
• rebase 08/14: Closed
• Fix findbugs warnings: Closed
• rebase 08/21: Closed
• Pre-merge: Update CHANGES.txt and version in branch: Closed
• Add an onError method to service plugin contexts: Closed
• Rename TaskAttemptListener and associated classes to TaskCommunicatorManager: Resolved