[TEZ-3991] Unmanaged tez sessions

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.10.0
• Labels: Kubernetes
• Target Version/s: 0.10.1
• Assignee: Eric Wohlstadter
• Reporter: Prasanth Jayachandran
• Created: 13/Sep/18 20:01
• Updated: 04/Apr/19 04:00

Description

Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, tez AMs can register itself with Zookeeper which clients (like HiveServer2) can discover via zk registry client. 

HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode will let AM come up on their own (can be via simple java launcher) and be discoverable for others. 

Example use case for this is, HiveServer2 can discover already running AMs and can attach to it for DAG submission and detach when done executing queries. AMs can similarly discover LLAP daemons via task scheduler plugin for submitting tasks. 

A mode to cut off interactions with RM will also useful since for LLAP no on-demand containers are required.

Sub-Tasks

• Make FrameworkClient Pluggable: Resolved
• Zookeeper based FrameworkServices and AmExtensions (3/3): Open
• Pluggable AM FrameworkServices and AmExtensions (2/3): Open
• Zookeeper based AM Registry: Resolved
• Allow DAGAppMaster to read configuration from plaintext (1/3): Patch Available
• Make TEZ_AM_LOCAL_RESOURCES_PB optional for unmanaged AM: Resolved