[TEZ-3892] getClient API for TezClient

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.2
• Assignee: Eric Wohlstadter
• Reporter: Eric Wohlstadter
• Created: 30/Jan/18 21:10
• Updated: 19/Apr/19 19:21
• Resolved: 07/Mar/18 23:53

Description

This is a proposed opt-in feature.

Tez AM already supports long-lived sessions, if desired a AM session can live indefinitely.

However, new clients cannot connect to a long-lived AM session through the standard TezClient API. 

TezClient API only provides a "start" method to initiate a connection, which always allocates a new AM from YARN.

1. For interactive BI use-cases, this startup time can be significant.
2. Hive is implementing a HiveServer2 High Availability feature.
   • When the singleton HS2 master server fails, the HS2 client is quickly redirected to a pre-warmed HS2 backup. 
3. For the failover to complete quickly end-to-end, a Tez AM must also be pre-warmed and ready to accept connections.

For more information, see design for: https://issues.apache.org/jira/browse/HIVE-18281.

---

Anticipated changes:

1. A getClient{{(ApplicationId)}} method is added to TezClient. The functionality is similar to start
   • Code related to launching a new AM from the RM is factored out.
   • Since start and getClient will share some code, this code is refactored into reusable helper methods.
   • A usage example is added to org/apache/tez/examples
2. It is not a goal of this JIRA to ensure that running Tez DAGs can be recovered by a client using the getClient API. The goal is only for maintaining a pool of warm Tez AMs to skip RM/container/JVM startup.

Issue Links

• is depended upon by: HIVE-18952 Tez session disconnect and reconnect on HS2 HA failover
• is required by: HIVE-18281 HiveServer2 HA for LLAP and Workload Manager