[TEZ-3892] getClient API for TezClient - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

This is a proposed opt-in feature.

Tez AM already supports long-lived sessions, if desired a AM session can live indefinitely.

However, new clients cannot connect to a long-lived AM session through the standard TezClient API.

TezClient API only provides a "start" method to initiate a connection, which always allocates a new AM from YARN.

For interactive BI use-cases, this startup time can be significant.
Hive is implementing a HiveServer2 High Availability feature.
- When the singleton HS2 master server fails, the HS2 client is quickly redirected to a pre-warmed HS2 backup.
For the failover to complete quickly end-to-end, a Tez AM must also be pre-warmed and ready to accept connections.

Anticipated changes:

A getClient{{(ApplicationId)}} method is added to TezClient. The functionality is similar to start
- Code related to launching a new AM from the RM is factored out.
- Since start and getClient will share some code, this code is refactored into reusable helper methods.
- A usage example is added to org/apache/tez/examples
It is not a goal of this JIRA to ensure that running Tez DAGs can be recovered by a client using the getClient API. The goal is only for maintaining a pool of warm Tez AMs to skip RM/container/JVM startup.