Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3892

getClient API for TezClient

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9.2
    • None
    • None

    Description

      This is a proposed opt-in feature.

      Tez AM already supports long-lived sessions, if desired a AM session can live indefinitely.

      However, new clients cannot connect to a long-lived AM session through the standard TezClient API. 

      TezClient API only provides a "start" method to initiate a connection, which always allocates a new AM from YARN.

      1. For interactive BI use-cases, this startup time can be significant.
      2. Hive is implementing a HiveServer2 High Availability feature.
        • When the singleton HS2 master server fails, the HS2 client is quickly redirected to a pre-warmed HS2 backup. 
      3. For the failover to complete quickly end-to-end, a Tez AM must also be pre-warmed and ready to accept connections.

      For more information, see design for: https://issues.apache.org/jira/browse/HIVE-18281.


      Anticipated changes:

      1. A getClient{{(ApplicationId)}} method is added to TezClient. The functionality is similar to start
        • Code related to launching a new AM from the RM is factored out.
        • Since start and getClient will share some code, this code is refactored into reusable helper methods.
        • A usage example is added to org/apache/tez/examples
      2. It is not a goal of this JIRA to ensure that running Tez DAGs can be recovered by a client using the getClient API. The goal is only for maintaining a pool of warm Tez AMs to skip RM/container/JVM startup.

      Attachments

        1. TEZ-3892.1.patch
          16 kB
          Eric Wohlstadter
        2. TEZ-3892.2.patch
          21 kB
          Eric Wohlstadter

        Issue Links

          Activity

            People

              ewohlstadter Eric Wohlstadter
              ewohlstadter Eric Wohlstadter
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: