Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3904

an API to update tokens for Tez AM and the DAG

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Nothing is permanent in this world, lest of all delegation tokens.
      The current way around token expiration (the one where you cannot keep renewing anymore) in Hive when Tez AM is used in session mode is to cycle Tez AM. It may happen though that a query is running at that time, and so the AM cannot be restarted with new tokens. We let the query run its course and it usually dies because it tries to do something with an expired token.
      To get around that, we cycle AMs a few hours before tokens are going to expire.
      However, that is still not ideal because it puts an upper bound on safe Hive query runtime (a query longer than 3 hours with current config may fail due to an expired token if its timing is unlucky), and also precludes setting tokens to expire much faster than the standard 7-day time frame.

      There should be a mechanism to replace tokens in the AM, including for a running DAG.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                sershe Sergey Shelukhin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: