Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23439

Hive sessions over 24 hours encounter Kerberos-related StatsTask errors

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Labels:
      None

      Description

      We have an application that uses Hive via JDBC. The interesting thing about them is that they have sessions that are established with HiveServer2 for multiple days. After 24 hours, their queries are failing with StatsTask-related errors. From looking in the logs, it looks like the communication breaks down between HiveServer2 and the MetaStore.

      Below is error seen:

      2020-04-22T21:25:53,248 ERROR [Thread-1202599]: exec.StatsTask (:()) - Failed to run stats task
      org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table tennis. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
       at org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4927) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:189) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:86) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:108) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:82) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table tennis. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
       at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1387) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       ... 6 more
      Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
       at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4790) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4858) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4838) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1378) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       at org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
       ... 6 more
      

      The problem appears to be because of delegation token issued by Hive Metastore could not be renewed by HiveServer2 within 24 hours period of time.
      There is similar issue reported in upstream HIVE-22033 which adderesses similar kind of issue, I backported that fix on my local cluster and deployed the same, but that does not seems to adderess the issue. Problem seems to be GC interval which removed th DT from HMS after expiry which by default is set to 1 hour.

      "hive.cluster.delegation.token.gc-interval", 1, TimeUnit.HOURS, "")

        Attachments

        1. HIVE-23439.patch
          1 kB
          Rajkumar Singh

          Activity

            People

            • Assignee:
              Rajkumar Singh Rajkumar Singh
              Reporter:
              chiran54321 Chiran Ravani
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: