Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16450

Some metastore operations are not retried even with desired underlining exceptions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      In RetryingHMSHandler class, we are expecting the operations should retry when the cause of MetaException is JDOException or NucleusException.

              if (e.getCause() instanceof MetaException && e.getCause().getCause() != null) {
                if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
                    e.getCause().getCause() instanceof NucleusException) {
                  // The JDOException or the Nucleus Exception may be wrapped further in a MetaException
                  caughtException = e.getCause().getCause();
             }
      

      While in ObjectStore, many places we are only throwing new MetaException(msg) without the cause, so we are missing retrying for some cases. e.g., with the following JDOException, we should retry but it's ignored.

      2017-04-04 17:28:21,602 ERROR metastore.ObjectStore (ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving statistics via jdo
      javax.jdo.JDOException: Exception thrown when executing query
              at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
              at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
              at org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
              at org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
              at org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
              at org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
              at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
              at org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
              at org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
              at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
              at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
              at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
              at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
              at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
              at com.sun.proxy.$Proxy3.alter_table_with_environment_context(Unknown Source)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9617)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9601)
              at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
              at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
              at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
              at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
              at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        1. HIVE-16450.1.patch
          8 kB
          Aihua Xu
        2. HIVE-16450.2.patch
          9 kB
          Aihua Xu
        3. HIVE-16450.3.patch
          8 kB
          Aihua Xu
        4. HIVE-16450.4.patch
          10 kB
          Aihua Xu

          Issue Links

            Activity

              People

              • Assignee:
                aihuaxu Aihua Xu
                Reporter:
                aihuaxu Aihua Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: