Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8008

InvocationTargetException when making HMS RPCs

    XMLWordPrintableJSON

    Details

    • Epic Color:
      ghx-label-2

      Description

      A bunch of tests failed with the following HMS exception in logs. Looks like a flaky HMS issue but raising this jira to rule out any obvious bugs.

      Failed tests

      authorization.test_grant_revoke.TestGrantRevoke.test_same_name_for_role_and_user_invalidate_metadata
      authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_with_grant[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_disabled[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_without_grant[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_different_cases[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      authorization.test_show_grant.TestShowGrant.test_show_grant_user[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      authorization.test_show_grant.TestShowGrant.test_show_grant_in_hs2[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      query_test.test_insert.TestInsertQueries.test_insert_large_string[protocol: beeswax | exec_option: {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      

      Full stack trace

      2018-12-19T14:29:05,730  WARN [pool-8-thread-3] metastore.ObjectStore: Failed to get database test_owner_privileges_without_grant_fd071aba, returning NoSuchObjectException
      2018-12-19T14:29:05,779  INFO [pool-8-thread-3] common.FileUtils: Creating directory if it doesn't exist: s3a://impala-test-uswest2-2/test-warehouse/test_owner_privileges_without_grant_fd071aba.db
      2018-12-19T14:29:05,950 ERROR [pool-8-thread-3] transport.RetryClientInvocationHandler: failed to execute notifyHmsEvent
      java.lang.reflect.InvocationTargetException: null
        at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
        at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
        at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
        at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) [hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
      Caused by: org.apache.sentry.core.common.exception.SentryUserException: Thrift exception occurred
        at org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1188) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        ... 33 more
      Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        ... 33 more
      2018-12-19T14:29:05,956 ERROR [pool-8-thread-3] transport.RetryClientInvocationHandler: Thrift call failed
      org.apache.thrift.transport.TTransportException: org.apache.thrift.transport.TTransportException
        at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:110) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
        at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
        at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) [hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
      Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
        at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
        at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95) ~[sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
        ... 29 more
      2018-12-19T14:29:06,258  INFO [pool-8-thread-3] metastore.HiveMetaStore: 3: source:127.0.0.1 get_database: test_owner_privileges_without_grant_fd071aba
      2018-12-19T14:29:06,258  INFO [pool-8-thread-3] HiveMetaStore.audit: ugi=jenkins  ip=127.0.0.1  cmd=source:127.0.0.1 get_database: test_owner_privileges_without_grant_fd071aba
      2018-12-19T14:29:06,271  INFO [pool-8-thread-173] metastore.HiveMetaStore: 54: source:127.0.0.1 get_database: owner_priv_db
      

        Attachments

          Activity

            People

            • Assignee:
              fredyw Fredy Wijaya
              Reporter:
              bharathv Bharath Vissapragada
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: