[IMPALA-8008] InvocationTargetException when making HMS RPCs - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: Impala 3.2.0
Fix Version/s: Impala 3.2.0
Component/s: Frontend
Labels:
- broken-build
- flaky

Epic Color:
ghx-label-2

Description

A bunch of tests failed with the following HMS exception in logs. Looks like a flaky HMS issue but raising this jira to rule out any obvious bugs.

Failed tests

authorization.test_grant_revoke.TestGrantRevoke.test_same_name_for_role_and_user_invalidate_metadata
authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_with_grant[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_disabled[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_without_grant[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_different_cases[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
authorization.test_show_grant.TestShowGrant.test_show_grant_user[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
authorization.test_show_grant.TestShowGrant.test_show_grant_in_hs2[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
query_test.test_insert.TestInsertQueries.test_insert_large_string[protocol: beeswax | exec_option: {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: text/none]

Full stack trace

2018-12-19T14:29:05,730  WARN [pool-8-thread-3] metastore.ObjectStore: Failed to get database test_owner_privileges_without_grant_fd071aba, returning NoSuchObjectException
2018-12-19T14:29:05,779  INFO [pool-8-thread-3] common.FileUtils: Creating directory if it doesn't exist: s3a://impala-test-uswest2-2/test-warehouse/test_owner_privileges_without_grant_fd071aba.db
2018-12-19T14:29:05,950 ERROR [pool-8-thread-3] transport.RetryClientInvocationHandler: failed to execute notifyHmsEvent
java.lang.reflect.InvocationTargetException: null
  at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
  at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
  at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
  at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
  at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
  at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) [hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
  at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.apache.sentry.core.common.exception.SentryUserException: Thrift exception occurred
  at org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1188) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  ... 33 more
Caused by: org.apache.thrift.transport.TTransportException
  at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  ... 33 more
2018-12-19T14:29:05,956 ERROR [pool-8-thread-3] transport.RetryClientInvocationHandler: Thrift call failed
org.apache.thrift.transport.TTransportException: org.apache.thrift.transport.TTransportException
  at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:110) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
  at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195) [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
  at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
  at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
  at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) [hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
  at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.apache.thrift.transport.TTransportException
  at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
  at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184) ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
  at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95) ~[sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
  ... 29 more
2018-12-19T14:29:06,258  INFO [pool-8-thread-3] metastore.HiveMetaStore: 3: source:127.0.0.1 get_database: test_owner_privileges_without_grant_fd071aba
2018-12-19T14:29:06,258  INFO [pool-8-thread-3] HiveMetaStore.audit: ugi=jenkins  ip=127.0.0.1  cmd=source:127.0.0.1 get_database: test_owner_privileges_without_grant_fd071aba
2018-12-19T14:29:06,271  INFO [pool-8-thread-173] metastore.HiveMetaStore: 54: source:127.0.0.1 get_database: owner_priv_db

Attachments

Activity

People

Assignee:: Fredy Wijaya

Reporter:: Bharath Vissapragada

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 19/Dec/18 23:13

Updated:: 16/Jan/19 07:55

Resolved:: 03/Jan/19 20:53