Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Cannot Reproduce
-
Impala 2.10.0
-
None
-
ghx-label-9
Description
Catalog failed to create the JNI CatalogObject during start-up in a recent S3 build due to failure to connect to Hive MetaStore:
#0 0x00000038b60328e5 in raise () from /lib64/libc.so.6
#1 0x00000038b60340c5 in abort () from /lib64/libc.so.6
#2 0x000000000295ba24 in google::DumpStackTraceAndExit() ()
#3 0x000000000295249d in google::LogMessage::Fail() ()
#4 0x0000000002953d42 in google::LogMessage::SendToLog() ()
#5 0x0000000002951e77 in google::LogMessage::Flush() ()
#6 0x000000000295543e in google::LogMessageFatal::~LogMessageFatal() ()
#7 0x0000000001242875 in impala::Catalog::Catalog (this=0x7d54f40) at /data/jenkins/workspace/impala-umbrella-build-and-test-s3/repos/Impala/be/src/catalog/catalog.cc:76
#8 0x0000000001203e08 in impala::CatalogServer::Start (this=0x7fff9cf8c380) at /data/jenkins/workspace/impala-umbrella-build-and-test-s3/repos/Impala/be/src/catalog/catalog-server.cc:172
#9 0x00000000011f7b26 in CatalogdMain (argc=6, argv=0x7fff9cf8c848) at /data/jenkins/workspace/impala-umbrella-build-and-test-s3/repos/Impala/be/src/catalog/catalogd-main.cc:84
#10 0x00000000011f70ff in main (argc=6, argv=0x7fff9cf8c848) at /data/jenkins/workspace/impala-umbrella-build-and-test-s3/repos/Impala/be/src/service/daemon-main.cc:41
Catalog::Catalog() {
JniMethodDescriptor methods[] = {
{"<init>", "([B)V", &catalog_ctor_},
.....
jbyteArray cfg_bytes;
ABORT_IF_ERROR(GetThriftBackendGflags(jni_env, &cfg_bytes));
jobject catalog = jni_env->NewObject(catalog_class_, catalog_ctor_, cfg_bytes);
EXIT_IF_EXC(jni_env); <<<--------
ABORT_IF_ERROR(JniUtil::LocalToGlobalRef(jni_env, catalog, &catalog_));
}
Log file created at: 2017/08/03 16:27:21
Running on machine: impala-boost-static-burst-slave-13c7.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0803 16:27:21.601891 10448 catalog.cc:76] java.lang.IllegalStateException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:99)
at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:72)
at org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:168)
at org.apache.impala.catalog.Catalog.<init>(Catalog.java:103)
at org.apache.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:163)
at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:104)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1562)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:93)
... 5 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1560)
... 9 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1560)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:93)
at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:72)
at org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:168)
at org.apache.impala.catalog.Catalog.<init>(Catalog.java:103)
at org.apache.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:163)
at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:104)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 15 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:512)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
... 13 more
(END)