Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
trunk
-
None
Description
HCatReplicationTest are failing in secure mode with the following error in the application logs:
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:523)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:334)
at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:311)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1554)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3208)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3227)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:504)
... 19 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1552)
... 25 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
Test case Name:
HCatReplicationTest.oneSourceOneTarget
Console log of test case
2016-07-12 08:50:09,236 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@14 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,237 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Try 50 of 50 (InstanceUtil:643) 2016-07-12 08:50:45,280 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@1 status is KILLED on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,280 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@2 status is KILLED on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,280 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@3 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@4 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@5 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@6 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@7 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@8 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@9 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@10 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@11 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@12 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@13 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653) 2016-07-12 08:50:45,281 INFO - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action 0000037-160712052909267-oozie-oozi-C@14 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/ (InstanceUtil:653)
Co-ordinator info:
http://qelog.hortonworks.com/log/nat-r6-ryrs-falcon-multi/test-logs/Falcon-Multicluster/captured_logs/oozie_logs/nat-r6-ryrs-falcon-multi-9.openstacklocal/0000037-160712052909267-oozie-oozi-C-info.log Coordinator Job ID : 0000037-160712052909267-oozie-oozi-C -------------------------------------------------------------------------------- Job Name : FALCON_FEED_REPLICATION_Acdd50630-37cacd58_Acdd50630-d3d8d630 App Path : hdfs://nat-r6-ryrs-falcon-multi-9.openstacklocal:8020/tmp/fs/falcon/workflows/feed/Acdd50630-37cacd58/70ca9590153960affad34071740d6f6a_1468311662316/REPLICATION/Acdd50630-d3d8d630/coordinator.xml Status : KILLED User : hrt_qa Started : Fri Jan 01 20:00:00 UTC 2010 EndTime : Thu Jan 01 00:00:00 UTC 2099 ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Job ID Status Ext ID Err Code Created Nominal Time ---------------------------------------------------------------------------------------------------------------------------------------------------------------- 0000037-160712052909267-oozie-oozi-C@1 KILLED 0000039-160712052909267-oozie-oozi-W null Tue Jul 12 08:21:04 UTC 2016 Fri Jan 01 20:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@2 KILLED 0000040-160712052909267-oozie-oozi-W null Tue Jul 12 08:21:04 UTC 2016 Fri Jan 01 21:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@3 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Fri Jan 01 22:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@4 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Fri Jan 01 23:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@5 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Sat Jan 02 00:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@6 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Sat Jan 02 01:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@7 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Sat Jan 02 02:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@8 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Sat Jan 02 03:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@9 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Sat Jan 02 04:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@10 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Sat Jan 02 05:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@11 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Sat Jan 02 06:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@12 KILLED null null Tue Jul 12 08:21:04 UTC 2016 Sat Jan 02 07:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@13 KILLED null null Tue Jul 12 08:24:30 UTC 2016 Sat Jan 02 08:00:00 UTC 2010 0000037-160712052909267-oozie-oozi-C@14 KILLED null null Tue Jul 12 08:24:30 UTC 2016 Sat Jan 02 09:00:00 UTC 2010 ----------------------------------------------------------------------------------------------------------------------------------------------------------------
0000039-160712052909267-oozie-oozi-W
Actions ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Job ID Status Ext ID Ext Status Err Code ---------------------------------------------------------------------------------------------------------------------------------------------------------------- 0000039-160712052909267-oozie-oozi-W@:start: OK - OK null 0000039-160712052909267-oozie-oozi-W@pre-processing OK job_1468298287018_0041 SUCCEEDED null 0000039-160712052909267-oozie-oozi-W@table-export OK job_1468296204243_0145 SUCCEEDED null 0000039-160712052909267-oozie-oozi-W@replication OK job_1468298287018_0043 SUCCEEDED null 0000039-160712052909267-oozie-oozi-W@table-import ERROR job_1468298287018_0045 FAILED/KILLED JA018 0000039-160712052909267-oozie-oozi-W@failed-post-processing OK job_1468298287018_0046 SUCCEEDED null 0000039-160712052909267-oozie-oozi-W@fail OK - OK E0729 ----------------------------------------------------------------------------------------------------------------------------------------------------------------