[HCATALOG-199] HCatOutputCommitter::cleanupJob() fails : Unable to connect to metastore.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.2, 0.3
• Fix Version/s: 0.3
• Component/s: mapreduce
• Assignee: Mithun Radhakrishnan
• Reporter: Mithun Radhakrishnan
• Created: 06/Jan/12 21:02
• Updated: 17/May/12 01:20
• Resolved: 12/Jan/12 23:59

Description

While running a Pig query, we ran into the following failure trace:

<trace>
2012-01-06 18:21:20,362 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoopqa (auth:SIMPLE) cause:org.apache.hcatalog.common.HCatException : 2006 : Error adding partition to metastore. Cause : MetaException(message:Could not connect to meta store using any of the URIs provided)
2012-01-06 18:21:20,363 WARN org.apache.hadoop.mapred.Child: Error running child
org.apache.hcatalog.common.HCatException : 2006 : Error adding partition to metastore. Cause : MetaException(message:Could not connect to meta store using any of the URIs provided)
at org.apache.hcatalog.mapreduce.HCatOutputCommitter.cleanupJob(HCatOutputCommitter.java:407)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.cleanupJob(PigOutputCommitter.java:154)
at org.apache.hadoop.mapreduce.OutputCommitter.commitJob(OutputCommitter.java:76)
at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1055)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:357)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1082)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:185)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:153)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:94)
at org.apache.hcatalog.mapreduce.HCatOutputFormat.createHiveClient(HCatOutputFormat.java:418)
at org.apache.hcatalog.mapreduce.HCatOutputCommitter.cleanupJob(HCatOutputCommitter.java:286)
... 9 more

</trace>

There was no problem connecting to the metastore from the client-side (i.e. Pig). The problem was only when the HCatOutputCommitter attempted to connect, during cleanup.

The cause boiled down to the delegation-token-signature not being available in the conf. A patch is forthcoming.