[HCATALOG-380] If pig script does load then order by, hive-site.xml doesn't seem to propagate properly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4, 0.5, 0.4.1
• Fix Version/s: 0.4.1
• Component/s: pig
• Assignee: Francis Christopher Liu
• Reporter: David Capwell
• Created: 19/Apr/12 18:09
• Updated: 09/May/12 23:43
• Resolved: 09/May/12 23:43

Description

Table is not partitioned and has an RCFile for the data.

The following pig script will cause the MR jobs to fail:
a = load 'default.table' USING org.apache.hcatalog.pig.HCatLoader();
b = order a by id;
dump b;

If I prefix the order by with a foreach statement, then the job will pass.

The MR job fails with the following exception:

Error: java.lang.ClassNotFoundException: javax.jdo.JDOException at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:346) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:333) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:371) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:278) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:248) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:114) at org.apache.hcatalog.mapreduce.InitializeInput.createHiveMetaClient(InitializeInput.java:58) at org.apache.hcatalog.mapreduce.InitializeInput.getSerializedHcatKeyJobInfo(InitializeInput.java:85) at org.apache.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:73) at org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:40) at org.apache.hcatalog.pig.HCatLoader.setLocation(HCatLoader.java:116) at org.apache.pig.impl.builtin.SampleLoader.setLocation(SampleLoader.java:98) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:134) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:112) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:489) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.mapred.Child.main(Child.java:249)

Issue Links

• is related to: PIG-2666 LoadFunc.setLocation() is not called when pig script only has Order By