[PIG-2532] Registered classes fail deserialization in frontend - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.10.0, 0.9.3, 0.11
Component/s: None
Labels:
None

Hadoop Flags:

Reviewed

Description

This issue came up while integrating HCatalog with our environment. HCatalog jars are added to the pig command-line with -Dpig.additional.jars but fails (exception below). When added to the pig classpath the error goes away.

We identified the issue as deserialization using the root class loader, not the context class loader set when the thread is created. This causes HCatSchema which is serialized into the context to fail deserialization in the thread.

2012-02-14 21:55:53,936 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6017: java.io.IOException: Deserialization error: org.apache.hcatalog.data.schema.HCatSchema
	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
	at org.apache.pig.impl.util.UDFContext.deserialize(UDFContext.java:181)
	at org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil.setupUDFContext(MapRedUtil.java:159)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:229)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:811)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:771)
	at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
	at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigJobControl.mainLoopAction(PigJobControl.java:144)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigJobControl.run(PigJobControl.java:121)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.data.schema.HCatSchema
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:247)
	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:603)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
	at java.util.Hashtable.readObject(Hashtable.java:859)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
	at java.util.HashMap.readObject(HashMap.java:1030)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53)
	... 15 more