Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2532

Registered classes fail deserialization in frontend

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0, 0.9.3, 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This issue came up while integrating HCatalog with our environment. HCatalog jars are added to the pig command-line with -Dpig.additional.jars but fails (exception below). When added to the pig classpath the error goes away.

      We identified the issue as deserialization using the root class loader, not the context class loader set when the thread is created. This causes HCatSchema which is serialized into the context to fail deserialization in the thread.

      2012-02-14 21:55:53,936 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6017: java.io.IOException: Deserialization error: org.apache.hcatalog.data.schema.HCatSchema
      	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
      	at org.apache.pig.impl.util.UDFContext.deserialize(UDFContext.java:181)
      	at org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil.setupUDFContext(MapRedUtil.java:159)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:229)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
      	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:811)
      	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:771)
      	at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
      	at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigJobControl.mainLoopAction(PigJobControl.java:144)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigJobControl.run(PigJobControl.java:121)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.data.schema.HCatSchema
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:247)
      	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:603)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
      	at java.util.Hashtable.readObject(Hashtable.java:859)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
      	at java.util.HashMap.readObject(HashMap.java:1030)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
      	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53)
      	... 15 more
      

        Attachments

        1. PIG-2532-h23.patch
          2 kB
          Daniel Dai
        2. PIG-2532-log.zip
          28 kB
          Thomas Weise
        3. PIG-2532-v4-branch-0.9.patch
          13 kB
          Julien Le Dem
        4. PIG-2532-v4.patch
          13 kB
          Julien Le Dem
        5. PIG-2532-v3.patch
          14 kB
          Julien Le Dem
        6. PIG-2532-v2.patch
          14 kB
          Travis Crawford
        7. PIG-253_javax.zip
          17 kB
          Travis Crawford
        8. PIG-2532.patch
          2 kB
          Travis Crawford

          Issue Links

            Activity

              People

              • Assignee:
                traviscrawford Travis Crawford
                Reporter:
                traviscrawford Travis Crawford
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: