Pig
  1. Pig
  2. PIG-2532

Registered classes fail deserialization in frontend

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0, 0.9.3, 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This issue came up while integrating HCatalog with our environment. HCatalog jars are added to the pig command-line with -Dpig.additional.jars but fails (exception below). When added to the pig classpath the error goes away.

      We identified the issue as deserialization using the root class loader, not the context class loader set when the thread is created. This causes HCatSchema which is serialized into the context to fail deserialization in the thread.

      2012-02-14 21:55:53,936 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6017: java.io.IOException: Deserialization error: org.apache.hcatalog.data.schema.HCatSchema
      	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
      	at org.apache.pig.impl.util.UDFContext.deserialize(UDFContext.java:181)
      	at org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil.setupUDFContext(MapRedUtil.java:159)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:229)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
      	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:811)
      	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:771)
      	at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
      	at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigJobControl.mainLoopAction(PigJobControl.java:144)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigJobControl.run(PigJobControl.java:121)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.data.schema.HCatSchema
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:247)
      	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:603)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
      	at java.util.Hashtable.readObject(Hashtable.java:859)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
      	at java.util.HashMap.readObject(HashMap.java:1030)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
      	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53)
      	... 15 more
      
      1. PIG-253_javax.zip
        17 kB
        Travis Crawford
      2. PIG-2532.patch
        2 kB
        Travis Crawford
      3. PIG-2532-h23.patch
        2 kB
        Daniel Dai
      4. PIG-2532-log.zip
        28 kB
        Thomas Weise
      5. PIG-2532-v2.patch
        14 kB
        Travis Crawford
      6. PIG-2532-v3.patch
        14 kB
        Julien Le Dem
      7. PIG-2532-v4.patch
        13 kB
        Julien Le Dem
      8. PIG-2532-v4-branch-0.9.patch
        13 kB
        Julien Le Dem

        Issue Links

          Activity

          Daniel Dai made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Thomas Weise made changes -
          Link This issue relates to PIG-2576 [ PIG-2576 ]
          Daniel Dai made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.11 [ 12318878 ]
          Resolution Fixed [ 1 ]
          Daniel Dai made changes -
          Attachment PIG-2532-h23.patch [ 12517484 ]
          Thomas Weise made changes -
          Attachment PIG-2532-log.zip [ 12516779 ]
          Thomas Weise made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Julien Le Dem made changes -
          Fix Version/s 0.9.3 [ 12319456 ]
          Julien Le Dem made changes -
          Attachment PIG-2532-v4-branch-0.9.patch [ 12516128 ]
          Julien Le Dem made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.10 [ 12316246 ]
          Resolution Fixed [ 1 ]
          Julien Le Dem made changes -
          Attachment PIG-2532-v4.patch [ 12516013 ]
          Julien Le Dem made changes -
          Attachment PIG-2532-v3.patch [ 12515643 ]
          Travis Crawford made changes -
          Attachment PIG-2532-v2.patch [ 12515026 ]
          Travis Crawford made changes -
          Attachment PIG-253_javax.zip [ 12514929 ]
          Travis Crawford made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Travis Crawford made changes -
          Attachment PIG-2532.patch [ 12514588 ]
          Travis Crawford made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          Travis Crawford created issue -

            People

            • Assignee:
              Travis Crawford
              Reporter:
              Travis Crawford
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development