Pig
  1. Pig
  2. PIG-3880

After compiling trunk, I am seeing ClassLoaderObjectInputStream ClassNotFoundException.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: None
    • Component/s: grunt
    • Labels:
      None

      Description

      I pulled trunk from subversion using the following commands:

      mkdir pig
      cd pig
      svn co http://svn.apache.org/repos/asf/pig/trunk
      cd trunk
      ant
      export PATH=$PATH:$HOME/pig/trunk/bin
      export ACCUMULO_HOME=/opt/accumulo
      export HADOOP_HOME=/opt/hadoop
      export PIG_HOME=$HOME/pig/trunk
      export PIG_CLASSPATH="$HOME/pig/trunk/build/ivy/lib/Pig/*"
      export PIG_CLASSPATH="$ACCUMULO_HOME/lib/*:$PIG_CLASSPATH"
      cd ~
      pig

      Then I ran into this error:

      java.lang.NoClassDefFoundError: org/apache/commons/io/input/ClassLoaderObjectInputStream
      at org.apache.pig.Main.run(Main.java:399)

      When I change PIG_JAR to use the fat jar, I was able to run the pig command without getting the exception.

        Activity

        Hide
        Josh Elser added a comment -

        What version of Hadoop are you using, David Medinets? I recall situations on other projects where the dependency management expected certain artifacts to be provided by hadoop when the user's version didn't actually provide that jar. I believe commons-io was one of these artifacts that I was bit by too.

        This seems to be a plausible explanation to what you're seeing. The jarwithhadoop would contain the dependencies and thus you wouldn't have the issues if your local hadoop install was missing necessary jars.

        Show
        Josh Elser added a comment - What version of Hadoop are you using, David Medinets ? I recall situations on other projects where the dependency management expected certain artifacts to be provided by hadoop when the user's version didn't actually provide that jar. I believe commons-io was one of these artifacts that I was bit by too. This seems to be a plausible explanation to what you're seeing. The jarwithhadoop would contain the dependencies and thus you wouldn't have the issues if your local hadoop install was missing necessary jars.
        Hide
        David Medinets added a comment -

        Good point. Perhaps my version of hadoop is too old?

        Hadoop 0.20.203.0
        Subversion
        http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203-r
        1099333
        Compiled by oom on Wed May 4 07:57:50 PDT 2011

        Show
        David Medinets added a comment - Good point. Perhaps my version of hadoop is too old? Hadoop 0.20.203.0 Subversion http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203-r 1099333 Compiled by oom on Wed May 4 07:57:50 PDT 2011
        Hide
        David Medinets added a comment -

        I tried to add the commons-io to my classpath. I got the same error. Here is the dry run showing the jar file in the path.

        $ pig
        dry run:
        HADOOP_CLASSPATH: /home/566453/pig/conf:/usr/java/jdk1.7.0_09/lib/tools.jar:/opt/accumulo/lib/accumulo-core-1.4.2.jar:/opt/accumulo/lib/libthrift-0.6.1.jar:/opt/accumulo/lib/cloudtrace-1.4.2.jar:/opt/zookeeper/zookeeper-3.3.3.jar:/home/566453/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/home/566453/pig/build/ivy/lib/Pig/jython-standalone-2.5.3.jar:/home/566453/pig/build/ivy/lib/Pig/jruby-complete-1.6.7.jar:/home/566453/pig/pig-withouthadoop.jar:
        HADOOP_OPTS: -Xmx1000m -Dpig.log.dir=/home/566453/pig/logs -Dpig.log.file=pig.log -Dpig.home.dir=/home/566453/pig
        /opt/hadoop/bin/hadoop jar /home/566453/pig/pig-withouthadoop.jar

        I tried both commons-io 1.4 and 2.1. I checked that the class is in the jar:

        $ jar tf .m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar | grep ClassLoaderObjectInputStream
        org/apache/commons/io/input/ClassLoaderObjectInputStream.class

        Anything else I can try?

        Show
        David Medinets added a comment - I tried to add the commons-io to my classpath. I got the same error. Here is the dry run showing the jar file in the path. $ pig dry run: HADOOP_CLASSPATH: /home/566453/pig/conf:/usr/java/jdk1.7.0_09/lib/tools.jar:/opt/accumulo/lib/accumulo-core-1.4.2.jar:/opt/accumulo/lib/libthrift-0.6.1.jar:/opt/accumulo/lib/cloudtrace-1.4.2.jar:/opt/zookeeper/zookeeper-3.3.3.jar:/home/566453/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/home/566453/pig/build/ivy/lib/Pig/jython-standalone-2.5.3.jar:/home/566453/pig/build/ivy/lib/Pig/jruby-complete-1.6.7.jar:/home/566453/pig/pig-withouthadoop.jar: HADOOP_OPTS: -Xmx1000m -Dpig.log.dir=/home/566453/pig/logs -Dpig.log.file=pig.log -Dpig.home.dir=/home/566453/pig /opt/hadoop/bin/hadoop jar /home/566453/pig/pig-withouthadoop.jar I tried both commons-io 1.4 and 2.1. I checked that the class is in the jar: $ jar tf .m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar | grep ClassLoaderObjectInputStream org/apache/commons/io/input/ClassLoaderObjectInputStream.class Anything else I can try?
        Hide
        Josh Elser added a comment -

        I'm a bit confused as to what you're showing here. Where is this "dry run:" output coming from? Can you verify that the following does (not) work:

        PIG_CLASSPATH=/home/566453/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar pig -x mapreduce my_script.pig

        Show
        Josh Elser added a comment - I'm a bit confused as to what you're showing here. Where is this "dry run:" output coming from? Can you verify that the following does (not) work: PIG_CLASSPATH=/home/566453/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar pig -x mapreduce my_script.pig
        Hide
        David Medinets added a comment -

        Responding to the dry run question first. On http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html, in the 'Parameter Substitution' section, it describes 'pig -dryrun' as follows: "With this option, the script is not run and a fully substituted Pig script produced in the current working directory named original_script_name.substituted"

        Show
        David Medinets added a comment - Responding to the dry run question first. On http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html , in the 'Parameter Substitution' section, it describes 'pig -dryrun' as follows: "With this option, the script is not run and a fully substituted Pig script produced in the current working directory named original_script_name.substituted"
        Hide
        David Medinets added a comment -
        $ PIG_CLASSPATH=/home/566453/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar pig -x mapreduce my_script.pig
        14/04/23 12:30:26 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
        14/04/23 12:30:26 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
        14/04/23 12:30:26 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
        2014-04-23 12:30:26,793 [main] INFO  org.apache.pig.Main - Apache Pig version 0.13.0-SNAPSHOT (r1587299) compiled Apr 15 2014, 09:16:52
        2014-04-23 12:30:26,793 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/566453/pig_1398270626790.log
        2014-04-23 12:30:26,839 [main] ERROR org.apache.pig.Main - ERROR 2998: Unhandled internal error. org/apache/commons/io/input/ClassLoaderObjectInputStream
        Details at logfile: /home/566453/pig_1398270626790.log
        
        Show
        David Medinets added a comment - $ PIG_CLASSPATH=/home/566453/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar pig -x mapreduce my_script.pig 14/04/23 12:30:26 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 14/04/23 12:30:26 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 14/04/23 12:30:26 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType 2014-04-23 12:30:26,793 [main] INFO org.apache.pig.Main - Apache Pig version 0.13.0-SNAPSHOT (r1587299) compiled Apr 15 2014, 09:16:52 2014-04-23 12:30:26,793 [main] INFO org.apache.pig.Main - Logging error messages to: /home/566453/pig_1398270626790.log 2014-04-23 12:30:26,839 [main] ERROR org.apache.pig.Main - ERROR 2998: Unhandled internal error. org/apache/commons/io/input/ClassLoaderObjectInputStream Details at logfile: /home/566453/pig_1398270626790.log
        Hide
        Josh Elser added a comment -

        I'm not sure what to say at this point. PIG_CLASSPATH should be placing it on the classpath of o.a.p.Main, I don't understand why it's not working for you. Maybe someone else might have some guidance.

        Show
        Josh Elser added a comment - I'm not sure what to say at this point. PIG_CLASSPATH should be placing it on the classpath of o.a.p.Main, I don't understand why it's not working for you. Maybe someone else might have some guidance.
        Hide
        Daniel Dai added a comment -

        Not sure what happen but here is something might be related. Pig fat jar contains commons-io.jar but pig-withouthadoop.jar does not. However, when you run bin/pig, Pig will invoke "hadoop jar pig-withouthadoop.jar", and hadoop will provide commons-io classes in runtime.

        Show
        Daniel Dai added a comment - Not sure what happen but here is something might be related. Pig fat jar contains commons-io.jar but pig-withouthadoop.jar does not. However, when you run bin/pig, Pig will invoke "hadoop jar pig-withouthadoop.jar", and hadoop will provide commons-io classes in runtime.
        Hide
        David Medinets added a comment -

        I feel the consensus is saying; Don't run old versions of hadoop. And I can live with that.

        Show
        David Medinets added a comment - I feel the consensus is saying; Don't run old versions of hadoop. And I can live with that.
        Hide
        Ryan Compton added a comment -

        On Hadoop 0.20.2-cdh3u3 and have the same problem with branch-0.13. The release Pig 12.1 release works fine. I can live with this.

        Show
        Ryan Compton added a comment - On Hadoop 0.20.2-cdh3u3 and have the same problem with branch-0.13. The release Pig 12.1 release works fine. I can live with this.

          People

          • Assignee:
            Unassigned
            Reporter:
            David Medinets
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development