Pig
  1. Pig
  2. PIG-183

Pig could not resolve my udf class

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      When I ran the following Pig script with the latest Pig stuff, I got an exception for unresolved class. I noticed that org.apache.pig.test.udf.storefunc is not among the packages that Pig searches ([, org.apache.pig.builtin., com.yahoo.pig.yst.sds.ULT., org.apache.pig.impl.builtin.]).

      I am sure that the package path for StringStore (org.apache.pig.test.udf.storefunc) is correct in testudf.jar. I attached testudf.jar here with this bug report.

      register /path/to/my/jar/testudf.jar;
      A = load '/user/pig/tests/data/singlefile/textdoc' using org.apache.pig.test.udf.storefunc.StringStore();
      store A into 'results_4_1';
      
      2008-04-03 13:20:25,154 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wilbur11.labs.corp.sp1.yahoo.com:8020
      2008-04-03 13:20:25,781 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.StringStore' with arguments '[]'
              at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:504)
              at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:510)
              at org.apache.pig.impl.io.ValidatingInputFileSpec.getSlicer(ValidatingInputFileSpec.java:50)
              at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:42)
              at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:37)
              at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:795)
              at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:628)
              at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:484)
              at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:334)
              at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
              at org.apache.pig.PigServer.registerQuery(PigServer.java:262)
              at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:446)
              at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:226)
              at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
              at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
              at org.apache.pig.Main.main(Main.java:265)
      Caused by: java.io.IOException: Could not resolve org.apache.pig.test.udf.storefunc.StringStore using imports: [, org.apache.pig.builtin., com.yahoo.pig.yst.sds.ULT., org.apache.pig.impl.builtin.]
              at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:16)
              at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:456)
              at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:486)
              ... 15 more
      Caused by: java.lang.ClassNotFoundException: Could not resolve org.apache.pig.test.udf.storefunc.StringStore using imports: [, org.apache.pig.builtin., com.yahoo.pig.yst.sds.ULT., org.apache.pig.impl.builtin.]
              at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:455)
              ... 16 more
      
      2008-04-03 13:20:25,784 [main] ERROR org.apache.pig.tools.grunt.Grunt - could not instantiate 'org.apache.pig.test.udf.storefunc.StringStore' with arguments '[]'
      
      1. testudf.jar
        6 kB
        Xu Zhang
      2. pig_183_v1.patch
        1 kB
        Pi Song

        Activity

        Alan Gates made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Alan Gates made changes -
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.1.0 [ 12312848 ]
        Hide
        Alan Gates added a comment -

        Fixed checked in revision 648789. Thanks Pi.

        Show
        Alan Gates added a comment - Fixed checked in revision 648789. Thanks Pi.
        Pi Song made changes -
        Patch Info [Patch Available]
        Pi Song made changes -
        Attachment pig_183_v1.patch [ 12379726 ]
        Hide
        Pi Song added a comment -

        Mismatch java versions is something I also come across very often. I think the real problem is that our code does swallow exception. Here is the patch that exposes it as RuntimeException.

        The new error message will look like this:-

        08/04/09 22:29:44 ERROR grunt.Grunt: java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.StringStore' with arguments '[]'
        	at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:509)
        	at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:515)
        	at org.apache.pig.impl.logicalLayer.LOLoad.<init>(LOLoad.java:54)
        	at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:802)
        	at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:635)
        	at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:491)
        	at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:341)
        	at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
        	at org.apache.pig.PigServer.registerQuery(PigServer.java:262)
        	at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:446)
        	at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:226)
        	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
        	at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
        	at org.apache.pig.test.TestGrunt.testUDF(TestGrunt.java:47)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:585)
        	at junit.framework.TestCase.runTest(TestCase.java:164)
        	at junit.framework.TestCase.runBare(TestCase.java:130)
        	at junit.framework.TestResult$1.protect(TestResult.java:110)
        	at junit.framework.TestResult.runProtected(TestResult.java:128)
        	at junit.framework.TestResult.run(TestResult.java:113)
        	at junit.framework.TestCase.run(TestCase.java:120)
        	at junit.framework.TestSuite.runTest(TestSuite.java:228)
        	at junit.framework.TestSuite.run(TestSuite.java:223)
        	at org.junit.internal.runners.OldTestClassRunner.run(OldTestClassRunner.java:35)
        	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:38)
        	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
        Caused by: java.lang.RuntimeException: java.lang.UnsupportedClassVersionError: Bad version number in .class file
        	at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:449)
        	at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:491)
        	... 32 more
        Caused by: java.lang.UnsupportedClassVersionError: Bad version number in .class file
        	at java.lang.ClassLoader.defineClass1(Native Method)
        	at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
        	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
        	at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
        	at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
        	at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        	at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        	at java.lang.Class.forName0(Native Method)
        	at java.lang.Class.forName(Class.java:242)
        	at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:442)
        	... 33 more
        
        Show
        Pi Song added a comment - Mismatch java versions is something I also come across very often. I think the real problem is that our code does swallow exception. Here is the patch that exposes it as RuntimeException. The new error message will look like this:- 08/04/09 22:29:44 ERROR grunt.Grunt: java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.StringStore' with arguments '[]' at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:509) at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:515) at org.apache.pig.impl.logicalLayer.LOLoad.<init>(LOLoad.java:54) at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:802) at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:635) at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:491) at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:341) at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47) at org.apache.pig.PigServer.registerQuery(PigServer.java:262) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:446) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:226) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60) at org.apache.pig.test.TestGrunt.testUDF(TestGrunt.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at junit.framework.TestCase.runTest(TestCase.java:164) at junit.framework.TestCase.runBare(TestCase.java:130) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:120) at junit.framework.TestSuite.runTest(TestSuite.java:228) at junit.framework.TestSuite.run(TestSuite.java:223) at org.junit.internal.runners.OldTestClassRunner.run(OldTestClassRunner.java:35) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:38) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196) Caused by: java.lang.RuntimeException: java.lang.UnsupportedClassVersionError: Bad version number in .class file at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:449) at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:491) ... 32 more Caused by: java.lang.UnsupportedClassVersionError: Bad version number in .class file at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:442) ... 33 more
        Pi Song made changes -
        Assignee Pi Song [ pi_song ]
        Hide
        Pi Song added a comment -

        Xu,

        You compiled your class using Java 6 but your Pig runs on Java 5 JVM.

        pi@pidev2008:~/dump/org/apache/pig/test/udf/storefunc$ javap -verbose StringStore
        Compiled from "StringStore.java"
        public class org.apache.pig.test.udf.storefunc.StringStore extends java.lang.Object implements org.apache.pig.StoreFunc,org.apache.pig.LoadFunc
          SourceFile: "StringStore.java"
          minor version: 0
          major version: 50     <=== This indicates Java 6
          Constant pool:   
        
        
        Show
        Pi Song added a comment - Xu, You compiled your class using Java 6 but your Pig runs on Java 5 JVM. pi@pidev2008:~/dump/org/apache/pig/test/udf/storefunc$ javap -verbose StringStore Compiled from "StringStore.java" public class org.apache.pig.test.udf.storefunc.StringStore extends java.lang.Object implements org.apache.pig.StoreFunc,org.apache.pig.LoadFunc SourceFile: "StringStore.java" minor version: 0 major version: 50 <=== This indicates Java 6 Constant pool:
        Hide
        Xu Zhang added a comment -

        How so? As you can see from the attached testudf.jar, the StringStore class can be correctly found in the org.apache.pig.test.udf.storefunc package.

        Show
        Xu Zhang added a comment - How so? As you can see from the attached testudf.jar, the StringStore class can be correctly found in the org.apache.pig.test.udf.storefunc package.
        Hide
        Olga Natkovich added a comment -

        My guess would be that your UDF.jar is not properly built.

        Show
        Olga Natkovich added a comment - My guess would be that your UDF.jar is not properly built.
        Xu Zhang made changes -
        Field Original Value New Value
        Attachment testudf.jar [ 12379319 ]
        Xu Zhang created issue -

          People

          • Assignee:
            Pi Song
            Reporter:
            Xu Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development