Pig
  1. Pig
  2. PIG-3729

Need to rebuild pig 0.12.0 jar from mvnrepository.com to resove error in hadoop 2.2 0

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Not A Problem
    • Affects Version/s: 0.12.0
    • Fix Version/s: None
    • Component/s: build, internal-udfs
    • Environment:

      debian single node pseudo distributed hadoop 2.2 cluster

      Description

      Unable to get pig 0.12.0 to run single node pseudo distributed hadoop 2.2 cluster using the jar from mvnrepository.com
      this is the error

      "org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias"

      if I download the jar from
      http://apache.claz.org/pig/pig-0.12.0/pig-0.12.0.tar.gz

      and then recompile the src with "ant clean jar -Dhadoopversion=23"
      when I install the recompiled jar in .m2 everything works

      If it is the case that the Apache hadoop 2.2.0 jars are backwardly compatible then could some one comment on what could be the issue?

        Activity

        Hide
        Daniel Dai added a comment -

        I see PIG-3738 opened for documentation. Close this ticket.

        Show
        Daniel Dai added a comment - I see PIG-3738 opened for documentation. Close this ticket.
        Hide
        Daniel Dai added a comment -

        Yes, document are definitely desired. Reopen the ticket for that.

        It is a developer document, feel like a new entry in https://cwiki.apache.org/confluence/display/PIG/Developer+Documentation

        Show
        Daniel Dai added a comment - Yes, document are definitely desired. Reopen the ticket for that. It is a developer document, feel like a new entry in https://cwiki.apache.org/confluence/display/PIG/Developer+Documentation
        Hide
        jay vyas added a comment -

        I think this is a still an open documentation issue.

        The use of this particular maven dependency hook:

        <groupId>org.apache.pig</groupId>
          <artifactId>pig</artifactId>
          <classifier>h2</classifier>
          <version>0.12.0</version>
          <scope>provided</scope>
        

        Doesnt seem to be explained anywhere.

        What is the right place in the Pig documentation for it? Then someone can put it in as a wiki update or patch.

        Show
        jay vyas added a comment - I think this is a still an open documentation issue. The use of this particular maven dependency hook: <groupId>org.apache.pig</groupId> <artifactId>pig</artifactId> <classifier>h2</classifier> <version>0.12.0</version> <scope>provided</scope> Doesnt seem to be explained anywhere. What is the right place in the Pig documentation for it? Then someone can put it in as a wiki update or patch.
        Hide
        Ashlee Lee added a comment -

        Nezih, my application works finally. The only difference that I did is to add a slash to the end of my classpath in my pom.xml.
        It should be "$HADOOP_HOME/etc/hadoop/", not "$HADOOP_HOME/etc/hadoop".

        Show
        Ashlee Lee added a comment - Nezih, my application works finally. The only difference that I did is to add a slash to the end of my classpath in my pom.xml. It should be "$HADOOP_HOME/etc/hadoop/", not "$HADOOP_HOME/etc/hadoop".
        Hide
        Nezih Yigitbasi added a comment -

        Can you use -secretDebugCmd to see what gets into the classpath (pig -secretDebugCmd)?

        Show
        Nezih Yigitbasi added a comment - Can you use -secretDebugCmd to see what gets into the classpath (pig -secretDebugCmd)?
        Hide
        Ashlee Lee added a comment -

        BTW, I tried using hadoop 1.2.1 + Pig 0.12.0 for my case, it is working......

        Show
        Ashlee Lee added a comment - BTW, I tried using hadoop 1.2.1 + Pig 0.12.0 for my case, it is working......
        Hide
        Ashlee Lee added a comment -

        Thanks for your reply! And sorry for the mistyping PIG_CLASS_PATH, I export PIG_CLASSPATH and HADOOP_CONF_DIR (which is equal to $HADOOP_HOME/etc/hadoop in hadoop 2.2.0), I and tried to add $HADOOP_HOME/etc/hadoop (which is not /conf anymore in hadoop 2.2.0) to the classpath, but it is still not working...

        Show
        Ashlee Lee added a comment - Thanks for your reply! And sorry for the mistyping PIG_CLASS_PATH, I export PIG_CLASSPATH and HADOOP_CONF_DIR (which is equal to $HADOOP_HOME/etc/hadoop in hadoop 2.2.0), I and tried to add $HADOOP_HOME/etc/hadoop (which is not /conf anymore in hadoop 2.2.0) to the classpath, but it is still not working...
        Hide
        Nezih Yigitbasi added a comment -

        Ashlee, seems like your classpath doesn't contain $HADOOP_HOME/conf so Pig can't find them. Can you try exporting PIG_CLASSPATH (no underscores between CLASS and PATH) with $HADOOP_HOME/conf and try again (assuming HADOOP_HOME is set properly)?

        Show
        Nezih Yigitbasi added a comment - Ashlee, seems like your classpath doesn't contain $HADOOP_HOME/conf so Pig can't find them. Can you try exporting PIG_CLASSPATH (no underscores between CLASS and PATH) with $HADOOP_HOME/conf and try again (assuming HADOOP_HOME is set properly)?
        Hide
        Ashlee Lee added a comment -

        Different situation for me. I tried both pig-0.12.0.jar in mvnrepository and my recomplied pig jar, but when I run my jar which use PigServer to run pigscripts, it failed with ERROR 4010: Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath). I have PIG_CLASS_PATH and HADOOP_CONF_DIR setted.

        Then I tried above h2 version dependency, it comes error>java.lang.NoClassDefFoundError: org/apache/pig/PigServer.

        My jar is working fine with Pig 0.11.0 and Hadoop 1.2.0, but now I want to run in Pig 0.12.0 and Hadoop 2.2.0.......I need help!

        Show
        Ashlee Lee added a comment - Different situation for me. I tried both pig-0.12.0.jar in mvnrepository and my recomplied pig jar, but when I run my jar which use PigServer to run pigscripts, it failed with ERROR 4010: Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath). I have PIG_CLASS_PATH and HADOOP_CONF_DIR setted. Then I tried above h2 version dependency, it comes error>java.lang.NoClassDefFoundError: org/apache/pig/PigServer. My jar is working fine with Pig 0.11.0 and Hadoop 1.2.0, but now I want to run in Pig 0.12.0 and Hadoop 2.2.0.......I need help!
        Hide
        jay vyas added a comment -

        Yes....... thanks Cheolsoo Park !!!

        Next step : I think there is some confusoin on this : Maybe this JIRA still has an active component, which is to clarify in the pig docs, somewhere, how pig depends on hadoop, and what the pig 0.12.0 artifact is actually designed to be used?

        Show
        jay vyas added a comment - Yes....... thanks Cheolsoo Park !!! Next step : I think there is some confusoin on this : Maybe this JIRA still has an active component, which is to clarify in the pig docs, somewhere, how pig depends on hadoop, and what the pig 0.12.0 artifact is actually designed to be used?
        Hide
        Nigel Savage added a comment -

        thank you once again for your very prompt and helpful response

        Show
        Nigel Savage added a comment - thank you once again for your very prompt and helpful response
        Hide
        Cheolsoo Park added a comment -

        Here I do it-

        <dependency>
          <groupId>org.apache.pig</groupId>
          <artifactId>pig</artifactId>
          <classifier>h2</classifier>
          <version>0.12.0</version>
          <scope>provided</scope>
        </dependency>
        
        Show
        Cheolsoo Park added a comment - Here I do it- <dependency> <groupId>org.apache.pig</groupId> <artifactId>pig</artifactId> <classifier>h2</classifier> <version>0.12.0</version> <scope>provided</scope> </dependency>
        Hide
        Nigel Savage added a comment -

        thank you very much for your prompt and very helpful response, I was hoping that I could find a dependency for the pom that would work, I have searched around for a dependency like this for the pig-h2 jar, searching thru this jira I think I need something like
        <dependency>
        <groupId>org.apache.pig</groupId>
        <artifactId>pig</artifactId>
        <version>0.12.0</version>
        <classifier>h23</classifier>
        </dependency>
        however this does not work
        does anyone know the correct for for the maven dependency tag

        Show
        Nigel Savage added a comment - thank you very much for your prompt and very helpful response, I was hoping that I could find a dependency for the pom that would work, I have searched around for a dependency like this for the pig-h2 jar, searching thru this jira I think I need something like <dependency> <groupId>org.apache.pig</groupId> <artifactId>pig</artifactId> <version>0.12.0</version> <classifier>h23</classifier> </dependency> however this does not work does anyone know the correct for for the maven dependency tag
        Hide
        Cheolsoo Park added a comment -

        Nigel Savage, did you try pig-0.12.0-h2.jar? The -h2 jar is compiled with hadoopversion=23.

        Show
        Cheolsoo Park added a comment - Nigel Savage , did you try pig-0.12.0-h2.jar ? The -h2 jar is compiled with hadoopversion=23.

          People

          • Assignee:
            Unassigned
            Reporter:
            Nigel Savage
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development