Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4183

Fix classpath error when using pig command with Spark

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      You can reproduce the bug in following steps
      1.Build spark -0.9 env and build hadoop1 env:
      2.Compile code: ant jar
      3.Export $PIG_CLASSPATH
      echo $PIG_CLASSPATH
      /home/zly/prj/oss/pig/build/ivy/lib/Pig/*:/home/zly/prj/oss/hadoop-1.2.1/conf
      4.Run: cd $PIG_HOME/bin; ./pig –x spark id.spark.pig
      5.Error message found in pig log
      549 ERROR 2998: Unhandled internal error. org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
      550
      551 java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
      552 at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:607)
      553 at org.eclipse.jetty.util.log.JettyAwareLogger.warn(JettyAwareLogger.java:431)
      554 at org.eclipse.jetty.util.log.Slf4jLog.warn(Slf4jLog.java:69)
      555 at org.eclipse.jetty.util.component.AbstractLifeCycle.setFailed(AbstractLifeCycle.java:204)
      556 at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:74)
      557 at org.apache.spark.HttpServer.start(HttpServer.scala:65)
      558 at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:130)
      559 at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:101)
      560 at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcast.scala:70)
      561 at org.apache.spark.broadcast.BroadcastManager.initialize(Broadcast.scala:81)
      562 at org.apache.spark.broadcast.BroadcastManager.<init>(Broadcast.scala:68)
      563 at org.apache.spark.SparkEnv$.create(SparkEnv.scala:175)
      564 at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
      565 at org.apache.spark.SparkContext.<init>(SparkContext.scala:100)
      566 at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:81)
      567 at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.startSparkIfNeeded(SparkLauncher.java:202)
      568 at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:114)
      569 at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:279)
      570 at org.apache.pig.PigServer.launchPlan(PigServer.java:1378)
      571 at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363)
      572 at org.apache.pig.PigServer.execute(PigServer.java:1352)
      573 at org.apache.pig.PigServer.executeBatch(PigServer.java:403)
      574 at org.apache.pig.PigServer.executeBatch(PigServer.java:386)
      575 at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
      576 at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:233)
      577 at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:204)
      578 at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
      579 at org.apache.pig.Main.run(Main.java:611)
      580 at org.apache.pig.Main.main(Main.java:164)
      581 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      582 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      583 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      584 at java.lang.reflect.Method.invoke(Method.java:606)
      585 at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

        Attachments

        1. fix-pig-4183.patch
          0.3 kB
          liyunzhang

          Activity

            People

            • Assignee:
              kellyzly liyunzhang
              Reporter:
              kellyzly liyunzhang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: