[PIG-4183] Fix classpath error when using pig command with Spark

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: liyunzhang
• Reporter: liyunzhang
• Created: 17/Sep/14 03:06
• Updated: 24/Feb/15 03:50
• Resolved: 18/Sep/14 06:39

Description

You can reproduce the bug in following steps
1.Build spark -0.9 env and build hadoop1 env:
2.Compile code: ant jar
3.Export $PIG_CLASSPATH
echo $PIG_CLASSPATH
/home/zly/prj/oss/pig/build/ivy/lib/Pig/*:/home/zly/prj/oss/hadoop-1.2.1/conf
4.Run: cd $PIG_HOME/bin; ./pig –x spark id.spark.pig
5.Error message found in pig log
549 ERROR 2998: Unhandled internal error. org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
550
551 java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
552 at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:607)
553 at org.eclipse.jetty.util.log.JettyAwareLogger.warn(JettyAwareLogger.java:431)
554 at org.eclipse.jetty.util.log.Slf4jLog.warn(Slf4jLog.java:69)
555 at org.eclipse.jetty.util.component.AbstractLifeCycle.setFailed(AbstractLifeCycle.java:204)
556 at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:74)
557 at org.apache.spark.HttpServer.start(HttpServer.scala:65)
558 at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:130)
559 at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:101)
560 at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcast.scala:70)
561 at org.apache.spark.broadcast.BroadcastManager.initialize(Broadcast.scala:81)
562 at org.apache.spark.broadcast.BroadcastManager.<init>(Broadcast.scala:68)
563 at org.apache.spark.SparkEnv$.create(SparkEnv.scala:175)
564 at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
565 at org.apache.spark.SparkContext.<init>(SparkContext.scala:100)
566 at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:81)
567 at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.startSparkIfNeeded(SparkLauncher.java:202)
568 at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:114)
569 at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:279)
570 at org.apache.pig.PigServer.launchPlan(PigServer.java:1378)
571 at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363)
572 at org.apache.pig.PigServer.execute(PigServer.java:1352)
573 at org.apache.pig.PigServer.executeBatch(PigServer.java:403)
574 at org.apache.pig.PigServer.executeBatch(PigServer.java:386)
575 at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
576 at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:233)
577 at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:204)
578 at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
579 at org.apache.pig.Main.run(Main.java:611)
580 at org.apache.pig.Main.main(Main.java:164)
581 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
582 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
583 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
584 at java.lang.reflect.Method.invoke(Method.java:606)
585 at org.apache.hadoop.util.RunJar.main(RunJar.java:160)