[PIG-4859] Need upgrade snappy-java.version to 1.1.1.3

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: liyunzhang
• Reporter: liyunzhang
• Created: 01/Apr/16 06:08
• Updated: 21/Jun/17 09:18
• Resolved: 12/Apr/16 03:20

Description

run pig on spark on yarn-client env as following:
export SPARK_MASTER="yarn-client"
./pig -x spark xxx.pig

Throw error like following:

main] 2016-03-30 16:52:26,115 INFO  scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Job 0 failed: saveAsNewAPIHadoopDataset at StoreConverter.java:101, took 73.980147 s
19895 [main] 2016-03-30 16:52:26,119 ERROR spark.JobGraphBuilder (JobGraphBuilder.java:sparkOperToRDD(166)) - throw exception in sparkOperToRDD:
19896 org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, zly1.sh.intel.com): java      .lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I
19897         at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
19898         at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:541)
19899         at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:350)
19900         at org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158)
19901         at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
19902         at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2313)
19903         at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2326)
19904         at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797)
19905         at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
19906         at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
19907         at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:64)
19908         at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:64)
19909         at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:103)
19910         at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:216)
19911         at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)