Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4859

Need upgrade snappy-java.version to 1.1.1.3

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      run pig on spark on yarn-client env as following:
      export SPARK_MASTER="yarn-client"
      ./pig -x spark xxx.pig

      Throw error like following:

      main] 2016-03-30 16:52:26,115 INFO  scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Job 0 failed: saveAsNewAPIHadoopDataset at StoreConverter.java:101, took 73.980147 s
      19895 [main] 2016-03-30 16:52:26,119 ERROR spark.JobGraphBuilder (JobGraphBuilder.java:sparkOperToRDD(166)) - throw exception in sparkOperToRDD:
      19896 org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, zly1.sh.intel.com): java      .lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I
      19897         at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
      19898         at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:541)
      19899         at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:350)
      19900         at org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158)
      19901         at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
      19902         at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2313)
      19903         at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2326)
      19904         at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797)
      19905         at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
      19906         at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
      19907         at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:64)
      19908         at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:64)
      19909         at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:103)
      19910         at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:216)
      19911         at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)
      

        Attachments

        1. PIG-4859.patch
          0.4 kB
          liyunzhang

          Activity

            People

            • Assignee:
              kellyzly liyunzhang
              Reporter:
              kellyzly liyunzhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: