Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7916

Snappy-java error when running hive query on spark [Spark Branch]

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • None
    • None
    • Spark

    Description

      Recently spark branch upgraded its dependency on Spark to 1.1.0-SNAPSHOT. While the new version addressed some lib conflicts (such as guava), I'm afraid that it also introduced new problems. The following might be one, when I set the master URL to be a spark standalone cluster:

      hive> set hive.execution.engine=spark;
      hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
      hive> set spark.master=spark://xzdt:7077;
      hive> select name, avg(value) from dec group by name;
      
      14/08/28 16:41:52 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 333.0 KB, free 128.0 MB)
      java.lang.reflect.InvocationTargetException
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:601)
          at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
          at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
          at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
          at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
          at org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:124)
          at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:207)
          at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:83)
          at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:68)
          at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:36)
          at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
          at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
          at org.apache.spark.SparkContext.broadcast(SparkContext.scala:809)
          at org.apache.spark.rdd.HadoopRDD.<init>(HadoopRDD.scala:116)
          at org.apache.spark.SparkContext.hadoopRDD(SparkContext.scala:541)
          at org.apache.spark.api.java.JavaSparkContext.hadoopRDD(JavaSparkContext.scala:318)
          at org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generateRDD(SparkPlanGenerator.java:160)
          at org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:88)
          at org.apache.hadoop.hive.ql.exec.spark.SparkClient.execute(SparkClient.java:156)
          at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.submit(SparkSessionImpl.java:52)
          at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:77)
          at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:161)
          at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
          at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1537)
          at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1304)
          at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1116)
          at org.apache.hadoop.hive.ql.Driver.run(Driver.java:940)
          at org.apache.hadoop.hive.ql.Driver.run(Driver.java:930)
          at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
          at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
          at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
          at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
          at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
          at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:601)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
          at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
          at java.lang.Runtime.loadLibrary0(Runtime.java:845)
          at java.lang.System.loadLibrary(System.java:1084)
          at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
          ... 42 more
      org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
          at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229)
          at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
          at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
          at org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:124)
          at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:207)
          at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:83)
          at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:68)
          at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:36)
          at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
          at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
          at org.apache.spark.SparkContext.broadcast(SparkContext.scala:809)
          at org.apache.spark.rdd.HadoopRDD.<init>(HadoopRDD.scala:116)
          at org.apache.spark.SparkContext.hadoopRDD(SparkContext.scala:541)
          at org.apache.spark.api.java.JavaSparkContext.hadoopRDD(JavaSparkContext.scala:318)
          at org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generateRDD(SparkPlanGenerator.java:160)
          at org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:88)
          at org.apache.hadoop.hive.ql.exec.spark.SparkClient.execute(SparkClient.java:156)
          at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.submit(SparkSessionImpl.java:52)
          at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:77)
          at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:161)
          at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
          at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1537)
          at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1304)
          at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1116)
          at org.apache.hadoop.hive.ql.Driver.run(Driver.java:940)
          at org.apache.hadoop.hive.ql.Driver.run(Driver.java:930)
          at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
          at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
          at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
          at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
          at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
          at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:601)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. [FAILED_TO_LOAD_NATIVE_LIBRARY] null
      14/08/28 16:41:52 ERROR ql.Driver: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. [FAILED_TO_LOAD_NATIVE_LIBRARY] null
      

      It could be a setup issue. Nevertheless, we need to take a look to be sure.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            xuefuz Xuefu Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment