Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6042

spark-submit giving Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.hive.HiveContext.sql(Ljava/lang/String;)Lorg/apache/spark/sql/SchemaRDD;

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Question
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • None
    • None
    • SQL

    Description

      i am trying to create a table in hive using spark , i tried the code in spark-shell and it worked and created the table , but when i use spark-submit it gives this error:
      Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.hive.HiveContext.sql(Ljava/lang/String;)Lorg/apache/spark/sql/SchemaRDD;

      at this line : sqlContext.sql("CREATE TABLE IF NOT EXISTS Test123 (key INT, value STRING)")

      the code i submit is :

      import org.apache.spark.SparkContext
      import org.apache.spark.SparkContext._
      import org.apache.spark.SparkConf
      import org.apache.spark._
      import org.apache.spark.streaming._
      import org.apache.spark.streaming.StreamingContext._
      import org.apache.spark.storage.StorageLevel
      import org.apache.spark.streaming.flume._
      import org.apache.spark.util.IntParam
      import org.apache.spark.sql._
      import org.apache.spark.sql.hive.HiveContext

      object WordCount {
      def main(args: Array[String]) {
      if (args.length < 2)

      { System.err.println( "Usage: WordCount <host> <port>") System.exit(1) }

      val Array(host, port) = args

      val batchInterval = Milliseconds(2000)

      // Create the context and set the batch size
      val sparkConf = new SparkConf().setAppName("WordCount")
      val sc = new SparkContext(sparkConf)
      val ssc = new StreamingContext(sc, batchInterval)

      // Create a flume stream
      val stream = FlumeUtils.createStream(ssc, host, port.toInt)

      // Print out the count of events received from this server in each batch
      stream.count().map(cnt => "Received !!!:::::" + cnt + " flume events." ).print()

      // it holds the string stream (converted event body array into string)
      val body = stream.map(e => new String(e.event.getBody.array))

      val counts = body.flatMap(line => line.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("
      s+"))
      .map(word => (word, 1))
      .reduceByKey(_ + _)

      val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
      sqlContext.sql("CREATE TABLE IF NOT EXISTS tarek (key INT, value STRING)")

      ssc.start()
      ssc.awaitTermination()
      }
      }

      i tried to submit this code on local[*] and yarn-master and both gave same error , the error is at this specific line :
      "sqlContext.sql("CREATE TABLE IF NOT EXISTS tarek (key INT, value STRING)")"
      but i executed the exact same line to execute create table query and it succeeded ,, i found a kinda similar issue here :
      https://issues.apache.org/jira/browse/SPARK-6018

      can any one helps please ?

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            tarek_abouzeid Tarek Abouzeid
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment