Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-3950

zeppelin can't retrive data from spark native k8s cluster correctly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: zeppelin-interpreter
    • Labels:
      None
    • Environment:

      centos 7.5 

      java 1.8

      zeppelin 0.8.0 0.9.0 SNAPSHOT

      spark 2.4.0

    • Flags:
      Important

      Description

      I set up native spark on K8S acording to http://spark.apache.org/docs/latest/running-on-kubernetes.html . and set spark interpreter's master :k8s://https://10.64.17.119:6443 

      I tried 2 case:

      case 1:

      import spark.implicits._
      val df = Seq(("Judy", "50"),("Randy", "60")).toDF("int_column", "string_column", "date_column")
      df.registerTempTable("peopleTB")
      %sql
      select * from peopleTB

      case 2:

      %spark
      case class WordCountTable(word: String, count: Integer)
      val textFile = sc.textFile("hdfs://10.64.17.119:9000/wordcounter.txt")
      val wordCount = textFile.map(line => line.split(",")).map(line => WordCountTable(line(0),line(1).toInt)).toDF()
      wordCount.registerTempTable("wordCountTB")

      %sql
      select * from wordCountTB

       

       

      Case1 works well , Case 2 alway hung at "select * from wordCountTB"

      when I change spark interpreter's master to local[*], the 2 case all works well.

       

      It seems  that zeppelin can't retrive data from spark's native k8s cluster correctly

       

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              yuanzhangzhu YuanzhangZhu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: