Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19134

Fix several sql, mllib and status api examples not working

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.2.0
    • MLlib, PySpark, SQL
    • None

    Description

      binary_classification_metrics_example.py

      ./bin/spark-submit examples/src/main/python/mllib/binary_classification_metrics_example.py
      
        File ".../spark/examples/src/main/python/mllib/binary_classification_metrics_example.py", line 39, in <lambda>
          .rdd.map(lambda row: LabeledPoint(row[0], row[1]))
        File ".../spark/python/pyspark/mllib/regression.py", line 54, in __init__
          self.features = _convert_to_vector(features)
        File ".../spark/python/pyspark/mllib/linalg/__init__.py", line 80, in _convert_to_vector
          raise TypeError("Cannot convert type %s into Vector" % type(l))
      TypeError: Cannot convert type <class 'pyspark.ml.linalg.SparseVector'> into Vector
      

      status_api_demo.py

      PYSPARK_PYTHON=python3 ./bin/spark-submit examples/src/main/python/status_api_demo.py
      
      Traceback (most recent call last):
        File ".../spark/examples/src/main/python/status_api_demo.py", line 22, in <module>
          import Queue
      ImportError: No module named 'Queue'
      

      bisecting_k_means_example.py

      ./bin/spark-submit examples/src/main/python/mllib/bisecting_k_means_example.py
      
      Traceback (most recent call last):
        File "/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/examples/src/main/python/mllib/bisecting_k_means_example.py", line 46, in <module>
          model.save(sc, path)
      AttributeError: 'BisectingKMeansModel' object has no attribute 'save'
      

      elementwise_product_example.py

      ./bin/spark-submit examples/src/main/python/mllib/elementwise_product_example.py
      
      Traceback (most recent call last):
        File "/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/examples/src/main/python/mllib/elementwise_product_example.py", line 48, in <module>
          for each in transformedData2.collect():
        File "/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/python/pyspark/mllib/linalg/__init__.py", line 478, in __getattr__
          return getattr(self.array, item)
      AttributeError: 'numpy.ndarray' object has no attribute 'collect'
      

      hive.py

      ./bin/spark-submit examples/src/main/python/sql/hive.py
      
      Traceback (most recent call last):
        File "/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/examples/src/main/python/sql/hive.py", line 47, in <module>
          spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING) USING hive")
        File "/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 541, in sql
        File "/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
        File "/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
      pyspark.sql.utils.AnalysisException: 'org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:./spark-warehouse);'
      

      SparkHiveExample

      ./bin/run-example sql.hive.SparkHiveExample
      
      Exception in thread "main" org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:./spark-warehouse
      	at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:498)
      	at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:484)
      	at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1668)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:722)
      	at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:686)
      	at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:686)
      	at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:686)
      	at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:283)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:230)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:229)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:272)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:685)
      

      JavaSparkHiveExample

      ./bin/run-example sql.hive.JavaSparkHiveExample
      
      Exception in thread "main" org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:./spark-warehouse
      	at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:498)
      	at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:484)
      	at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1668)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:722)
      	at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:686)
      	at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:686)
      	at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:686)
      

      Attachments

        Activity

          People

            hyukjin.kwon Hyukjin Kwon
            hyukjin.kwon Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: