Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6574

Python Example sql.py not working in version 1.3

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.1, 1.4.0
    • Component/s: PySpark, SQL
    • Labels:
      None
    • Target Version/s:

      Description

      I downloaded spark version spark-1.3.0-bin-hadoop2.4.

      When the python version of sql.py is run the following error occurs:

      [root@nde-dev8-template python]#
      /root/spark-1.3.0-bin-hadoop2.4/bin/spark-submit sql.py
      Spark assembly has been built with Hive, including Datanucleus jars on
      classpath
      Traceback (most recent call last):
      File "/root/spark-1.3.0-bin-hadoop2.4/examples/src/main/python/sql.py",
      line 22, in <module>
      from pyspark.sql import Row, StructField, StructType, StringType,
      IntegerType
      ImportError: cannot import name StructField

      ----------------------------------------------------------------------
      The sql.py version, spark-1.2.1-bin-hadoop2.4, does not throw the error:

      [root@nde-dev8-template python]#
      /root/spark-1.2.1-bin-hadoop2.4/bin/spark-submit sql.py
      Spark assembly has been built with Hive, including Datanucleus jars on
      classpath
      15/03/27 14:18:44 WARN NativeCodeLoader: Unable to load native-hadoop
      library for your platform... using builtin-java classes where applicable
      15/03/27 14:19:41 WARN ThreadLocalRandom: Failed to generate a seed from
      SecureRandom within 3 seconds. Not enough entrophy?
      root

      – age: integer (nullable = true)
      – name: string (nullable = true)

      root

      – person_name: string (nullable = false)
      – person_age: integer (nullable = false)

      root

      – age: integer (nullable = true)
      – name: string (nullable = true)

      Justin

      -------------------------------------------------

      The OS/JAVA environments are:

      OS: Linux nde-dev8-template 2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11
      17:27:00 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

      JAVA: [root@nde-dev8-template bin]# java -version
      java version "1.7.0_51"
      Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
      Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

      The same error occurs when using bin/pyspark shell.

      >>> from pyspark.sql import StructField
      Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      ImportError: cannot import name StructField

      ---------------------------------------------------

      Any advice for resolving? Thanks in advance.

      Peter

        Attachments

          Activity

            People

            • Assignee:
              davies Davies Liu
              Reporter:
              davies Davies Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: