[SPARK-6574] Python Example sql.py not working in version 1.3

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.1, 1.4.0
• Component/s: PySpark, SQL
• Target Version/s: 1.3.1
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 27/Mar/15 16:58
• Updated: 27/Mar/15 18:42
• Resolved: 27/Mar/15 18:42

Description

I downloaded spark version spark-1.3.0-bin-hadoop2.4.

When the python version of sql.py is run the following error occurs:

[root@nde-dev8-template python]#
/root/spark-1.3.0-bin-hadoop2.4/bin/spark-submit sql.py
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
Traceback (most recent call last):
File "/root/spark-1.3.0-bin-hadoop2.4/examples/src/main/python/sql.py",
line 22, in <module>
from pyspark.sql import Row, StructField, StructType, StringType,
IntegerType
ImportError: cannot import name StructField

----------------------------------------------------------------------
The sql.py version, spark-1.2.1-bin-hadoop2.4, does not throw the error:

[root@nde-dev8-template python]#
/root/spark-1.2.1-bin-hadoop2.4/bin/spark-submit sql.py
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
15/03/27 14:18:44 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/27 14:19:41 WARN ThreadLocalRandom: Failed to generate a seed from
SecureRandom within 3 seconds. Not enough entrophy?
root

– age: integer (nullable = true)

– name: string (nullable = true)

root

– person_name: string (nullable = false)

– person_age: integer (nullable = false)

root

– age: integer (nullable = true)

– name: string (nullable = true)

Justin

-------------------------------------------------

The OS/JAVA environments are:

OS: Linux nde-dev8-template 2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11
17:27:00 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

JAVA: [root@nde-dev8-template bin]# java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

The same error occurs when using bin/pyspark shell.

>>> from pyspark.sql import StructField
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name StructField

---------------------------------------------------

Any advice for resolving? Thanks in advance.

Peter

Issue Links

• links to: [Github] Pull Request #5230 (davies)