[SPARK-14017] dataframe.dtypes -> pyspark.sql.types aliases

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 1.5.0
• Component/s: PySpark
• Labels: dataframe datatypes pyspark python

Environment

Python 2.7; Spark 1.5; Java 1.7; Hadoop 2.6; Scala 2.10

• Assignee: Unassigned
• Reporter: Ruslan Dautkhanov
• Created: 18/Mar/16 23:35
• Updated: 08/Oct/16 19:03
• Resolved: 08/Oct/16 19:03

Description

Running following:

#fix schema for gaid which should not be Double
from pyspark.sql.types import *
customSchema = StructType()
for (col,typ) in tsp_orig.dtypes:
if col=='Agility_GAID':
typ='string'
customSchema.add(col,typ,True)

Getting
ValueError: Could not parse datatype: bigint

Looks like pyspark.sql.types doesn't know anything about bigint..
Should it be aliased to LongType in pyspark.sql.types?