Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14017

dataframe.dtypes -> pyspark.sql.types aliases

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 1.5.0
    • Fix Version/s: None
    • Component/s: PySpark
    • Environment:

      Python 2.7; Spark 1.5; Java 1.7; Hadoop 2.6; Scala 2.10

      Description

      Running following:

      #fix schema for gaid which should not be Double
      from pyspark.sql.types import *
      customSchema = StructType()
      for (col,typ) in tsp_orig.dtypes:
      if col=='Agility_GAID':
      typ='string'
      customSchema.add(col,typ,True)

      Getting
      ValueError: Could not parse datatype: bigint

      Looks like pyspark.sql.types doesn't know anything about bigint..
      Should it be aliased to LongType in pyspark.sql.types?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Tagar Ruslan Dautkhanov
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: