Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6315

SparkSQL 1.3.0 (RC3) fails to read parquet file generated by 1.1.1

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.3.0
    • 1.3.1, 1.4.0
    • SQL
    • None

    Description

      Parquet files generated by Spark 1.1 have a deprecated representation of the schema. In Spark 1.3 we fail to read these files through the new Parquet code path. We should continue to read these files until we formally deprecate this representation.

      As a workaround:

      SET spark.sql.parquet.useDataSourceApi=false
      

      Attachments

        Activity

          People

            lian cheng Cheng Lian
            marmbrus Michael Armbrust
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: