Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6315

SparkSQL 1.3.0 (RC3) fails to read parquet file generated by 1.1.1

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.1, 1.4.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      Parquet files generated by Spark 1.1 have a deprecated representation of the schema. In Spark 1.3 we fail to read these files through the new Parquet code path. We should continue to read these files until we formally deprecate this representation.

      As a workaround:

      SET spark.sql.parquet.useDataSourceApi=false
      

        Attachments

          Activity

            People

            • Assignee:
              lian cheng Cheng Lian
              Reporter:
              marmbrus Michael Armbrust
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: