[SPARK-6504] Cannot read Parquet files generated from different versions at once

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.1
• Fix Version/s: 1.3.1
• Component/s: SQL
• Assignee: Michael Armbrust
• Reporter: Marius Soutier
• Created: 24/Mar/15 14:03
• Updated: 14/Mar/16 09:58
• Resolved: 16/Sep/15 20:24

Description

When trying to read Parquet files generated by Spark 1.1.1 and 1.2.1 at the same time via `sqlContext.parquetFile("fileFrom1.1.parqut,fileFrom1.2.parquet")` an exception occurs:

could not merge metadata: key org.apache.spark.sql.parquet.row.metadata has conflicting values: [{"type":"struct","fields":[{"name":"date","type":"string","nullable":true,"metadata":{}},{"name":"account","type":"string","nullable":true,"metadata":{}},{"name":"impressions","type":"long","nullable":false,"metadata":{}},{"name":"cost","type":"double","nullable":false,"metadata":{}},{"name":"clicks","type":"long","nullable":false,"metadata":{}},{"name":"conversions","type":"long","nullable":false,"metadata":{}},{"name":"orderValue","type":"double","nullable":false,"metadata":{}}]}, StructType(List(StructField(date,StringType,true), StructField(account,StringType,true), StructField(impressions,LongType,false), StructField(cost,DoubleType,false), StructField(clicks,LongType,false), StructField(conversions,LongType,false), StructField(orderValue,DoubleType,false)))]

The Schema is exactly equal.