[SPARK-23348] append data using saveAsTable should adjust the data types

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.2, 2.2.1, 2.3.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Wenchen Fan
• Reporter: Wenchen Fan
• Created: 07/Feb/18 08:35
• Updated: 08/Feb/18 08:26
• Resolved: 08/Feb/18 08:26

Description

 

Seq(1 -> "a").toDF("i", "j").write.saveAsTable("t")

Seq("c" -> 3).toDF("i", "j").write.mode("append").saveAsTable("t")

scala> sql("select * from t").show

 
This query will fail with a strange error:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 10.0 failed 1 times, most recent failure: Lost task 1.0 in stage 10.0 (TID 15, localhost, executor driver): java.lang.UnsupportedOperationException: Unimplemented type: IntegerType
 at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBinaryBatch(VectorizedColumnReader.java:473)
 at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:214)
 at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:261)
...

 
All Spark 2.X are the same. For Spark 1.6.3,

scala> sql("select * from tx").show
+----+---+
|   i|  j|
+----+---+
|null|  3|
|   1|  a|
+----+---+

Issue Links

• links to: [Github] Pull Request #20527 (cloud-fan)