[SPARK-17093] Roundtrip encoding of array<struct<>> fields is wrong when whole-stage codegen is disabled

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: SQL
• Labels: correctness
• Target Version/s: 2.0.1
• Assignee: Liwei Lin(Inactive)
• Reporter: Josh Rosen
• Created: 16/Aug/16 21:25
• Updated: 31/Aug/16 21:16
• Resolved: 25/Aug/16 12:20

Description

The following failing test demonstrates a bug where Spark mis-encodes array-of-struct fields if whole-stage codegen is disabled:

withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false") {
  val data = Array(Array((1, 2), (3, 4)))
  val ds = spark.sparkContext.parallelize(data).toDS()
  assert(ds.collect() === data)
}

When wholestage codegen is enabled (the default), this works fine. When it's disabled, as in the test above, Spark returns Array(Array((3,4), (3,4))). Because the last element of the array appears to be repeated my best guess is that the interpreted evaluation codepath forgot to copy() somewhere.

Issue Links

• relates to: SPARK-17061 Incorrect results returned following a join of two datasets and a map step where total number of columns >100
• links to: [Github] Pull Request #14698 (lw-lin)
• links to: [Github] Pull Request #14806 (hvanhovell)