[SPARK-15985] Reduce runtime overhead of a program that reads an primitive array in Dataset

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: SQL
• Assignee: Kazuaki Ishizaki
• Reporter: Kazuaki Ishizaki
• Created: 16/Jun/16 07:43
• Updated: 31/Aug/16 04:42
• Resolved: 31/Aug/16 04:42

Description

When a program read an array in Dataset, the code generator create some copy operations. If an array is for primitive type, there are some opportunities for optimizations in generated code to reduce runtime overhead.

val ds = Seq(Array(1.0, 2.0, 3.0), Array(4.0, 5.0, 6.0)).toDS()
ds.map(p => {
     var s = 0.0
     for (i <- 0 to 2) { s += p(i) }
     s
   }).show

Issue Links

• links to: [Github] Pull Request #13704 (kiszk)