[SPARK-7691] Use type-specific row accessor functions in CatalystTypeConverters' toScala functions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Epic Link: Tungsten Phase 1
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 17/May/15 18:26
• Updated: 03/Jun/15 05:47
• Resolved: 03/Jun/15 05:47

Description

CatalystTypeConverters's Catalyst row to Scala row converters access columns' values via the generic Row.get() call rather than using type-specific accessor methods. If we refactor the internal converter interfaces slightly, we can pass the row and column number into the converter function and allow it to do its own type-specific field extraction, similar to what we do in UnsafeRowConverter. This is a blocker for being able to unit test new operators that I'm developing as part of Project Tungsten, since those operators may output UnsafeRow instances which don't support the generic get().

Issue Links

• relates to: SPARK-6620 Speed up toDF() and rdd() functions by constructing converters in ScalaReflection
• links to: [Github] Pull Request #6222 (JoshRosen)