[SPARK-13745] Support columnar in memory representation on Big Endian platforms

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: SQL
• Labels: big-endian
• Assignee: Peter George Robbins
• Reporter: Tim Preece
• Created: 08/Mar/16 17:25
• Updated: 02/May/16 20:17
• Resolved: 02/May/16 20:17

Description

SPARK-12785 introduced a columnar in memory representation.

Currently this feature is explicitly only supported on Little Endian platorms. On Big Endian platforms the following exception is thrown:
"org.apache.commons.lang.NotImplementedException: Only little endian is supported."

This JIRA should be used to extend support to Big Endian architectures, and decide whether the "in memory" columnar format should be consistent with parquet format.

Issue Links

• links to: [Github] Pull Request #12397 (robbinspg)
• links to: [Github] Pull Request #12501 (robbinspg)