[PHOENIX-2784] phoenix-spark: Allow coercion of DATE fields to TIMESTAMP when loading DataFrames

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.7.0
• Fix Version/s: 4.8.0
• Flags: Patch
• Assignee: Josh Mahonin
• Reporter: Josh Mahonin
• Created: 19/Mar/16 18:32
• Updated: 11/Aug/16 08:29
• Resolved: 16/May/16 16:38

Description

The Phoenix DATE type is internally represented as an 8 bytes, which can store a full 'yyyy-MM-dd hh:mm:ss' time component. However, Spark SQL follows the SQL Date spec and keeps only the 'yyyy-MM-dd' portion as a 4 byte type. When loading Phoenix DATE columns using the Spark DataFrame API, the 'hh:mm:ss' component is lost.

This patch allows setting a new 'dateAsTimestamp' option when loading a DataFrame, which will coerce the underlying Date object to a Timestamp so that the full time component is loaded.