[SPARK-17781] datetime is serialized as double inside dapply()

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0
• Component/s: SparkR
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Hossein Falaki
• Created: 04/Oct/16 23:30
• Updated: 21/May/19 04:33
• Resolved: 21/May/19 04:33

Description

When we ship a SparkDataFrame to workers for dapply family functions, inside the worker DateTime objects are serialized as double.

To reproduce:

df <- createDataFrame(data.frame(id = 1:10, date = Sys.Date()))
dapplyCollect(df, function(x) { return(x$date) })