[SPARK-23314] Pandas grouped udf on dataset with timestamp column error

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: PySpark
• Assignee: Li Jin
• Reporter: Felix Cheung
• Created: 02/Feb/18 07:13
• Updated: 12/Dec/22 18:10
• Resolved: 11/Feb/18 08:32

Description

Under  SPARK-22216

When testing pandas_udf on group bys, I saw this error with the timestamp column.

File "pandas/_libs/tslib.pyx", line 3593, in pandas._libs.tslib.tz_localize_to_utc

AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:29:30'), try using the 'ambiguous' argument

For details, see Comment box. I'm able to reproduce this on the latest branch-2.3 (last change from Feb 1 UTC)

Issue Links

• links to: [Github] Pull Request #20537 (icexelloss)