[SPARK-22010] Slow fromInternal conversion for TimestampType

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.2.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Maciej Bryński
• Created: 14/Sep/17 10:44
• Updated: 12/Dec/22 18:10
• Resolved: 07/Nov/17 09:06

Description

To convert timestamp type to python we are using

datetime.datetime.fromtimestamp(ts // 1000000).replace(microsecond=ts % 1000000)

code.

In [34]: %%timeit
    ...: datetime.datetime.fromtimestamp(1505383647).replace(microsecond=12344)
    ...:
4.58 µs ± 558 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

It's slow, because:

1. we're trying to get TZ on every conversion
2. we're using replace method

Proposed solution: custom datetime conversion and move calculation of TZ to module

Issue Links

• links to: [Github] Pull Request #19234 (maver1ck)