[SPARK-31359] Speed up timestamps rebasing

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 06/Apr/20 11:29
• Updated: 10/Apr/20 06:21
• Resolved: 10/Apr/20 03:24

Description

Currently, rebasing of timestamps is performed via conversions to local timestamps and back to microseconds. This is CPU intensive operation which can be avoid by converting via pre-calculated tables per each time zone. For example, the below is timestamps when diffs are changed in America/Los_Angeles time zone for the range 0001-01-01...2100-01-01

0001-01-01T00:00 diff = -2872 minutes
0100-03-01T00:00 diff = -1432 minutes
0200-03-01T00:00 diff = 7 minutes
0300-03-01T00:00 diff = 1447 minutes
0500-03-01T00:00 diff = 2887 minutes
0600-03-01T00:00 diff = 4327 minutes
0700-03-01T00:00 diff = 5767 minutes
0900-03-01T00:00 diff = 7207 minutes
1000-03-01T00:00 diff = 8647 minutes
1100-03-01T00:00 diff = 10087 minutes
1300-03-01T00:00 diff = 11527 minutes
1400-03-01T00:00 diff = 12967 minutes
1500-03-01T00:00 diff = 14407 minutes
1582-10-15T00:00 diff = 7 minutes
1883-11-18T12:22:58 diff = 0 minutes

It seems it is possible to build rebasing maps, and perform rebasing via the maps.

Issue Links

• is related to: SPARK-31385 Results of Julian-Gregorian rebasing don't match to Gregorian-Julian rebasing
• links to: GitHub Pull Request #28119
• links to: GitHub Pull Request #28163
• links to: GitHub Pull Request #28166