[SPARK-18350] Support session local timezone

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: SQL
• Labels: releasenotes
• Target Version/s: 2.2.0
• Assignee: Takuya Ueshin
• Reporter: Reynold Xin
• Created: 08/Nov/16 06:37
• Updated: 12/Oct/17 17:30
• Resolved: 08/May/17 07:50

Description

As of Spark 2.1, Spark SQL assumes the machine timezone for datetime manipulation, which is bad if users are not in the same timezones as the machines, or if different users have different timezones.

We should introduce a session local timezone setting that is used for execution.

An explicit non-goal is locale handling.

Issue Links

• relates to: SPARK-12297 Add work-around for Parquet/Hive int96 timestamp bug.
• relates to: SPARK-15834 Time zone / locale sensitivity umbrella
• links to: [Github] Pull Request #16308 (ueshin)

Sub-Tasks

• Infrastructure for session local timezone support: Resolved
• Timezone support in CSV/JSON parsing: Resolved
• Timezone support in partition values.: Resolved
• make it clear that `timeZone` option is a general option in DataFrameReader/Writer, DataStreamReader/Writer: Resolved